Blog Infos
Author
Published
Topics
,
Published

 

TL;DR: The Android API’s MediaCodec class provides access to low-level media encoder/decoder components for processing audio, video, and compressed data. Qualcomm Technologies, Inc. provides vendor extensions for MediaCodec to unlock additional capabilities on Android devices powered by Snapdragon mobile platforms. This article describes how to use these extensions to enable regions of interest, long-term references, and encoder stats for videos in your media-rich Android apps.

If you’re developing media-rich apps for Android, you’ll want to become familiar with the Android API’s MediaCodec class. MediaCodec provides access to low-level media encoder/decoder (CODEC) components for processing audio, video, and compressed data. With it, you can incorporate features like low-latency decoding to enhance your app’s capabilities. So when the app runs on devices powered by a Snapdragon platform, you can unlock further capabilities thanks to the MediaCodec vendor extensions.

Let’s take a closer look at how MediaCodec works and how you can take advantage of it.

How MediaCodec Works

Developers work with MediaCodec as follows:

  • The app (client) configures MediaCodec.
  • MediaCodec provides your app with empty input buffers.
  • The app fills those buffers with data and sends them to the codec for processing.
  • The codec transforms the data received in those buffers.
  • The transformed data is made available to your app via output buffers.
  • At the end of the app’s lifecycle, the app stops and releases MediaCodec.

This workflow is shown below:

Features from Snapdragon Platforms

In addition to the built-in codecs and features provided by the Android API, the Android on MediaCodec extensions with Snapdragon technology include codecs and other features which unlock several hardware-accelerated capabilities on Snapdragon platforms, such as:

  • Regions of Interest (ROI) — dynamically improves image quality in certain regions of the video.
  • Long-Term Reference (LTR) — encodes reference frames for use in motion-compensated prediction in future frames.
  • Encoder Statistics — provides per-frame statistics describing the complexity of a scene. Apps can use this information to control coding parameters dynamically (e.g., bitrate) and adjust pre-encoding filtering to achieve better compression without introducing high-compression artifacts.

The following diagram shows the overall Android development stack and how the extensions fit:

MediaCodec Extensions with Snapdragon technology are located near the bottom of the stack, close to the hardware.

Extensions may also be used by support libraries (e.g., Media3 or CameraX), in which case some of the features of those extensions may already be enabled. The following diagram shows where the support libraries fit in the stack:

Support libraries with Snapdragon technology sit between your app and the Android SDK, and access functionality provided by the vendor extensions

Job Offers

Job Offers

There are currently no vacancies.

OUR VIDEO RECOMMENDATION

No results found.

Jobs

Four Steps to set up MediaCodec Vendor Extensions

Setting up MediaCodec vendor extensions at runtime involves the following steps:

  1. Enumerate the extensions supported by the device.
  2. Query for the values (e.g., range of values) which can be used by a supported extension.
  3. Apply a configuration to the extension.

Let’s take a closer look at each.

  1. Enumerate the Extensions Supported by the Device

It’s imperative to first check if an extension is supported on the device before using it. After instantiating MediaCodec, invoke its getSupportedVendorParameters() method, which returns a collection of supported extension names. If the name of the extension you want to use is in the collection, then it’s supported by the device.

The following code sample shows how to check if the High Efficiency Video Coding (HEVC) codec is available on a device:

MediaCodec codec = MediaCodec.createEncoderByType("video/hevc");

List supportedExtensions = codec.getSupportedVendorParameters();

if (supportedExtensions.indexOf(qti.video.KEY_INIT_QP_I_FRAME_ENABLE) != -1)
{
    //Device supports extension
}

Note: See QMediaExtensions.java for the definition of KEY_INIT_QP_I_FRAME_ENABLE.

2. Query for the Values Which can be used by a Supported Extension

Once you’ve established that the device supports the extension you want to use, you then need to determine the parameters it supports (can vary by device).

Start by requesting the extension’s capabilities using the CreateForCodec() method of the QMediaCodecCapabilities class. This returns a collection supported parameter names. If the name of the parameter that you want to use is in the list, then it’s supported by the device. You can then invoke getParameterDescriptor() to determine the parameter’s supported values.

The following code sample shows how to check if an HEVC codec supports LTR:

MediaCodec codec = MediaCodec.createByCodecName("c2.qti.hevc.encoder");

// codec.getSupportedVendorParameters() should contain KEY_LTR_MAX_FRAMES, KEY_LTR_MARK_FRAME, KEY_LTR_USE_FRAME and KEY_LTR_RESPONSE

/*
* vendorParams should contain KEY_LTR_MAX_FRAMES, KEY_LTR_MARK_FRAME, KEY_LTR_USE_FRAME and KEY_LTR_RESPONSE
*/
if (supportedExtensions.indexOf(KEY_LTR_MAX_FRAMES) != -1
     && supportedExtensions.indexOf(KEY_LTR_MARK_FRAME) != -1
     && supportedExtensions.indexOf(KEY_LTR_USE_FRAME) != -1
     && supportedExtensions.indexOf(KEY_LTR_RESPONSE) != -1)
{
     // LTR feature is supported }


QMediaCodecCapabilities qCodecCaps = QMediaCodecCapabilities.CreateForCodec("c2.qti.hevc.encoder", "video/hevc");

switch (descriptor.getType()) {
     case MediaFormat.TYPE_INTEGER:
         SupportedValues erLtrMaxFramesValues = qCodecCaps.getParameterRangeInteger(KEY_LTR_MAX_FRAMES);

         if (intValues.getType() == SupportedValues.TYPE_RANGE) {
             ValueRange intRange = intvalues.getRange();
            // min: intRange.getMin()
            // max: intRange.getMax()
            // step: intRange.getstep()
        ) else {
            // not a range
     default:
//not integer value type
}

// qCodecCaps.getParameterDescriptor(KEY_ER_LTR_MARK_FRAME).getType() -> MediaFormat.TYPE_INTEGER
SupportedValues erLtrMarkFramesValues = qCodecCaps.getParameterRangeInteger(KEY_ER_LTR_MARK_FRAME);
// erLtrMarkFramesValues.getType() -> SupportedValues.TYPE_RANGE

// qCodecCaps.getParameterDescriptor(KEY_ER_LTR_USE_FRAME).getType() -> MediaFormat.TYPE_INTEGER
SupportedValues erLtrmaxFramesValues = qCodecCaps.getParameterRangeInteger(KEY_ER_LTR_USE_FRAME);
// erLtrmaxFramesValues.getType() -> SupportedValues.TYPE_RANGE

// qCodecCaps.getParameterDescriptor(KEY_ER_LTR_RESPONSE).getType() -> MediaFormat.TYPE_INTEGER
SupportedValues erLtrResponseValues = qCodecCaps.getParameterRangeInteger(KEY_ER_LTR_RESPONSE);
// erLtrResponseValues.getType() -> SupportedValues.TYPE_RANGE

3. Apply a Configuration to the Vendor Extension

At this point, you have all the information you need to configure the supported extension. You can now configure static extension properties which remain unchanged throughout the application’s lifecycle, and dynamic extension properties which can be changed at any point.

Begin by checking if the name of the parameter you want to use is in the supported extensions list. For a static extension property, use the MediaFormat’s set methods (e.g., setInteger()) to configure the name/value pairs for the various parameters. Then pass the MediaFormat instance to the configure() method of MediaCodec. The following code sample shows how to set the maximum frame count for HEVC at a resolution of 1280×720:

MediaCodec codec = MediaCodec.createEncoderByType ("video/hevc");
MediaFormat format = MediaFormat.createVideoFormat("video/hevc", 1280, 720);

// check if the extension is supported first, before using it
if (supportedExtensions.indexOf{qti.video.KEY_LTR_MAX_COUNT_FRAME) == -1) {
     // return error, extension KEY_LTR_MAX_COUNT_FRAME is not supported by the Android Device
}
else {
     format.setInteger(qti.video.KEY_LTR_MAX_COUNT_FRAMES, 4);
     codec.configure(format);
}

Setting a value for a dynamic extension property is similar, except that the key/value pair is stored in a Bundle and passed to the MediaCodec instance via its setParameters() method. The following code sample shows how set a QP number for a frame:

codec.start();
if (supportedExtensions.indexOf(qti.video.KEY_FRAME_QP) ! = -1) {
     Bundle frameQP = new Bundle();
     frameQP.setinteger(qti.video.KEY_FRAME_QP, 24);
     codec.setParameters(frameQP);
     codec.queueinputBuffer(...);
}

Conclusion

Android’s MediaCodec class, in conjunction with the Android on Snapdragon vendor extensions, can unlock powerful features for media-rich apps. You can use it to discover, configure, and adjust your app’s capabilities based on the underlying hardware’s support. And when running on the latest devices with Snapdragon, you can incorporate ROI, LTR, and encoder statistics as well as several built-in Android codec features into your apps.

For more information, check out Android on SnapdragoMediaCodec Extensions developer documentation.

Author: Morris Novello
Morris Novello is Staff Manager and Head of Developer Marketing at Qualcomm Technologies, Inc. In this role, he provides marketing leadership, strategy, and execution for both Developers and Partners across QTI technologies.

Snapdragon branded products are products of Qualcomm Technologies, Inc. and/or its subsidiaries.

This article was previously published on proandroiddev.com

YOU MAY BE INTERESTED IN

YOU MAY BE INTERESTED IN

blog
This tutorial is the second part of the series. It’ll be focussed on developing…
READ MORE
blog
We recently faced a problem with our application getting updated and reaching slowly to…
READ MORE
blog
A few weeks ago I started with a simple question — how to work…
READ MORE
blog
One of the main functions of a mobile phone was to store contacts information.…
READ MORE
Menu