Features from Snapdragon Platforms

In addition to the built-in codecs and features provided by the Android API, the Android on MediaCodec extensions with Snapdragon technology include codecs and other features which unlock several hardware-accelerated capabilities on Snapdragon platforms, such as:

• Regions of Interest (ROI) — dynamically improves image quality in certain regions of the video.
• Long-Term Reference (LTR) — encodes reference frames for use in motion-compensated prediction in future frames.
• Encoder Statistics — provides per-frame statistics describing the complexity of a scene. Apps can use this information to control coding parameters dynamically (e.g., bitrate) and adjust pre-encoding filtering to achieve better compression without introducing high-compression artifacts.

The following diagram shows the overall Android development stack and how the extensions fit:

MediaCodec Extensions with Snapdragon technology are located near the bottom of the stack, close to the hardware.

Extensions may also be used by support libraries (e.g., Media3 or CameraX), in which case some of the features of those extensions may already be enabled. The following diagram shows where the support libraries fit in the stack:

Support libraries with Snapdragon technology sit between your app and the Android SDK, and access functionality provided by the vendor extensions

Four Steps to set up MediaCodec Vendor Extensions

Setting up MediaCodec vendor extensions at runtime involves the following steps:

1. Enumerate the extensions supported by the device.
2. Query for the values (e.g., range of values) which can be used by a supported extension.
3. Apply a configuration to the extension.

Let’s take a closer look at each.

1. Enumerate the Extensions Supported by the Device

It’s imperative to first check if an extension is supported on the device before using it. After instantiating MediaCodec, invoke its getSupportedVendorParameters() method, which returns a collection of supported extension names. If the name of the extension you want to use is in the collection, then it’s supported by the device.

The following code sample shows how to check if the High Efficiency Video Coding (HEVC) codec is available on a device:

MediaCodec codec = MediaCodec.createEncoderByType("video/hevc");

List supportedExtensions = codec.getSupportedVendorParameters();

if (supportedExtensions.indexOf(qti.video.KEY_INIT_QP_I_FRAME_ENABLE) != -1)
{
    //Device supports extension
}

Note: See QMediaExtensions.java for the definition of KEY_INIT_QP_I_FRAME_ENABLE.

2. Query for the Values Which can be used by a Supported Extension

Once you’ve established that the device supports the extension you want to use, you then need to determine the parameters it supports (can vary by device).

Start by requesting the extension’s capabilities using the CreateForCodec() method of the QMediaCodecCapabilities class. This returns a collection supported parameter names. If the name of the parameter that you want to use is in the list, then it’s supported by the device. You can then invoke getParameterDescriptor() to determine the parameter’s supported values.

The following code sample shows how to check if an HEVC codec supports LTR:

MediaCodec codec = MediaCodec.createByCodecName("c2.qti.hevc.encoder");

// codec.getSupportedVendorParameters() should contain KEY_LTR_MAX_FRAMES, KEY_LTR_MARK_FRAME, KEY_LTR_USE_FRAME and KEY_LTR_RESPONSE

/*
* vendorParams should contain KEY_LTR_MAX_FRAMES, KEY_LTR_MARK_FRAME, KEY_LTR_USE_FRAME and KEY_LTR_RESPONSE
*/
if (supportedExtensions.indexOf(KEY_LTR_MAX_FRAMES) != -1
     && supportedExtensions.indexOf(KEY_LTR_MARK_FRAME) != -1
     && supportedExtensions.indexOf(KEY_LTR_USE_FRAME) != -1
     && supportedExtensions.indexOf(KEY_LTR_RESPONSE) != -1)
{
     // LTR feature is supported }


QMediaCodecCapabilities qCodecCaps = QMediaCodecCapabilities.CreateForCodec("c2.qti.hevc.encoder", "video/hevc");

switch (descriptor.getType()) {
     case MediaFormat.TYPE_INTEGER:
         SupportedValues erLtrMaxFramesValues = qCodecCaps.getParameterRangeInteger(KEY_LTR_MAX_FRAMES);

         if (intValues.getType() == SupportedValues.TYPE_RANGE) {
             ValueRange intRange = intvalues.getRange();
            // min: intRange.getMin()
            // max: intRange.getMax()
            // step: intRange.getstep()
        ) else {
            // not a range
     default:
//not integer value type
}

// qCodecCaps.getParameterDescriptor(KEY_ER_LTR_MARK_FRAME).getType() -> MediaFormat.TYPE_INTEGER
SupportedValues erLtrMarkFramesValues = qCodecCaps.getParameterRangeInteger(KEY_ER_LTR_MARK_FRAME);
// erLtrMarkFramesValues.getType() -> SupportedValues.TYPE_RANGE

// qCodecCaps.getParameterDescriptor(KEY_ER_LTR_USE_FRAME).getType() -> MediaFormat.TYPE_INTEGER
SupportedValues erLtrmaxFramesValues = qCodecCaps.getParameterRangeInteger(KEY_ER_LTR_USE_FRAME);
// erLtrmaxFramesValues.getType() -> SupportedValues.TYPE_RANGE

// qCodecCaps.getParameterDescriptor(KEY_ER_LTR_RESPONSE).getType() -> MediaFormat.TYPE_INTEGER
SupportedValues erLtrResponseValues = qCodecCaps.getParameterRangeInteger(KEY_ER_LTR_RESPONSE);
// erLtrResponseValues.getType() -> SupportedValues.TYPE_RANGE

3. Apply a Configuration to the Vendor Extension

At this point, you have all the information you need to configure the supported extension. You can now configure static extension properties which remain unchanged throughout the application’s lifecycle, and dynamic extension properties which can be changed at any point.

Begin by checking if the name of the parameter you want to use is in the supported extensions list. For a static extension property, use the MediaFormat’s set methods (e.g., setInteger()) to configure the name/value pairs for the various parameters. Then pass the MediaFormat instance to the configure() method of MediaCodec. The following code sample shows how to set the maximum frame count for HEVC at a resolution of 1280×720:

MediaCodec codec = MediaCodec.createEncoderByType ("video/hevc");
MediaFormat format = MediaFormat.createVideoFormat("video/hevc", 1280, 720);

// check if the extension is supported first, before using it
if (supportedExtensions.indexOf{qti.video.KEY_LTR_MAX_COUNT_FRAME) == -1) {
     // return error, extension KEY_LTR_MAX_COUNT_FRAME is not supported by the Android Device
}
else {
     format.setInteger(qti.video.KEY_LTR_MAX_COUNT_FRAMES, 4);
     codec.configure(format);
}

Setting a value for a dynamic extension property is similar, except that the key/value pair is stored in a Bundle and passed to the MediaCodec instance via its setParameters() method. The following code sample shows how set a QP number for a frame:

codec.start();
if (supportedExtensions.indexOf(qti.video.KEY_FRAME_QP) ! = -1) {
     Bundle frameQP = new Bundle();
     frameQP.setinteger(qti.video.KEY_FRAME_QP, 24);
     codec.setParameters(frameQP);
     codec.queueinputBuffer(...);
}

This article was previously published on proandroiddev.com