TL;DR: Developers have often turned to MediaPlayer for its audio offload capabilities, but found its lack of dynamic streaming support over HTTP, smooth streaming, and persistent caching, pushed them to find an alternative API like ExoPlayer. For over a year, the audio team at Qualcomm Technologies, Inc. (QTI) collaborated with Google, the ExoPlayer team, and YouTube to bring ExoPlayer’s audio power consumption to within 5% of MediaPlayer’s. This article explains how this has been accomplished on Snapdragon Mobile Platforms.
If you build rich media apps, you’ve probably used the Android API’s MediaPlayer class for video playback. Like many developers, you may have found that the API’s lack of dynamic streaming support over HTTP, smooth streaming, and persistent caching pushed you to find an alternative. Recognizing these shortcomings, Google added a custom media player called ExoPlayer to Android API Level 16.
ExoPlayer is now one of the most popular media players for Android developers. As far back as 2016, it was already in use by over 10,000 apps, and today, that list includes household names like Netflix and YouTube. ExoPlayer is currently being migrated within the Android API to the JetPack Media3 module — a collection of media libraries for rich audio and visual experiences.
Despite ExoPlayer’s advantages, MediaPlayer has held the upper hand when it comes to audio offload (i.e., decoding audio downstream from the application processor (AP) such as on the DSP). While enabled by default in MediaPlayer, audio offload support in ExoPlayer was introduced in v2.12 as an experimental feature. In addition, audio offload in ExoPlayer has only been available to devices running Android 10+ due to API limitations, while MediaPlayer supports offload in earlier versions.
Developers who chose ExoPlayer over MediaPlayer for its increased functionality, did so at the cost of increased power consumption for audio (albeit minimal) in several common scenarios:
- Audio-only
- Screen-off music and podcast playback where developers want to ensure audio remains synchronized with non-audio tracks (e.g., video and subtitles) when the screen is off
- High-latency (a few seconds or more)
- Long playback durations of at least one minute
Recognizing the importance of these use cases, Qualcomm Technologies, Inc. set out to improve the power efficiency of ExoPlayer.
Reaching Goals with a Cross-Company, Multi-Team Effort
Given ExoPlayer’s popularity, QTI wanted to bring its power efficiency closer to MediaPlayer’s when offloading audio to the Snapdragon Mobile Platform and its audio digital signal processor (ADSP) found in the Qualcomm Hexagon DSP.
For over a year, the audio team at QTI collaborated with Google, the ExoPlayer team, and YouTube to bring ExoPlayer’s audio power consumption to within 5% of MediaPlayer’s. This cross-company, multi-team effort set out to improve YouTube’s audio power consumption via ExoPlayer on Snapdragon and, eventually, any app built around ExoPlayer through the following goals:
- Enable chip-level DSP
- Increase battery life for the end user, notably during screen-off music playback with offload enabled
- Reduce power consumption for the above scenarios
- Enable non-CPU audio playback management (e.g., playback speed control as described later in this blog). Previously, there was no control over audio samples, and apps had to rely on Audio Framework APIs to control the stream
- Support the Opus codec for audio decoding in offload. Opus is becoming the preferred codec over AAC and MP3 for high-quality audio at lower bit rates. Opus support was added in Android R11 (API level 30)
- Support for offload in gapless playback with Opus and all other support formats in offload mode playback
Increasing Efficiency
To accomplish these goals, QTI explored two main approaches:
- Increasing hardware audio decoding efficiency using less AP time/cycle.
2. Queuing more audio in buffers to reduce the number of AP wakeups.
Previously, encoded audio played via MediaCodecAudioRenderer had to go through MediaCodec and the decoded audio through AudioTrack, as shown in Figure 1 below:
Figure 1 — Comparison of existing ExoPlayer audio rendering via MediaCodec versus audio offload.
This MediaCodec round trip in the passthrough case is unnecessary, and adds significant latency, buffering, and generally increases power usage. This had to be avoided to reach the goal of power efficiency via fast and infrequent AP wakeups.
To bypass MediaCodec, the renderer needs to skip the MediaCodec feed and drain loops. Now, compressed audio will be directly written to the AudioTrack, as shown in Figure 2:
Figure 2 — Top: Existing ExoPlayer audio rendering for normal and passthrough modes; Bottom: Updated ExoPlayer path for passthrough along with new offload mode.
Since the audio path is now the same in both passthrough and offload, passthrough latency and decoding jitter should be improved with this rework.
Implementation Details
ExoPlayer runs its main loop, doSomeWork(), quite frequently. To support an offload mode, ExoPlayer must run as rarely as possible and only when necessary to reduce AP usage. Its only critical work is to fill the AudioTrack buffer to avoid underrun. All other work (e.g., timeline updates, error handling, etc.) is delayed to the next invocation of doSomeWork() when triggered for filling an AudioTrack buffer (approximately every minute).
To schedule doSomeWork() either from an AudioTrack callback or by sleeping just the right amount of time, QTI chose to use StreamEventCallback.onDataRequest. This callback is invoked when the system has consumed some audio, and the app needs to refill the buffer. This method has the advantage of being triggered by an audio server consumption event. Thus ExoPlayer should be scheduled just after some data was consumed, at the earliest time to refill the buffer.
Another advantage is that as ExoPlayer is woken up by a message sent from the AudioServer, the AP only has to wake up once for both the consumption and refilling of the AudioTrack offload buffer.
AudioTrack.setOffloadDelayPadding() can be invoked at any time and will affect the track currently playing until AudioTrack.setOffloadEndOfStream() is invoked. Setting the offload end of stream will transiently put the AudioTrack in STOPPING state until the user space buffers are drained to the DSP. All subsequent calls to AudioTrack.setOffloadDelayPadding() will be attributed to the next track. The ability to invoke AudioTrack.setOffloadDelayPadding() at any time helps cases like playback via Opus, where the padding might not be known until the last samples are extracted and written.
Job Offers
OUR VIDEO RECOMMENDATION
Jobs
As a result, doSomeWork() is run much less frequently in offload mode, and fields like PlaybackInfo.currentPosition are typically updated every minute instead of every 10ms. While this is acceptable for most state updates to be delayed, the player’s current position is used extensively by clients (notably for the UI). Now the app requests the current position with a new API like updateCurrentPosition() that triggers doSomeWork() to update the position and call a new method (e.g., onPositionUpdate()).
QTI also added playback speed control for offload mode. ExoPlayer reads AudioManager parameters when offloadVariableRateSupported is true, to identify if a device supports variable playback rate for offloaded audio. QTI uses this parameter since there is no AudioManager API that provides that information.
Results
So far, the results of this design have been impressive. When audio streaming in the YouTube Music APK is offloaded to the ADSP, there is an approximate 13% decrease in milliamperes (mA) consumed compared to non-offload, as shown in Table 1:
Table 1 — Approximate energy savings, in mA, when comparing audio non-offload to offload for YouTube audio streaming with ExoPlayer.
Table 2 below shows the battery power measurements for playback of an on-device MP3 file using compressed offload versus deep buffer modes on test devices:
Table 2 — Comparison of MP3 playback power consumption using offload versus deep buffer on test devices powered by Snapdragon.
Conclusion
The collaboration between QTI, Google, YouTube, and the ExoPlayer teams has led the effort to make audio offload an official feature of ExoPlayer. This effort brings audio playback power efficiency close to that of MediaPlayer, so developers can more confidently choose ExoPlayer for its additional capabilities.
Audio offload is enabled for AAC format in the YouTube Music APK available in Google Play. It’s currently available to YouTube (Music) premium account holders, and will be rolled out more broadly after the team performs additional confirmations and verifications.
You can find audio offload support in ExoPlayer2’s standalone GitHub (dev-v2 branch) and ExoPlayer2’s AndroidX GitHub.
To learn more about developing Android apps for Snapdragon, visit the Android on Snapdragon Developer Portal.
Author: Banajit Goswami
Banajit Goswami is a Senior Staff Engineer at Qualcomm Innovation Center, Inc. In this role, he provides technical leadership, and drives strategy and execution of new audio technologies for Snapdragon powered Handheld devices
Snapdragon and Qualcomm branded products are products of Qualcomm Technologies, Inc. and/or its subsidiaries.
This article was previously published on proandroiddev.com
1 Comment. Leave new
Very Informative. Great Article