This series of articles will explain how to generate a code coverage report for Android instrumentation tests running on Firebase Test Lab.
Final goal
We’ll build up a step at a time, with a final goal of an implementation that will
- Produce XML & HTML reports for code coverage of on-device tests
- Run on API 28, API 29, and API 30 devices in Firebase Test Lab
- Support Android applications targeting API 28, API 29, and API 30
- Support Android Test Orchestrator
- Combine results with off-device unit tests
- Support multi-module applications
- Integrate with flank
- Use scripts to run in a CI/CD pipeline
Requirements for Part 1
Later articles will build up to satisfy all the requirements mentioned, but for now we’ll concentrate on these requirements:
- Produce XML & HTML reports for test coverage
- Run on API 28, API 29, and API 30 devices in Firebase Test Lab
- Support Android applications targeting API 28, API 29, and API 30
Getting started with API 28
We’ll start with an extremely simple API 28 application, created by starting a new Android project in Android Studio Arctic Fox without any activities, and then editing the project-level build.gradle to target API 28. We’ll add a Java class and a Kotlin class, each with two methods, and add an instrumentation test that will call one method on each class.
public class JavaClass { | |
int function1() { | |
return 2 + 2; | |
} | |
int function2() { | |
return 3 + 3; | |
} | |
} |
class KotlinClass { | |
fun function1(): Int = | |
2 + 2 | |
fun function2(): Int = | |
3 + 3 | |
} |
@RunWith(AndroidJUnit4::class) | |
class Tests { | |
@Test | |
fun basicTests() { | |
assertEquals(4, JavaClass().function1()) | |
assertEquals(4, KotlinClass().function1()) | |
} | |
} |
Building and running
You can easily build this with
./gradlew clean assembleDebug assembleDebugAndroidTest
This article assumes that you already have Firebase Test Lab set up for command line execution, if not then follow the directions at https://firebase.google.com/docs/test-lab/android/command-line
Once you’re ready to use the Google Cloud command line tool, it’s a simple matter of running the right command. We’ll base this on the example at https://firebase.google.com/docs/test-lab/android/command-line#running_your_instrumentation_tests but tweak the parameters to write the coverage file to /sdcard/Download/coverage.ec
instead of /sdcard/coverage.ec
, for reasons that will become clear later.
gcloud firebase test android run \ --type instrumentation \ --no-performance-metrics \ --no-record-video \ --app app/build/outputs/apk/debug/app-debug.apk \ --test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk \ --device model=Pixel2,version=28,locale=en,orientation=portrait \ --environment-variables coverage=true,coverageFile=/sdcard/Download/coverage.ec \ --directories-to-pull /sdcard/Download
If you watch the output, you’ll see a line that looks like
Raw results will be stored in your GCS bucket at [https://console.developers.google.com/storage/browser/test-lab-<project-id>/<timestamp>/]
If you follow the link you’ll come to a page that looks like
And if you click the link for the device we used (Pixel2–28-en-portrait) you’ll find yourself on a page that looks like
And then if you click the link for instrumentation.results
you’ll come to a page where you can download the contents of the file. It should look something like
And if you click the “download” link and scroll to the bottom, you’ll see the results of our attempt to generate a code coverage report…
Error: Failed to generate Emma/JaCoCo coverage. Is Emma/JaCoCo jar on classpath?
Oh.
Enabling code coverage
Looks like we haven’t enabled code coverage on our build. We’ll need to turn this on by adding testCoverageEnabled
within the module-level build.gradle. However, there is some performance impact of building with code coverage enabled, so we’ll gate it behind an optional flag.
buildTypes { debug { testCoverageEnabled (project.hasProperty('coverage')) } }
We can then do a clean build by passing the –Pcoverage
parameter like this:
./gradlew -Pcoverage clean assembleDebug assembleDebugAndroidTest
After building and running the gcloud command again, once we open instrumentation.results
again, we see
Error: Failed to generate Emma/JaCoCo coverage.
We’ll need to open logcat to see exactly what’s going on, so we’ll click on the link for logcat in the same place we found the instrumentation.results link, click download, and search through the logcat spew for “jacoco”. There we find
E/CoverageListener(6620): Failed to generate Emma/JaCoCo coverage. E/CoverageListener(6620): java.lang.reflect.InvocationTargetException E/CoverageListener(6620): at java.lang.reflect.Method.invoke(Native Method) E/CoverageListener(6620): at androidx.test.internal.runner.listener.CoverageListener.generateCoverageReport(CoverageListener.java:101) E/CoverageListener(6620): at androidx.test.internal.runner.listener.CoverageListener.instrumentationRunFinished(CoverageListener.java:70) E/CoverageListener(6620): at androidx.test.internal.runner.TestExecutor.reportRunEnded(TestExecutor.java:92) E/CoverageListener(6620): at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:65) E/CoverageListener(6620): at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:395) E/CoverageListener(6620): at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2145) E/CoverageListener(6620): Caused by: java.io.FileNotFoundException: /sdcard/Download/coverage.ec (Permission denied)
The issue is that jacoco agent is trying to write the coverage data to /sdcard/Download/coverage.ec
, but it doesn’t have permission to write external storage.
Granting permission
The solution is to grant the WRITE_EXTERNAL_STORAGE permission in AndroidManifest.xml. (we’ll create a new AndroidManifest.xml
in app/src/debug/AndroidManifest.xml
so that we’re not affecting the production version of our application)
<?xml version="1.0" encoding="utf-8"?> | |
<manifest xmlns:android="http://schemas.android.com/apk/res/android" | |
package="com.github.aidan128.coverage1"> | |
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/> | |
</manifest> |
Job Offers
One final time, we’ll run gcloud, and now instrumentation.results shows that we’ve succeeded!
Generated code coverage data to /sdcard/Download/coverage.ec
If we look back in Google Cloud Storage (in the same location we saw the instrumentation.results and logcat links) there is now an “artifacts” folder that contains coverage.ec
, at last.
Supporting Android 10 (API 29)
To update our application to target API 29, we just need to change the project-level build.gradle’s compileSdk and targetSdk settings.
android { compileSdk 29 defaultConfig { targetSdk 29 ... } ... }
Running the same gcloud command as before, we see that we work just fine on an API 28 device. coverage.ec is generated and copied to the artifacts folder, just like before.
But we also want to check that we can run on an API 29 device in Firebase Test Lab. So we’ll change the “version” portion of the --device
parameter:
gcloud firebase test android run \ --type instrumentation \ --no-performance-metrics \ --no-record-video \ --app app/build/outputs/apk/debug/app-debug.apk \ --test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk \ --device model=Pixel2,version=29,locale=en,orientation=portrait \ --environment-variables coverage=true,coverageFile=/sdcard/Download/coverage.ec \ --directories-to-pull /sdcard/Download
However, when we run the gcloud command now instrumentation.results
reports
Error: Failed to generate Emma/JaCoCo coverage
and logcat contains
E/CoverageListener(10124): Failed to generate Emma/JaCoCo coverage. E/CoverageListener(10124): java.lang.reflect.InvocationTargetException E/CoverageListener(10124): at java.lang.reflect.Method.invoke(Native Method) E/CoverageListener(10124): at androidx.test.internal.runner.listener.CoverageListener.generateCoverageReport(CoverageListener.java:101) E/CoverageListener(10124): at androidx.test.internal.runner.listener.CoverageListener.instrumentationRunFinished(CoverageListener.java:70) E/CoverageListener(10124): at androidx.test.internal.runner.TestExecutor.reportRunEnded(TestExecutor.java:92) E/CoverageListener(10124): at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:65) E/CoverageListener(10124): at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:395) E/CoverageListener(10124): at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2189) E/CoverageListener(10124): Caused by: java.io.FileNotFoundException: /sdcard/Download/coverage.ec: open failed: EACCES (Permission denied)
The problem is that API 29 introduces new restrictions on where applications can write to. (“scoped storage”) Luckily, we can easily opt out by adding requestLegacyExternalStorage to app/src/debug/AndroidManifest.xml.
<?xml version="1.0" encoding="utf-8"?> | |
<manifest xmlns:android="http://schemas.android.com/apk/res/android" | |
package="com.github.aidan128.coverage1"> | |
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" /> | |
<application | |
android:requestLegacyExternalStorage="true" | |
/> | |
</manifest> |
Running the gcloud command again, coverage.ec is generated and copied to the artifacts folder, as we saw before running on an API 28 device.
Supporting Android 11 (API 30)
To update our application to target API 30, we again change the project-level build.gradle’s compileSdk and targetSdk settings.
android { compileSdk 30 defaultConfig { targetSdk 30 ... } ... }
Running against API 28 and API 29 devices in Firebase Test Lab (by modifying the version portion of the --device
parameter of our gcloud command) works just fine.
And amazingly, even when we run against API 30 devices, everything just works, even though neither WRITE_EXTERNAL_STORAGE nor requestLegacyExternalStorage are supported on Android 11. What’s going on?
This is where the switch to /sdcard/Download
pays off; on Android 11 devices /sdcard/Download
is accessible to everyone. (see https://medium.com/androiddevelopers/scope-storage-myths-ca6a97d7ff37)
And while we don’t strictly need to make any more changes, for the purposes of good code hygiene we shouldn’t request WRITE_EXTERNAL_STORAGE permission on Android 11 devices, since that permission doesn’t do anything there. We can add maxSdkVersion to the uses-permission block, like this:
<?xml version="1.0" encoding="utf-8"?> | |
<manifest xmlns:android="http://schemas.android.com/apk/res/android" | |
package="com.github.aidan128.coverage1"> | |
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" | |
android:maxSdkVersion="29"/> | |
<application | |
android:requestLegacyExternalStorage="true" | |
/> | |
</manifest> |
Excellent, so now we can collect code coverage numbers on test devices whether they’re running Android Pie (API 28), Android 10 (API 29) or Android 11. (API 30)
Now let’s move on to actually using the coverage data.
Downloading the .ec file
The gsutil
utility allows us to easily copy files from gs://
links, and we can use pieces of the GCS results bucket’s URL to compute that link.
For example, if the URL for the GCS results bucket is
https://console.developers.google.com/storage/browser/<project-id>/<timestamp>
then the gs:// link for the coverage.ec file (for our chosen device) is
gs://<project-id>/<timestamp>/Pixel2-30-en-portrait/artifacts/coverage.ec
And we can download it by issuing the gsutil command. We’ll put it into app/build/outputs/coverage for now.
mkdir app/build/outputs/coverage gsutil cp gs://<project-id>/<timestamp>/Pixel2-30-en-portrait/artifacts/coverage.ec app/build/outputs/coverage
Building reports from external .ec files
The .ec file is not fit for human consumption, however. Since our goal is to build XML and HTML reports to view the coverage information, we’ll need to use JaCoCo’s report generating activities. Unlike the support for instrumenting binaries (which is built into gradle as long as you pass testCodeCoverage
), if we want to generate reports for arbitrary .ec files obtained externally, we’ll need to explicitly define a dependency on JaCoCo in our project-level build.gradle:
dependencies { | |
classpath "org.jacoco:org.jacoco.core:0.8.7" | |
} |
In our module-level build.gradle, we’ll also need to add a dependency on the JaCoCo plugin as well as add a custom task to build the report.
plugins { | |
id 'jacoco' | |
} | |
... | |
task jacocoReport(type: JacocoReport) { | |
group "Coverage" | |
description "Generate XML/HTML code coverage reports for coverage.ec" | |
reports { | |
xml.enabled = true | |
html.enabled = true | |
} | |
getSourceDirectories().setFrom("${project.projectDir}/src/main/java") | |
def fileFilter = ['**/R.class', '**/R$*.class', '**/BuildConfig.*', '**/Manifest*.*', '**/*Test*.*', 'android/**/*.*'] | |
getClassDirectories().setFrom( | |
fileTree(dir: "${buildDir}/intermediates/javac/debug", excludes: fileFilter), | |
fileTree(dir: "${buildDir}/tmp/kotlin-classes/debug", excludes: fileFilter)) | |
getExecutionData().setFrom( | |
fileTree(dir: "${buildDir}/outputs/code_coverage", includes: ['*.ec'])) | |
} |
Once all this is in place, we can generate the report by calling
./gradlew jacocoReport
and then can open the report found at app/build/reports/jacoco/jacocoReport/html/index.html
TL;DR
And at this point we’ve achieved the requirements we targeted for this article. Looking back, these were the required steps to get here:
- Set
testCoverageEnabled
to true for debug builds to enable code coverage (but only when -Pcoverage is passed) - Add the
WRITE_EXTERNAL_STORAGE
permission for debug builds (with the appropriate maxSdkVersion) to allowcoverage.ec
to be written on pre-Android 11 devices. - Add requestLegacyExternalStorage for debug builds to allow
coverage.ec
to be written on Android 10 devices. - Use
gsutil
to downloadcoverage.ec
from the results bucket - Add a dependency on jacoco to the project-level and app-level
build.gradle
files. - Add a new gradle task to generate reports from arbitrary .ec files.
You can see a solution that combines all of these changes by looking at https://github.com/Aidan128/CoverageExample1 and checking out the git tag part_one
.
Next time
This article achieved the basic requirements that we targeted for this article, but we still have a long way to go. In Part Two we’ll look at Android Test Orchestrator, multi-module projects, and combining Firebase Test Lab results with off-device unit tests.
Special thanks
Many thanks to the Firebase engineers in the #testlab Slack channel. (https://firebase.community/) Without them I never would have solved the permissions issues that came up in Android 11.
In addition, a number of people have written on this topic in the past, and their articles helped me a lot in my investigations. I thank them.