This continuing series of articles will explain how to generate a code coverage report for Android instrumentation tests running on Firebase Test Lab.
Requirements for Part 2
In Part 1 of this series, we saw how to:
- Produce XML & HTML reports for code coverage of on-device tests
- Run on API 28, API 29, and API 30 devices in Firebase Test Lab
- Support Android applications targeting API 28, API 29, and API 30
In Part 2, we’ll extend this with:
- Supporting Android Test Orchestrator
- Combining results with off-device unit tests
- Supporting multi-module applications
Supporting Android Test Orchestrator
Android Test Orchestrator is a tool that allows us to run on-device tests in independent processes from one another, so that a crashing or misbehaving test cannot pollute the results of other tests.
To better illustrate this, we’ll first give our classes a few more methods, and then add another test case so we can see the Orchestrator at work.
| public class JavaClass { | |
| int function1() { | |
| return 2 + 2; | |
| } | |
| int function2() { | |
| return 3 + 3; | |
| } | |
| int function3() { | |
| return 4 + 4; | |
| } | |
| int function4() { | |
| return 5 + 5; | |
| } | |
| } |
| class KotlinClass { | |
| fun function1(): Int = | |
| 2 + 2 | |
| fun function2(): Int = | |
| 3 + 3 | |
| fun function3(): Int = | |
| 4 + 4 | |
| fun function4(): Int = | |
| 5 + 5 | |
| } |
| @RunWith(AndroidJUnit4::class) | |
| class Tests { | |
| @Test | |
| fun basicTests() { | |
| assertEquals(4, JavaClass().function1()) | |
| assertEquals(4, KotlinClass().function1()) | |
| } | |
| @Test | |
| fun basicTests2() { | |
| assertEquals(6, JavaClass().function2()) | |
| assertEquals(6, KotlinClass().function2()) | |
| } | |
| } |
To enable Orchestrator for our tests, we need to make a few changes to our module-level build.gradle
| android { | |
| ... | |
| defaultConfig { | |
| ... | |
| // The following argument makes the Android Test Orchestrator run its | |
| // "pm clear" command after each test invocation. This command ensures | |
| // that the app's state is completely cleared between tests. | |
| testInstrumentationRunnerArguments clearPackageData: 'true' | |
| } | |
| ... | |
| testOptions { | |
| execution 'ANDROIDX_TEST_ORCHESTRATOR' | |
| } | |
| } | |
| dependencies { | |
| ... | |
| androidTestImplementation 'androidx.test:runner:1.4.0' | |
| androidTestUtil 'androidx.test:orchestrator:1.4.0' | |
| } |
Finally, we’ll build the app (just like in the previous article)
./gradlew -Pcoverage clean assembleDebug assembleDebugAndroidTest
And execute our tests with our gcloud command by taking the same one from the previous article and adding the --use-orchestrator option.
gcloud firebase test android run \ --type instrumentation \ --use-orchestrator \ --no-performance-metrics \ --no-record-video \ --app app/build/outputs/apk/debug/app-debug.apk \ --test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk \ --device model=Pixel2,version=29,locale=en,orientation=portrait \ --environment-variables coverage=true,coverageFile=/sdcard/Download/coverage.ec \ --directories-to-pull /sdcard/Download
Then download the resulting coverage file
mkdir app/build/outputs/coverage gsutil cp gs://<project-id>/<timestamp>/Pixel2-30-en-portrait/artifacts/coverage.ec app/build/outputs/coverage
And generate a report
./gradlew jacocoReport
Opening the report, we see
Wait… why are we only seeing the coverage results from one of our tests?
The problem is that each test in running in its own process, and each process is writing its coverage file to the same location specified in the coverageFile parameter passed to the gcloud command.
So we’ll replace coverageFile with coverageFilePath (note: coverageFilePath must end in / but directories-to-pull must not) so that each individual coverage file will be written to that directory. Our updated gcloud command is now
gcloud firebase test android run \ --type instrumentation \ --use-orchestrator \ --no-performance-metrics \ --no-record-video \ --app app/build/outputs/apk/debug/app-debug.apk \ --test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk \ --device model=Pixel2,version=29,locale=en,orientation=portrait \ --environment-variables coverage=true,coverageFilePath=/sdcard/Download/ \ --directories-to-pull /sdcard/Download
Then when we look in the artifacts folder we see each coverage file stored separately.
We’ll need to update our download command to download all .ec files:
mkdir app/build/outputs/coverage gsutil cp gs://<project-id>/<timestamp>/Pixel2-30-en-portrait/artifacts/*.ec app/build/outputs/coverage
But our report generation task can stay the same, since it is already looking for all .ec files. Running gradlew jacocoReport, we now get the coverage that we’re expecting:
Generating .exec files for off-device unit tests
Adding off-device unit tests to the mix is comparatively simple. First we’ll add an off-device test to app/src/test/java/com/github128/coverage1/OffDeviceTests.kt:
| class OffDeviceTests { | |
| @Test | |
| fun basicTests3() { | |
| Assert.assertEquals(8, JavaClass().function3()) | |
| Assert.assertEquals(8, KotlinClass().function3()) | |
| } | |
| } |
Job Offers
OUR VIDEO RECOMMENDATION
Jobs
We can easily run this from the command line:
./gradlew testDebugUnitTest
And a coverage file is generated in app/build/jacoco/testDebugUnitTest.exec.
A warning about testCoverageEnabled in unit tests
If you pass the -Pcoverage option (thereby setting testCoverageEnabled to true, as described in the previous article) then this will generate the .exec file in a different location
(app/build/outputs/unit_test_code_coverage/debugUnitTest
/testDebugUnitTest.exec)
but more importantly will not create the file properly for library modules, causing issues once we get to multi-module Android projects. (this issue is tracked at https://issuetracker.google.com/issues/210500600)
Combining .ec and .exec files
In any case, now we just need to tweak the getExecutionData().setFrom in our jacocoReport job in our module-level build.gradle to pull coverage files from this new location.
getExecutionData().setFrom(
fileTree(dir: "${buildDir}/outputs/coverage",
includes: ['*.ec']),
fileTree(dir: "${buildDir}/jacoco",
includes: ['*.exec'])
)
Now our coverage report will include the calls from our unit tests combined with the calls from our Firebase Test Lab tests. Our entire execution is now
./gradlew -Pcoverage clean assembleDebug assembleDebugAndroidTest gcloud firebase test android run \ --type instrumentation \ --use-orchestrator \ --no-performance-metrics \ --no-record-video \ --app app/build/outputs/apk/debug/app-debug.apk \ --test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk \ --device model=Pixel2,version=29,locale=en,orientation=portrait \ --environment-variables coverage=true,coverageFilePath=/sdcard/Download/ \ --directories-to-pull /sdcard/Download mkdir app/build/outputs/coverage gsutil cp gs://<project-id>/<timestamp>/Pixel2-30-en-portrait/artifacts/*.ec app/build/outputs/coverage ./gradle testDebugUnitTest ./gradlew jacocoReport
Which will create the desired report of
Multi-module applications: per-module report
There are two parts to supporting code coverage for a multi-module application: coverage reports for each individual module as well as a unified report combining the coverage of all modules.
First we’ll add a second module to our application. Like our application module, this new library module will include a pair of classes with some functions as well as both on-device and off-device tests.
| public class LibraryJavaClass { | |
| int function1() { | |
| return 2 + 2; | |
| } | |
| int function2() { | |
| return 3 + 3; | |
| } | |
| int function3() { | |
| return 4 + 4; | |
| } | |
| int function4() { | |
| return 5 + 5; | |
| } | |
| } |
| class LibraryKotlinClass { | |
| fun function1(): Int = | |
| 2 + 2 | |
| fun function2(): Int = | |
| 3 + 3 | |
| fun function3(): Int = | |
| 4 + 4 | |
| fun function4(): Int = | |
| 5 + 5 | |
| } |
| @RunWith(AndroidJUnit4::class) | |
| class LibraryTests { | |
| @Test | |
| fun basicTests() { | |
| assertEquals(4, LibraryJavaClass().function1()) | |
| assertEquals(4, LibraryKotlinClass().function1()) | |
| } | |
| @Test | |
| fun basicTests2() { | |
| assertEquals(6, LibraryJavaClass().function2()) | |
| assertEquals(6, LibraryKotlinClass().function2()) | |
| } | |
| } |
| class LibraryOffDeviceTests { | |
| @Test | |
| fun basicTests3() { | |
| assertEquals(8, LibraryJavaClass().function3()) | |
| assertEquals(8, LibraryKotlinClass().function3()) | |
| } | |
| } |
One approach is to essentially replicate what we did before, adding a dependency on the jacoco plugin and a new task to the new module’s build.gradle. A better solution, however, is to refactor the plugin dependency and task into a separate gradle script and then import it into the build.gradle for both our primary module and our new module. The separate script would then be
| apply plugin: 'jacoco' | |
| jacoco { | |
| toolVersion '0.8.7' | |
| } | |
| task jacocoReport(type: JacocoReport) { | |
| reports { | |
| xml.enabled = true | |
| html.enabled = true | |
| } | |
| getSourceDirectories().setFrom("${project.projectDir}/src/main/java") | |
| def fileFilter = ['**/R.class', '**/R$*.class', '**/BuildConfig.*', '**/Manifest*.*', '**/*Test*.*', 'android/**/*.*'] | |
| getClassDirectories().setFrom( | |
| fileTree(dir: "${buildDir}/intermediates/javac/debug", excludes: fileFilter), | |
| fileTree(dir: "${buildDir}/tmp/kotlin-classes/debug", excludes: fileFilter)) | |
| getExecutionData().setFrom( | |
| fileTree(dir: "${buildDir}/outputs/code_coverage", includes: ['*.ec']), | |
| fileTree(dir: "${buildDir}/jacoco", includes: ['*.exec'])) | |
| } |
Then we just need to modify our new library module’s build.gradle by adding
apply from: '../module-jacoco.gradle'
as well as our standard
buildTypes {
debug {
testCoverageEnabled (project.hasProperty('coverage'))
}
}
Similarly, we can edit our original module’s build.gradle by removing the dependency on the jacoco plugin, deleting the jacocoReport job entirely, and adding in the dependency on module-jacoco.gradle.
At this point, we can run the jacocoReport task in each module (after either running the unit tests or downloading coverage reports from Firebase Test Lab). Remember that when running the Firebase Test Lab tests the APK passed to gcloud with the --app parameter should still be the APK generated from the application module. (i.e. app/build/outputs/apk/debug/app-debug.apk)
cd app ../gradlew -Pcoverage clean assembleDebug assembleDebugAndroidTest <gcloud to run tests, gsutil to download *.ec files> ../gradlew testDebugUnitTest ../gradlew jacocoReport <view build/reports/jacoco/jacocoReport/html/index.html> cd ../library ../gradlew -Pcoverage clean assembleDebug assembleDebugAndroidTest <gcloud to run tests, gsutil to download *.ec files> ../gradlew testDebugUnitTest ../gradlew jacocoReport <view build/reports/jacoco/jacocoReport/html/index.html>
And the resulting code coverage reports for each module will accurately reflect the coverage of the unit tests combined with the coverage of the on-device tests that ran in Firebase Test Lab.
Multi-module applications: unified report
We could put all of this into the project-level build.gradle, but as with the individual modules’ gradle scripts, it’s a bit cleaner to put it into its own gradle script and then import it into the project-level build.gradle. We can follow the collect/flatten pattern to build up the desired list of files or directories. The project-level separate script will look like
| apply plugin: 'jacoco' | |
| jacoco { | |
| toolVersion '0.8.7' | |
| } | |
| task jacocoUnifiedReport(type: JacocoReport) { | |
| reports { | |
| xml.enabled = true | |
| html.enabled = true | |
| } | |
| def sourceDirectories = subprojects.collect { subproject -> | |
| "${subproject.projectDir}/src/main/java" | |
| } | |
| def classDirectories = subprojects.collect { subproject -> | |
| ["${subproject.buildDir}/intermediates/javac/debug", "${subproject.buildDir}/tmp/kotlin-classes/debug"] | |
| }.flatten() | |
| def classFiles = classDirectories.collect { directory -> | |
| def excludesFilter = ['**/R.class', '**/R$*.class', '**/BuildConfig.*', '**/Manifest*.*', '**/*Test*.*', 'android/**/*.*'] | |
| fileTree(dir: file(directory), excludes: excludesFilter) | |
| } | |
| def executionDirectories = subprojects.collect { subproject -> | |
| ["${subproject.buildDir}/outputs/code_coverage", "${subproject.buildDir}/jacoco"] | |
| }.flatten() | |
| def executionFiles = executionDirectories.collect { directory -> | |
| def includesFilter = ['*.ec', '*.exec'] | |
| fileTree(dir: file(directory), includes: includesFilter) | |
| } | |
| getSourceDirectories().setFrom(sourceDirectories) | |
| getClassDirectories().setFrom(classFiles) | |
| getExecutionData().setFrom(executionFiles) | |
| } |
And we’ll just need to add a
apply from: 'project-jacoco.gradle'
to our the build.gradle in the root of our project. Once this is in place, after going through the steps mentioned above to generate the .exec files for off-device tests and download the .ec files for Firebase Test Lab tests, we just need to go to the root of the project and run
./gradlew jacocoUnifiedReport
and we’ll then see a report for the entire project’s coverage in build/reports/jacoco/jacocoUnifiedReport/html/index.html.
TL;DR
We’ve now achieved the requirements for this article. Looking back, these were the steps we needed to take:
- Make some small changes to the build.gradle for each module to enable Android Test Orchestrator, and then pass
--use-orchestratortogcloudto instruct Firebase Test Lab to use it. - Because Android Test Orchestrator generates multiple .ec files, we’ll also need to change the
coverageFileparameter tocoverageFilePathand then change our download step to pull*.ecrather than justcoverage.ec. - To generate the coverage files for off-device unit tests, all we need to do is make sure that the jacoco plugin is included (already done) and that we don’t pass -Pcoverage when running the actual unit tests. (because of https://issuetracker.google.com/issues/210500600)
- To cleanly support generating per-module reports for multi-module applications, we refactored the
jacocoReporttask out of each module’s build.gradle and into a commonmodule-jacoco.gradlefile. - To support a project-wide unified coverage report, we added a new
jacocoUnifiedReporttask to aproject-jacoco.gradlefile that is included by the project-level build.gradle.
You can see a solution that combines all of these changes by looking at https://github.com/Aidan128/CoverageExample1 and checking out the git tag part_two.
Next time
Now we have the ability to generate code coverage reports for unit tests and Firebase Test Lab tests, as well as to combine them across the entire project. This process is rather cumbersome, however, and requires many steps. In the third and final article in this series, we’ll look at how this process can be scripted, integrated with Firebase Test Lab automation tools like flank, and finally integrated with CI/CD systems such as gitlab.
Special thanks
Saravana Thiyargaraj wrote a great article (and built a great example on github) that were very useful. Thank you.
