Blog Infos
Author
Published
Topics
,
Author
Published
Requirements for Part 2

In Part 1 of this series, we saw how to:

  • Run on API 28, API 29, and API 30 devices in Firebase Test Lab
  • Support Android applications targeting API 28, API 29, and API 30
  • Combining results with off-device unit tests
  • Supporting multi-module applications
Supporting Android Test Orchestrator

Android Test Orchestrator is a tool that allows us to run on-device tests in independent processes from one another, so that a crashing or misbehaving test cannot pollute the results of other tests.

public class JavaClass {
int function1() {
return 2 + 2;
}
int function2() {
return 3 + 3;
}
int function3() {
return 4 + 4;
}
int function4() {
return 5 + 5;
}
}
view raw JavaClass.java hosted with ❤ by GitHub
class KotlinClass {
fun function1(): Int =
2 + 2
fun function2(): Int =
3 + 3
fun function3(): Int =
4 + 4
fun function4(): Int =
5 + 5
}
view raw KotlinClass.kt hosted with ❤ by GitHub
@RunWith(AndroidJUnit4::class)
class Tests {
@Test
fun basicTests() {
assertEquals(4, JavaClass().function1())
assertEquals(4, KotlinClass().function1())
}
@Test
fun basicTests2() {
assertEquals(6, JavaClass().function2())
assertEquals(6, KotlinClass().function2())
}
}
view raw Tests.kt hosted with ❤ by GitHub

To enable Orchestrator for our tests, we need to make a few changes to our module-level build.gradle

android {
...
defaultConfig {
...
// The following argument makes the Android Test Orchestrator run its
// "pm clear" command after each test invocation. This command ensures
// that the app's state is completely cleared between tests.
testInstrumentationRunnerArguments clearPackageData: 'true'
}
...
testOptions {
execution 'ANDROIDX_TEST_ORCHESTRATOR'
}
}
dependencies {
...
androidTestImplementation 'androidx.test:runner:1.4.0'
androidTestUtil 'androidx.test:orchestrator:1.4.0'
}
./gradlew -Pcoverage clean assembleDebug assembleDebugAndroidTest

And execute our tests with our gcloud command by taking the same one from the previous article and adding the --use-orchestrator option.

gcloud firebase test android run \
--type instrumentation \
--use-orchestrator \
--no-performance-metrics \
--no-record-video \
--app app/build/outputs/apk/debug/app-debug.apk \
--test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk \
--device model=Pixel2,version=29,locale=en,orientation=portrait \
--environment-variables coverage=true,coverageFile=/sdcard/Download/coverage.ec \
--directories-to-pull /sdcard/Download

Then download the resulting coverage file

mkdir app/build/outputs/coverage
gsutil cp gs://<project-id>/<timestamp>/Pixel2-30-en-portrait/artifacts/coverage.ec app/build/outputs/coverage

And generate a report

./gradlew jacocoReport

Opening the report, we see

 

gcloud firebase test android run \
--type instrumentation \
--use-orchestrator \
--no-performance-metrics \
--no-record-video \
--app app/build/outputs/apk/debug/app-debug.apk \
--test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk \
--device model=Pixel2,version=29,locale=en,orientation=portrait \
--environment-variables coverage=true,coverageFilePath=/sdcard/Download/ \
--directories-to-pull /sdcard/Download

Then when we look in the artifacts folder we see each coverage file stored separately.

mkdir app/build/outputs/coverage
gsutil cp gs://<project-id>/<timestamp>/Pixel2-30-en-portrait/artifacts/*.ec app/build/outputs/coverage

But our report generation task can stay the same, since it is already looking for all .ec files. Running gradlew jacocoReport, we now get the coverage that we’re expecting:

 

 

Generating .exec files for off-device unit tests

Adding off-device unit tests to the mix is comparatively simple. First we’ll add an off-device test to app/src/test/java/com/github128/coverage1/OffDeviceTests.kt:

class OffDeviceTests {
@Test
fun basicTests3() {
Assert.assertEquals(8, JavaClass().function3())
Assert.assertEquals(8, KotlinClass().function3())
}
}

Job Offers

Job Offers


    Kotlin Multiplatform Mobile Developer

    Touchlab
    Remote
    • Full Time
    apply now

    Sr. Software Development Engineer, Last Mile Driver Assistance Technology

    Amazon
    Berlin
    • Full Time
    apply now

OUR VIDEO RECOMMENDATION

, ,

Treasure.map(): Functional Programming in Kotlin

What is a monad? Do I need a PhD to write in functional style? Any hidden gems that will improve my developer experience? In this talk, we’ll cover the core concepts of functional programming, discuss…
Watch Video

Treasure.map(): Functional Programming in Kotlin

KARIN-ALEKSANDRA MONOID
Senior Software Engineer

Treasure.map(): Functional Programming in Kotlin

KARIN-ALEKSANDRA M ...
Senior Software Engi ...

Treasure.map(): Functional Programming in Kotlin

KARIN-ALEKSANDRA ...
Senior Software Engineer

Jobs

A warning about testCoverageEnabled in unit tests

If you pass the -Pcoverage option (thereby setting testCoverageEnabled to true, as described in the previous article) then this will generate the .exec file in a different location

(app/build/outputs/unit_test_code_coverage/debugUnitTest

/testDebugUnitTest.exec)
but more importantly will not create the file properly for library modules, causing issues once we get to multi-module Android projects. (this issue is tracked at https://issuetracker.google.com/issues/210500600)

Combining .ec and .exec files

In any case, now we just need to tweak the getExecutionData().setFrom in our jacocoReport job in our module-level build.gradle to pull coverage files from this new location.

getExecutionData().setFrom(
  fileTree(dir: "${buildDir}/outputs/coverage", 
           includes: ['*.ec']),
  fileTree(dir: "${buildDir}/jacoco", 
           includes: ['*.exec'])
)
./gradlew -Pcoverage clean assembleDebug assembleDebugAndroidTest
gcloud firebase test android run \
--type instrumentation \
--use-orchestrator \
--no-performance-metrics \
--no-record-video \
--app app/build/outputs/apk/debug/app-debug.apk \
--test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk \
--device model=Pixel2,version=29,locale=en,orientation=portrait \
--environment-variables coverage=true,coverageFilePath=/sdcard/Download/ \
--directories-to-pull /sdcard/Download
mkdir app/build/outputs/coverage
gsutil cp gs://<project-id>/<timestamp>/Pixel2-30-en-portrait/artifacts/*.ec app/build/outputs/coverage
./gradle testDebugUnitTest
./gradlew jacocoReport

Which will create the desired report of

 

 

Multi-module applications: per-module report

There are two parts to supporting code coverage for a multi-module application: coverage reports for each individual module as well as a unified report combining the coverage of all modules.

public class LibraryJavaClass {
int function1() {
return 2 + 2;
}
int function2() {
return 3 + 3;
}
int function3() {
return 4 + 4;
}
int function4() {
return 5 + 5;
}
}
class LibraryKotlinClass {
fun function1(): Int =
2 + 2
fun function2(): Int =
3 + 3
fun function3(): Int =
4 + 4
fun function4(): Int =
5 + 5
}
@RunWith(AndroidJUnit4::class)
class LibraryTests {
@Test
fun basicTests() {
assertEquals(4, LibraryJavaClass().function1())
assertEquals(4, LibraryKotlinClass().function1())
}
@Test
fun basicTests2() {
assertEquals(6, LibraryJavaClass().function2())
assertEquals(6, LibraryKotlinClass().function2())
}
}
view raw LibraryTests.kt hosted with ❤ by GitHub
class LibraryOffDeviceTests {
@Test
fun basicTests3() {
assertEquals(8, LibraryJavaClass().function3())
assertEquals(8, LibraryKotlinClass().function3())
}
}

One approach is to essentially replicate what we did before, adding a dependency on the jacoco plugin and a new task to the new module’s build.gradle. A better solution, however, is to refactor the plugin dependency and task into a separate gradle script and then import it into the build.gradle for both our primary module and our new module. The separate script would then be

apply plugin: 'jacoco'
jacoco {
toolVersion '0.8.7'
}
task jacocoReport(type: JacocoReport) {
reports {
xml.enabled = true
html.enabled = true
}
getSourceDirectories().setFrom("${project.projectDir}/src/main/java")
def fileFilter = ['**/R.class', '**/R$*.class', '**/BuildConfig.*', '**/Manifest*.*', '**/*Test*.*', 'android/**/*.*']
getClassDirectories().setFrom(
fileTree(dir: "${buildDir}/intermediates/javac/debug", excludes: fileFilter),
fileTree(dir: "${buildDir}/tmp/kotlin-classes/debug", excludes: fileFilter))
getExecutionData().setFrom(
fileTree(dir: "${buildDir}/outputs/code_coverage", includes: ['*.ec']),
fileTree(dir: "${buildDir}/jacoco", includes: ['*.exec']))
}

Then we just need to modify our new library module’s build.gradle by adding

apply from: '../module-jacoco.gradle'

as well as our standard

buildTypes {
    debug {
        testCoverageEnabled (project.hasProperty('coverage'))
    }
}
cd app
../gradlew -Pcoverage clean assembleDebug assembleDebugAndroidTest
<gcloud to run tests, gsutil to download *.ec files>
../gradlew testDebugUnitTest
../gradlew jacocoReport
<view build/reports/jacoco/jacocoReport/html/index.html>
cd ../library
../gradlew -Pcoverage clean assembleDebug assembleDebugAndroidTest
<gcloud to run tests, gsutil to download *.ec files>
../gradlew testDebugUnitTest
../gradlew jacocoReport
<view build/reports/jacoco/jacocoReport/html/index.html>

And the resulting code coverage reports for each module will accurately reflect the coverage of the unit tests combined with the coverage of the on-device tests that ran in Firebase Test Lab.

 

 

Multi-module applications: unified report

We could put all of this into the project-level build.gradle, but as with the individual modules’ gradle scripts, it’s a bit cleaner to put it into its own gradle script and then import it into the project-level build.gradle. We can follow the collect/flatten pattern to build up the desired list of files or directories. The project-level separate script will look like

apply plugin: 'jacoco'
jacoco {
toolVersion '0.8.7'
}
task jacocoUnifiedReport(type: JacocoReport) {
reports {
xml.enabled = true
html.enabled = true
}
def sourceDirectories = subprojects.collect { subproject ->
"${subproject.projectDir}/src/main/java"
}
def classDirectories = subprojects.collect { subproject ->
["${subproject.buildDir}/intermediates/javac/debug", "${subproject.buildDir}/tmp/kotlin-classes/debug"]
}.flatten()
def classFiles = classDirectories.collect { directory ->
def excludesFilter = ['**/R.class', '**/R$*.class', '**/BuildConfig.*', '**/Manifest*.*', '**/*Test*.*', 'android/**/*.*']
fileTree(dir: file(directory), excludes: excludesFilter)
}
def executionDirectories = subprojects.collect { subproject ->
["${subproject.buildDir}/outputs/code_coverage", "${subproject.buildDir}/jacoco"]
}.flatten()
def executionFiles = executionDirectories.collect { directory ->
def includesFilter = ['*.ec', '*.exec']
fileTree(dir: file(directory), includes: includesFilter)
}
getSourceDirectories().setFrom(sourceDirectories)
getClassDirectories().setFrom(classFiles)
getExecutionData().setFrom(executionFiles)
}

And we’ll just need to add a

apply from: 'project-jacoco.gradle'

to our the build.gradle in the root of our project. Once this is in place, after going through the steps mentioned above to generate the .exec files for off-device tests and download the .ec files for Firebase Test Lab tests, we just need to go to the root of the project and run

./gradlew jacocoUnifiedReport

and we’ll then see a report for the entire project’s coverage in build/reports/jacoco/jacocoUnifiedReport/html/index.html.

 

 

TL;DR

We’ve now achieved the requirements for this article. Looking back, these were the steps we needed to take:

  1. Because Android Test Orchestrator generates multiple .ec files, we’ll also need to change the coverageFile parameter to coverageFilePath and then change our download step to pull *.ec rather than just coverage.ec.
  2. To generate the coverage files for off-device unit tests, all we need to do is make sure that the jacoco plugin is included (already done) and that we don’t pass -Pcoverage when running the actual unit tests. (because of https://issuetracker.google.com/issues/210500600)
  3. To cleanly support generating per-module reports for multi-module applications, we refactored the jacocoReport task out of each module’s build.gradle and into a common module-jacoco.gradle file.
  4. To support a project-wide unified coverage report, we added a new jacocoUnifiedReport task to a project-jacoco.gradle file that is included by the project-level build.gradle.
Next time

Now we have the ability to generate code coverage reports for unit tests and Firebase Test Lab tests, as well as to combine them across the entire project. This process is rather cumbersome, however, and requires many steps. In the third and final article in this series, we’ll look at how this process can be scripted, integrated with Firebase Test Lab automation tools like flank, and finally integrated with CI/CD systems such as gitlab.

Special thanks

Saravana Thiyargaraj wrote a great article (and built a great example on github) that were very useful. Thank you.

YOU MAY BE INTERESTED IN

YOU MAY BE INTERESTED IN

blog
It’s one of the common UX across apps to provide swipe to dismiss so…
READ MORE
blog
Hi, today I come to you with a quick tip on how to update…
READ MORE
blog
Automation is a key point of Software Testing once it make possible to reproduce…
READ MORE
blog
Drag and Drop reordering in Recyclerview can be achieved with ItemTouchHelper (checkout implementation reference).…
READ MORE

Leave a Reply

Your email address will not be published.

Fill out this field
Fill out this field
Please enter a valid email address.

Menu