Photo by Lina Castaneda on Unsplash

Screenshot testing is a valuable technique to validate UI and prevent regressions when updating existing screens or components. Yet, as with any kind of testing it has the drawback of requiring a significant time investment to write and maintain the tests.

Showkase

One way to automate screenshot testing is by using Showkase, a library by Airbnb created to generate a component browser which additionally allows to automatically test all methods annotated with @Preview using Paparazzi. Unfortunately, this requires you to add Showkase as a dependency for each module that has any previews you want covered with tests and because Showkase relies on code generation, which will increase the build time for all those modules. So using it just for screenshot tests might not the best solution as it’s not it’s primary function.

Reflection

Another alternative is to use reflection to find all previews at runtime and dynamically create tests for them. A basic implementation of this approach was previously described in this blogpost. This article builds up on the idea described there to provide an improved solution that’s more flexible and covers some additional edge-cases.

For the tests we’ll use the Paparazzi library developed by Cashapp. Mainly because of the fact that it doesn’t require an emulator to run the tests, which makes them both faster and more reliable.

That said, the proposed solution can work with other screenshot testing libraries — if you’re curious about alternatives, the absolute best place to look would be this Github repository by

, which provides detailed descriptions, implementation examples and a lot of useful information about all the available options for screenshot testing in Android.

Example project

I’ve implemented the reflection based solution on an example project, forked from the official Now in Android app. Most of the code is in the PreviewTests class in the screenshot-test module created specifically for the tests. Additionally, some helper annotations and classes where added to the core:ui module.

Finding all previews

Just as in the original blogpost, we’ll use the Reflections library to find all methods annotated with Preview:

private fun findPreviewMethodsInCodebase(): Set<Method> {
    val configuration = ConfigurationBuilder()
        .forPackage(APPLICATION_PACKAGE)
        .filterInputsBy(FilterBuilder().includePackage(APPLICATION_PACKAGE))
        .addScanners(Scanners.MethodsAnnotated)
    val reflections = Reflections(configuration)
    return reflections.getMethodsAnnotatedWith(Preview::class.java) +
        reflections.getMethodsAnnotatedWith(ThemePreviews::class.java) +
        reflections.getMethodsAnnotatedWith(DevicePreviews::class.java)
}

This method will be used by ComposablePreviewProvider to create a list of ComposablePreview, which we’ll use to run our tests.

Besides finding all methods with the Preview annotation, we also handle an edge-case: the multipreview annotations ThemePreviews and DevicePreviews used in the Now in Android project. To keep things simple, we’ll treat them just as a regular Preview annotation, but if necessary the code of ComposablePreviewProvider could easily be modified to handle them differently.

At this point, it’s necessary to select the strategy we’ll use for the automation. I’ve chosen to create tests for all previews (private methods included), but depending on your specific needs you could either generate tests for public previews only or use a custom annotation to explicitly mark the functions that you want tested.

I find that generating tests for all previews by default works best as the main downside of this is increasing the time required to run the tests. Unless you’re working on a very big project this time is likely to be negligible — in the project this approach was initially implemented over 250 tests take around a minute to run on CI, including time spent building the module.

Ignoring previews and Java Dynamic Proxy

If you decide to generate tests for all previews, there still might be cases when you want to exclude a preview and a custom IgnoreScreenshotTest annotation can be used for that purpose.

override fun provideValues(): List<ComposablePreview> {
    val composablePreviews = mutableListOf<ComposablePreview>()

    findPreviewMethodsInCodebase()
        .filterNot { method -> method.annotations.any { it is IgnoreScreenshotTest } }
        .onEach { if (Modifier.isPrivate(it.modifiers)) it.isAccessible = true }
        .forEach { method ->
            composablePreviews.add(method.toComposablePreview())
        }

    return composablePreviews
}

After finding all our preview methods, filtering out the ones we want to ignore and making those methods accessible, we can now transform those methods into ComposablePreviews. The reason we need to do that is because the last parameter each method we get via reflection will expect is a Composer interface instance. The Composer parameter links the composable function code we write to the runtime and is added to each method by the compose compiler plugin.

We can’t access it directly, but we can leverage Java’s Dynamic Proxy and an InvocationHandler to get the Composer instance when invoking out method. If you’re interested in Java Dynamic Proxy, a more detailed explanation of how it works is available in one of my previous articles.

Naming tests

To run our tests, we’ll use TestParameterInjector:

@RunWith(TestParameterInjector::class)
class PreviewTests {
    @get:Rule val paparazzi: Paparazzi = PaparazziRuleProvider.get()

    @Test
    fun snapshot(
        @TestParameter(valuesProvider = ComposablePreviewProvider::class)
        composablePreview: ComposablePreview,
    ) {
        paparazzi.snapshot { composablePreview() }
    }
}

An important feature about how it works is that it’ll call the toString() method of each parameter we pass to it and use it’s result as a suffix for the name of the test, which will also be the name of the recorded snapshot.

In order to have a meaningful name, we can override this method when creating the wrapper:

private fun Method.toComposablePreview(...): ComposablePreview {
    val proxy = Proxy.newProxyInstance(...) as ComposablePreview

    return object : ComposablePreview by proxy {
        override fun toString(): String {
            return buildList<String> {
                add(declaringClass.simpleName)
                add(name)
            }.joinToString("_")
        }
    }
}

PreviewParameterProvider

A very important edge case that needs to be handled before we can start generating screenshots is previews that use PreviewParameterProvider.

To be able to call those and still have meaningful names, we can create a helper class:

abstract class NamedPreviewParameterProvider<T> : PreviewParameterProvider<T> {
    abstract val nameToValue: Sequence<Pair<String?, T>>
    final override val values: Sequence<T>
        get() = nameToValue.map { it.second }
}


// usage example
class ExampleProvider: NamedPreviewParameterProvider<String>() {
    override val nameToValue = sequenceOf(
        "One" to "1",
        "Two" to "2",
        null to "A parameter that will be ignored for tests"
    )
}

The code in ComposablePreviewProvider.provideValues will check each method to see if it uses parameters with the helper function:

private fun Method.findPreviewParameterAnnotation(): PreviewParameter? {
    return this.parameterAnnotations
        .flatMap { it.toList() }
        .find { it is PreviewParameter } as PreviewParameter?
}

If it does, we can get the class type of the PreviewParameterProvider, create an instance and access it’s values to create several ComposablePreviews. We’ll use either the name or index of the parameter as a suffix for the test and snapshot name:

val providerAnnotation = method.findPreviewParameterAnnotation()

if (providerAnnotation == null) { ... } else { 
    // Create an instance of the PreviewParameterProvider.
    val provider = providerAnnotation.provider.createInstanceUnsafe()

    // Get a sequence with the name and value for each parameter
    // that the [Preview] should be called with.
    val nameToValue = if (provider is NamedPreviewParameterProvider<*>) {
        provider.nameToValue.mapNotNull { (name, value) ->
            // ignore previews with a parameter that has null name
            if (name == null) return@mapNotNull null

            name to value
        }
    } else {
        provider.values.mapIndexed { index, value ->
            index.toString() to value
        }
    }

    // Create a [ComposablePreview] for each name and value pair.
    nameToValue.forEach { (nameSuffix, value) ->
        composablePreviews.add(method.toComposablePreview(nameSuffix, value))
    }
}

Custom configuration

If any custom configuration is needed for a specific test, a custom annotation can be created and added to the relevant preview:

@Target(AnnotationTarget.FUNCTION)
annotation class ScreenshotTestParameters(
    val renderingMode: TestRenderingMode = TestRenderingMode.SHRINK,
)

enum class TestRenderingMode { NORMAL, SHRINK }

After some simple changes to the code in PreviewTests, we can extract any data passed via this annotation and use it to alter the snapshot test in any way we see fit. In the repository example we use it to change the rendering mode of Paparazzi in one of the previews from the Shrink default we’ve selected for our tests to Normal.

Troubleshooting

When running the tests to either record ./gradlew screenshot-test:recordPaparazziProdRelease or verify ./gradlew screenshot-test:verifyPaparazziProdRelease we’ll not see the details about the reason of a test failing if there’s something wrong with the test itself:

com.google.samples.apps.nowinandroid.screenshottest.PreviewTests > snapshot[NewsFeedKt_NewsFeedLoadingPreview] FAILED
    java.lang.IllegalArgumentException at PreviewTests.kt:65

32 tests completed, 1 failed

Luckily, if the test is run as a regular unit test, we’ll be able to see the full stack trace:

Afterword

The proposed solution to automate screenshot testing for Composable previews takes just a couple of hours to integrate into a project, has a maintenance cost close to zero (re-recording the snapshots after any UI changes) but allows you to be sure that any UI change will not cause unexpected changes making the experience of working with Jetpack Compose even better. If you have any ideas or suggestions about ways to improve this — leave a comment!

Links

1. Example project
2. Paparazzi
3. Reflections library Github
4. Android screenshot testing playground by Sergio Sastre

This article was previously published on proandroiddev.com