The Regression Your Test Suite Cannot See
A Compose UI test asks the semantics tree questions. Does a node with this text exist? Is it enabled? Does clicking it change that other node? Those are the right questions for behaviour, and they are structurally blind to everything a user actually looks at.
Collapse a card's padding from 16dp to 0 and every semantic assertion still passes. Swap two colour tokens so the dark theme renders black text on a near-black surface: green. Ship a button whose label truncates to "Contin…" at a 1.3 font scale: green. The tree is identical; only the pixels changed, and nothing in the suite looks at pixels.
That gap is what screenshot tests close. Google's own description is precise: a screenshot test "takes a screenshot of a piece of UI and then compares it against a previously approved reference image. If the images don't match, the test fails and produces an HTML report to help you compare and find the differences." Google's testing strategy guide files them as component tests and uses exactly this case as its example: "Screenshot test for a custom button."
The other reason to adopt them is cost. The modern tools render on the JVM, which puts screenshot tests in the category Google's testing fundamentals describes as local tests: "usually small and fast, isolating the subject under test from the rest of the app." No emulator farm, no device flake, no forty-minute instrumented job. That changes screenshot testing from a quarterly visual audit into something that runs on every pull request.
Three Tools, Two Rendering Engines
Every host-side screenshot tool for Compose makes one foundational choice, and Google's screenshot testing guide names it: host-side solutions render with Layoutlib, "Android Studio's rendering engine for previews," or with "Robolectric Native Graphics (RNG)." The trade-off is stated just as plainly: "Layoutlib-based frameworks are focused on rendering static components, using different states to show different behavior. They're typically easier to use. Frameworks that integrate with RNG can use all the features from Robolectric, allowing for tests with a bigger scope."
Paparazzi's pitch is in its first line: it renders "your application screens without a physical device or emulator." Roborazzi exists because of an incompatibility its README spells out: "Paparazzi is a great tool for visualizing displays within the JVM. However, it's incompatible with Robolectric, which also mocks the Android framework. Roborazzi fills this gap."
The decision rule that falls out of this is simple. If what you want to protect is a component in a set of states (a button, a card, a design-system token sheet), a Layoutlib tool is less code and you probably already wrote the previews. If what you want to protect is a screen after an interaction (the error state after a failed submit, the expanded row after a tap), you need Robolectric underneath, and that means Roborazzi.
| Tool | Rendering engine | What you write | Record / verify |
|---|---|---|---|
| Compose Preview Screenshot Testing (Google) | Layoutlib | A @Preview marked @PreviewTest | Update and validate Gradle tasks, HTML report |
| Paparazzi (Cash App) | Layoutlib | A JUnit test with a Paparazzi rule | recordPaparazziDebug, verifyPaparazziDebug |
| Roborazzi | Robolectric Native Graphics | A Robolectric test that can click, scroll and navigate first | recordRoborazziDebug, verifyRoborazziDebug |
Google’s Tool Just Moved Into the Android Gradle Plugin
Compose Preview Screenshot Testing turns previews you already maintain into tests, which is its whole appeal: a design-system module with forty
@Preview functions is forty screenshot tests waiting to be switched on. @Preview parameters such as uiMode and fontScale, and multi-previews, multiply them across themes and text sizes for free.
It is also a moving target, and it has just moved. The page now opens with a deprecation notice: "Starting with Android Gradle Plugin (AGP) 9.5.0-alpha03 and Compose Preview Screenshot Testing 0.0.1-alpha16, we recommend configuring screenshot tests using AGP test suites. The standalone plugin method described on this page is deprecated." It still carries the experimental label too: "Compose Preview Screenshot Testing is still in development. Its features and APIs are subject to change substantially during the alpha phase."
In the test-suites setup, you enable two flags in gradle.properties, declare a suite under testOptions, and get suite-named tasks: update{SuiteName}{Target}{Variant}TestSuite writes the references and test{SuiteName}{Target}{Variant}TestSuite compares against them. Failures produce an HTML report under build/reports/tests/{taskName}/. If you are still on the standalone plugin, its floor is AGP 9.0, Kotlin 2.2.10, JDK 17 and plugin 0.0.1-alpha16, and its tasks are updateDebugScreenshotTest and validateDebugScreenshotTest.
The honest recommendation for a production team: adopt it for a design-system module where previews already exist, pin the engine version, and budget for configuration churn. An alpha that has already changed its setup model once will change it again.kotlin
# gradle.properties
android.experimental.enableScreenshotTest=true
android.experimental.testSuiteSupport=true
// build.gradle.kts (module)
android {
testOptions {
screenshotTests.create("screenshotTest") {
engineVersion = "0.0.1-alpha16"
targetVariants.add("debug")
// plus the suite's dependencies block from the setup guide
}
}
}
// src/screenshotTest/kotlin/.../ButtonScreenshots.kt
@PreviewTest
@Preview(name = "light", showBackground = true)
@Preview(name = "dark", uiMode = Configuration.UI_MODE_NIGHT_YES, showBackground = true)
@Preview(name = "large-text", fontScale = 1.5f, showBackground = true)
@Composable
fun PrimaryButtonPreview() {
AppTheme { PrimaryButton(text = "Continue", onClick = {}) }
}The Threshold Decides Whether the Suite Is Worth Running
Every screenshot suite dies the same way: it fails on something nobody can see, a developer re-records the goldens to get green, and after the third time the team stops reading the diffs. The setting that prevents that death is the comparison threshold, and Google is candid that it cuts both ways. Tolerance-based comparison helps, but the guide warns it "could create false positives, and not catch errors that are either below the threshold, or erroneously considered similar enough."
Compose Preview Screenshot Testing added a module-wide
imageDifferenceThreshold in 0.0.1-alpha06, and from alpha10 the update task "will only update images that have differences greater than a specified threshold", so re-recording no longer churns every golden by a sub-pixel.
The better fix is to make a tiny threshold safe by removing the sources of noise, and Google names the main one: "To use a pixel-perfect screenshot comparator, you must make sure that your tests take screenshots in the same conditions. To do so, you can use your Continuous Integration (CI) system or employ a cloud service." In practice:
- Record where you verify. Goldens recorded on a MacBook and verified on a Linux runner differ in font rasterisation. Generate references in CI, or in the same container CI uses.
- Pin the renderer. Paparazzi's changelog shows why: its 2.0.0 alphas moved LayoutLib versions and, from alpha04, require Java 21. A renderer upgrade is a golden-regeneration event; do it in its own pull request.
- Feed fixed data. No clocks, no network images, no random avatars. A preview that renders "3 minutes ago" is a flaky test with extra steps.
- Store goldens outside normal Git history. Paparazzi's README: "It is recommended you use Git LFS to store your snapshots." A few hundred PNGs re-recorded weekly will bloat a repository fast.Wiring It Into CI So Diffs Get Read
A screenshot failure is only useful if the reviewer can see it in the pull request. The pattern that works: run the verify task on every pull request, and when it fails, upload the HTML report as a build artifact so the diff is one click from the check that went red. Intentional visual changes are recorded by the author, committed with the code that caused them, and reviewed as images in the same diff.
That last rule is the one teams skip. If goldens are regenerated by a bot after merge, nobody ever looks at the visual change before it ships, and the suite quietly becomes a change detector that no one reads. Keep the golden update in the author's pull request, where the image diff sits next to the code that explains it.
yaml
# .github/workflows/screenshots.yml
name: screenshots
on: pull_request
jobs:
verify:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
lfs: true # goldens live in Git LFS
- uses: actions/setup-java@v4
with:
distribution: temurin
java-version: 21
- name: Verify screenshots
# test{SuiteName}{Target}{Variant}TestSuite
run: ./gradlew testScreenshotTestDefaultDebugTestSuite
- name: Upload diff report
if: failure()
uses: actions/upload-artifact@v4
with:
name: screenshot-report
path: '**/build/reports/tests/**'What to Screenshot, and What to Leave to Assertions
Screenshot tests are the most expensive tests to maintain per assertion, because every intentional design change touches them. Spend them where pixels are the contract:
Two rules keep the suite honest. First, one golden per meaningful state, not per screen: a whole-screen screenshot fails on every unrelated tweak and teaches the team to re-record without looking. Second, never let a screenshot test replace a semantic one. A golden image proves the checkout button looks right; only a semantic assertion proves it is enabled, announced correctly to TalkBack, and wired to the right action.
Start with the design-system module. It changes least, it has previews already, and a broken token there breaks every screen at once, which is exactly the class of regression a semantic suite will never catch.
| Screenshot it | Assert it instead |
|---|---|
| Design-system components across light, dark and large font scale | Business rules and state transitions |
| Empty, loading and error states of key screens | Navigation and back-stack behaviour |
| Right-to-left and long-string locales | That a button is enabled or a field is valid |
| Adaptive layouts at compact, medium and expanded widths | Data mapping and formatting logic |
Key Takeaways
- 1Compose UI tests assert semantics; a collapsed margin, a broken dark theme or a truncated label passes them. Screenshot tests are the layer that sees pixels.
- 2Host-side tools render with Layoutlib (Compose Preview Screenshot Testing, Paparazzi) or Robolectric Native Graphics (Roborazzi); pick RNG when the state you need requires interaction.
- 3Google now recommends configuring Compose Preview Screenshot Testing through AGP test suites (AGP 9.5.0-alpha03, engine 0.0.1-alpha16); the standalone plugin is deprecated and the tool is still experimental.
- 4Thresholds cut both ways: too loose misses real regressions. Record and verify in the same CI environment so a tight threshold stays stable.
- 5Keep goldens in Git LFS, pin the renderer version, and feed previews fixed data.
- 6Upload the HTML diff report on failure and keep golden updates in the author’s pull request, where reviewers see the image next to the code.
Frequently Asked
Do Compose screenshot tests need an emulator?
No. Compose Preview Screenshot Testing and Paparazzi render on the JVM with Layoutlib, and Roborazzi renders with Robolectric Native Graphics. All three run as host-side tests.
Should I use the standalone Compose Preview Screenshot Testing plugin?
Google marks it deprecated from AGP 9.5.0-alpha03 and engine 0.0.1-alpha16 and recommends AGP test suites instead. Both paths are still experimental, so pin versions.
Why do my screenshot tests fail on CI but pass locally?
Rendering differs between machines, most often in fonts. Record and verify references in the same CI environment, pin the renderer version, and remove clocks and network images from previews.
Ready to architect your next Android app?
ANDROID-ARCHITECT generates production-ready Kotlin code, architecture blueprints, and CI/CD configurations from plain-language descriptions. Start building for free.
ANDROID-ARCHITECT · CONSOLE
S/N · AX-1A-00001
STRATEGIA-X · ENGINEERED · IN · CALIFORNIA