Skip to content

Commit 8564417

Browse files
committed
Document RNTester visual tests
Adds `packages/rn-tester/visual-tests/README.md` on how the tests run in CI, adding a test, running flows locally, handling a failed check and tolerances, and links it from RNTester's README. Changelog: [Internal]
1 parent 34fb6d0 commit 8564417

2 files changed

Lines changed: 76 additions & 0 deletions

File tree

‎packages/rn-tester/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,3 +67,7 @@ This is different from apps created using `react-native init` which have a depen
6767
## Running tests (iOS)
6868

6969
RNTester provides both integration and unit tests. You can run them all by hitting `Cmd+U` in Xcode. Alternatively, it is possible to run just unit tests or integration tests using the keybinding after selecting the appropriate scheme (either `RNTesterUnitTests` or `RNTesterIntegrationTests`).
70+
71+
## Visual tests
72+
73+
CI captures screenshots of selected RNTester examples on Android and iOS and compares them with `main`. See [visual-tests/README.md](visual-tests/README.md) to add a test or review a failure.
Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
# RNTester visual tests
2+
3+
Visual tests capture screenshots of RNTester examples in CI and compare them with the screenshots of the same tests on `main`. They catch rendering changes, particularly in text layout, that functional E2E tests don't.
4+
5+
Each test is a [Maestro](https://maestro.dev) flow in `flows/`. Maestro only opens the example and takes the screenshot - comparison happens afterwards in CI, so a flow never decides whether it passed.
6+
7+
## How it runs in CI
8+
9+
1. Test All builds RNTester Release for Android and iOS. Once a build is ready, the `test_visual_android_rntester` and `test_visual_ios_rntester` jobs (`.github/workflows/visual-capture-rntester.yml`) run every flow in this directory on one canonical device per platform: an API 35 Pixel 7 emulator and an iPhone 17 Pro simulator on iOS 26.2. They upload the screenshots and a manifest of the device environment as `visual-capture-android` and `visual-capture-ios` artifacts.
10+
2. When Test All completes, Visual Compare RNTester (`.github/workflows/visual-compare-rntester.yml`) compares the screenshots with the baseline and reports a `Visual regression (android)` and `Visual regression (ios)` check and a `Visual regression approval` gate, plus a single PR comment with the expected, captured and diff images for anything that changed.
11+
3. On pushes to `main`, the captures become the baseline for that commit.
12+
13+
The baseline for a PR is the one for the commit it's based on, or the nearest earlier commit on `main` that has one. Baselines are keyed by device environment too (OS image, screen size, density, font scale, locale and so on), so a change to the CI device starts a fresh baseline rather than failing every PR.
14+
15+
## Adding a test
16+
17+
1. Pick or add an RNTester example, and give its root view a `testID`. Keep the example's content fixed: no dates, random values, network images or animations that are still running when the screenshot is taken.
18+
2. Add a flow to `flows/`. File names must be unique, and each `takeScreenshot` path becomes the test ID, so make it descriptive:
19+
20+
```yaml
21+
appId: ${APP_ID}
22+
---
23+
- runFlow:
24+
file: ../helpers/open-example.yml
25+
env:
26+
EXAMPLE_URL: rntester://example/TextExample/textWidthMode
27+
- extendedWaitUntil:
28+
visible:
29+
id: 'text-width-mode-example'
30+
timeout: 15000
31+
- takeScreenshot:
32+
path: ${VISUAL_OUTPUT_DIR}/text/width-mode/default
33+
cropOn:
34+
id: 'text-width-mode-example'
35+
```
36+
37+
`open-example.yml` deep-links to `rntester://example/<module>/<example>`. Wait for something that's only visible once the example has rendered, and crop to the example's `testID` so that the status bar and surrounding UI aren't part of the screenshot.
38+
39+
3. Open a PR. The new test has no baseline yet, so the report lists it as new, and like any other difference it needs a maintainer's approval (see below). Once the PR lands, `main` is captured with it and later PRs are compared against that.
40+
41+
To try a flow locally, build and install RNTester Release on a simulator or emulator and run it with Maestro:
42+
43+
```sh
44+
maestro test \
45+
-e APP_ID=com.meta.RNTester.localDevelopment \
46+
-e VISUAL_OUTPUT_DIR=/tmp/visual \
47+
packages/rn-tester/visual-tests/flows/text-width-mode.yml
48+
```
49+
50+
Use `com.facebook.react.uiapp` as the `APP_ID` on Android. Local screenshots are only useful for checking the flow and the crop - they won't match CI's pixel for pixel unless you use the same device and OS image.
51+
52+
## When a check fails
53+
54+
Open the PR comment or the check run and compare the expected, captured and diff images.
55+
56+
- If the change is unintended, fix it and push.
57+
- If it's intended, a maintainer applies the `visual-change-approved` label. The approval covers the exact set of differences in the report: if a later push changes them, the label is removed and the PR needs a new review. The baseline itself only changes once the PR lands and `main` is captured again.
58+
59+
## Tolerances
60+
61+
Comparison is strict by default: any changed pixel is a difference. If a test is genuinely nondeterministic, loosen it for that test in `config.json`, keyed by test ID, and say why in the PR:
62+
63+
```json
64+
{
65+
"defaults": {"threshold": 0, "includeAA": true, "maxDiffPixels": 0},
66+
"tests": {
67+
"text/width-mode/default": {"maxDiffPixels": 20}
68+
}
69+
}
70+
```
71+
72+
`threshold` is pixelmatch's per-pixel color distance from 0 to 1, `includeAA` counts anti-aliased pixels, and `maxDiffPixels` and `maxDiffRatio` set how many changed pixels (or what share of the image) can pass. Prefer making the example deterministic over adding a tolerance.

0 commit comments

Comments
 (0)