Repository navigation
Conversation
robhogan
force-pushed
the
visual-spike/chain-no-change
branch
from
October 16, 2026 12:13
3252051 to
0924bfd
Compare
cortinico
pushed a commit
to cortinico/react-native
that referenced
this pull request
Oct 22, 2026
Summary: Add screenshot comparison for RNTester in public CI, including fork pull requests, without committing baseline PNGs. Visual flows run alongside the existing Maestro E2E coverage. ### How it works Visual tests are Maestro flows in `packages/rn-tester/visual-tests/flows/`. Each one deep-links to an RNTester example, waits for it to be ready and captures a screenshot cropped to a `testID`. Maestro only captures - it never decides pass or fail. The work is split across a trust boundary, so a fork PR never runs with credentials: - **Visual capture** runs as two jobs in Test All, reusing its RNTester Release builds: `test_visual_android_rntester` after `build_android` and `test_visual_ios_rntester` after `test_ios_rntester`. Each runs every flow in one Maestro invocation, on a Pixel 7 API 35 emulator (1080x2400, 420dpi) or an iPhone 17 Pro on iOS 26.2, and uploads the PNGs and a manifest recording the commit and device environment as an artifact. They have no secrets. - **Visual Compare RNTester** (`workflow_run` on Test All, from `main`) treats that artifact as data - it's validated before any image is decoded and nothing in it is executed. It finds the baseline, compares with pixelmatch, posts a check run per platform and a single PR comment with expected, captured and diff images, and on pushes to `main` publishes the captures as a new baseline. It runs once Test All completes, including when unrelated jobs fail, so the report arrives with the rest of CI rather than as soon as the captures are ready. - **Visual Approve RNTester** (`pull_request_target`, on the `visual-change-approved` label) records a maintainer's approval of an intentional change. The approval is tied to the exact set of differences, so a later push that changes them removes the label and needs a new review. Comparison is strict by default: any changed pixel is a difference. A test can be loosened in `visual-tests/config.json`, with evidence. Captures have been byte-identical across separate builds and runners so far, so I'd expect that to be rare. ### What defines the baseline A baseline is the full set of captures from a commit on `main`, keyed by commit and by device environment. Every push to `main` publishes one, and nothing is ever overwritten. A PR is compared with the baseline for the commit it's based on (the first parent of the merge commit CI tests), or the nearest earlier commit on `main` that has one, and the report says which. So a PR based on last week's `main` is compared with last week's rendering, not with whatever has landed since. Approving a PR doesn't promote its images - `main` is captured again after merge and that becomes the baseline. The environment key covers everything that affects rendering (OS runtime, device, screen size, density, font scale, locale, appearance, animations, system image and, on Android, the emulator version, which is pinned). If the CI image drifts, the key changes and a fresh baseline is published rather than every PR showing a spurious diff. ### Settings Storage is any GCS bucket, configured through repository variables. Writes use GitHub's OIDC token, so there's no stored secret, and only the two trusted workflows on `main` can authenticate. They need create-only access (`roles/storage.objectCreator`). Reads are public. | Variable | | |---|---| | `VISUAL_BUCKET` | Bucket to upload to | | `VISUAL_PUBLIC_BASE_URL` | Public URL for reads and report images - the bucket, or a CDN in front of it | | `VISUAL_GCP_PROJECT` | GCP project for authentication | | `VISUAL_WORKLOAD_IDENTITY_PROVIDER` | Workload identity provider trusting this repo's `visual-compare-rntester.yml` and `visual-approve-rntester.yml` on `main` | Expo can host the bucket and a CDN for this. Reports and approvals expire after 30 days by lifecycle rule. Baselines are small (8-175 KB per image), and at the current rate of pushes to `main` the cost is cents a month. `packages/rn-tester/visual-tests/README.md` covers adding a test and handling a failure. This starts with two text tests and runs non-blocking. The plan is to expand high-value text and rendering coverage, then make the checks required once they've proven reliable. Changelog: [Internal] - Visual regression testing for RNTester in CI Pull Request resolved: react#59115 Test Plan: Ran the full flow in a fork (https://github.com/robhogan/react-native), with Test All trimmed to the jobs this depends on, against a personal GCS bucket: - A PR with no visual change passes on both platforms (robhogan#3). - A deliberate text layout change fails both platforms, with expected, captured and diff images on the checks page and in the PR comment (robhogan#4). - Applying `visual-change-approved` passes the approval check. Pushing a further, different change removed the label with a comment and failed the check again. - Captures were byte-identical across 5 Android and 4 iOS runs, from 3 separate builds on fresh runners. - When an unrelated Test All job fails, the comparison still runs. Run against a Test All run whose Android capture had failed, it reports Android as having no capture and compares iOS as usual. - Pixel Launcher raised an ANR dialog during emulator boot in 3 of 8 probe runs. After the capture's setup step, none of the 8 had a dialog showing. - Running all flows in one Maestro invocation brought the iOS capture step to 2m20s for two tests, from 3m49s for one test when each flow started Maestro separately. Failed flows are retried individually. - New unit tests for artifact validation (path traversal, symlinks, hash and size checks), comparison, storage keys, approval and report rendering. Not yet tested: a PR from a third-party fork, to confirm it can't reach storage credentials. Local validation after fixes: - `arc lint` passes for all 28 changed paths. - Workflow Node unit tests: 3 suites, 46 tests pass. - Yarn validation passes for `xplat/js`, `xplat/js/public`, and `xplat/js/react-native-github`; offline mirror verified. - Reformatted new workflow scripts with the OSS Prettier/ESLint configuration and rebased onto current stable. CI review follow-up: - Make approval-label removal idempotent when another workflow already removed it; retain errors for permission failures. - Reject invalid `VISUAL_CAPTURE_ATTEMPTS` before app installation. - `js1 test` with a focused Node workflow configuration: 3 suites, 54 tests pass, including both regression cases. - `arc lint` passes after these fixes. Additional CI follow-up: - Fix the RN Babel/regenerator failure by reading the HTTP error body before interpolating it. - Distinguish a failed GCS listing or missing listed report from a successful empty listing; storage errors preserve the approval label. - Standard `js1 test` in the `rn-oss` project: all 3 workflow suites and 57 tests pass, using the same RN transformation pipeline as CI. - OSS Prettier and `arc lint` pass. Dependency alignment follow-up: - Align `arvr/js` pngjs with the new xplat dependency and regenerate its lockfile and checked-in package. - Existing `yarn test -p xplat-js-versions-aligned-test` gate: 186 tests pass. - `arc lint` passes for the dependency changes. - Yarn validator verifies the offline mirror; `arvr/js` is excluded by the validator configuration, so the ARVR alignment gate above provides its package-sync check. Capture environment follow-up: - Reject invalid or missing required environment properties before constructing a baseline key, including zero/NaN API levels and densities; retain optional native-bridge/emulator metadata. - Standard `js1 test` in `rn-oss`: all 3 workflow suites and 73 tests pass, including Android/iOS environment regression cases. - OSS Prettier and `arc lint` pass. Review disposition: `latestResults` intentionally stops approval when the newest listed report has no `results.json`. A partial newer upload must not authorize an older fingerprint. A successful empty listing is the only missing-report case treated as no report; storage/read failures remain errors. Complete ARVR install validation: - Remove 34 legacy PNG files not shipped by `pngjs` 7 and regenerate package files/metadata produced by the CI installer. - All 25 tracked PNG files match the reused centralized archive byte for byte (`/tmp/rn-visual-pngjs-upgrade-proof.json`). - Exact CI command `fbpython arvr/js/.castle/scripts/check_yarn_install.py --quiet` passes all four package roots with `Success. No changes detected.` - `arc lint` passes for all applicable regenerated files. - V6 CI confirms `yarn-validation`, `fbsource-yarn-package-sync-validation`, and all three workflow unit-test suites pass. The 3PL gate classified the existing in-place upgrade as new vendoring; provenance is documented above. Its manual retry was denied by the `llms_ci_claude_code` scheduling ACL. Flow compatibility follow-up: - Bound `defineRecordClass` defaults with `T extends interface {}` to satisfy the regenerated Immutable `Record` declarations. - `yarn flow focus-check --max-workers=8 --show-all-errors libraries/flux/records/Record.js` reports zero errors, checking the helper and its dependents (162 files). Full-root Flow validation remains a CI gate. - `arc f` passes. `arc lint` reports only the existing `local/testing-no-untested-method` warning for this pre-existing helper; all new type-bound lint findings are fixed. V8 CI and release-boundary follow-up: - Both `3pl-duplicate-detector` checks pass in V8. The earlier manual retry request is resolved. - Split the Gauntlet-only generated metadata into prerequisite `D126750229`; the base and prerequisite both pass `validate_jest_e2e_and_non_jest_e2e_changes.sh` locally. - The aggregate patch of the prerequisite, base, and ten suite drafts is byte-identical to the pre-split stack. - Gauntlet policy requires the prerequisite to land and enter a promoted release before this dependent diff can land; peer acceptance and final CI remain required. V9 validation results: - All three ARVR installer CI gates pass: Linux, Windows, and macOS. Both `flow-check-fbs` checks, `yarn-validation`, and `fbsource-yarn-package-sync-validation` pass. - Reverified all ten suite diffs remain drafts in the dependency chain above this base; each contains a Maestro flow and scenario inventory. All ten current YAML flows parse, and all 50 new capture IDs are unique and valid. - The prerequisite `D126750229` is blocked by a proven ScanCode name collision: its private Meta Gauntlet workspace is incorrectly identified as public npm `gauntlet`. The exact log evidence and documented TPMS first-party remediation are attached to that prerequisite. A maintainer must correct the TPMS classification, then land and release the framework prerequisite. Base peer acceptance and any remaining CI are still required. October 22 CI follow-up: - Update the Helix invalid-PNG assertions to the error emitted by `pngjs` 7. The tests continue verifying invalid data rejection, capture cleanup, preservation of a decode error when cleanup fails, and no retry. - `yarn test -p "HelixHTestImages-test|HelixHTest-test" --runInBand`: both affected suites pass, 300 tests total. - `arc f` and `arc lint` pass for both changed test files. - The later export job fails because the linked GitHub PR belongs to Rob Hogan; a PR ownership decision is pending. The Gauntlet TPMS classification correction remains pending. Reviewed By: cipolleschi Differential Revision: D126719236 Pulled By: cortinico
cortinico
pushed a commit
to cortinico/react-native
that referenced
this pull request
Oct 22, 2026
Summary: RNTester's rendering coverage, particularly text, currently lives in Meta's internal jest-e2e suite. To move it into public CI we need screenshot comparison that works for fork PRs and doesn't add PNGs to the repo. This adds that, as a separate set of visual flows alongside the existing Maestro E2E. ### How it works Visual tests are Maestro flows in `packages/rn-tester/visual-tests/flows/`. Each one deep-links to an RNTester example, waits for it to be ready and captures a screenshot cropped to a `testID`. Maestro only captures - it never decides pass or fail. The work is split across a trust boundary, so a fork PR never runs with credentials: - **Visual capture** runs as two jobs in Test All, reusing its RNTester Release builds: `test_visual_android_rntester` after `build_android` and `test_visual_ios_rntester` after `test_ios_rntester`. Each runs every flow in one Maestro invocation, on a Pixel 7 API 35 emulator (1080x2400, 420dpi) or an iPhone 17 Pro on iOS 26.2, and uploads the PNGs and a manifest recording the commit and device environment as an artifact. They have no secrets. - **Visual Compare RNTester** (`workflow_run` on Test All, from `main`) treats that artifact as data - it's validated before any image is decoded and nothing in it is executed. It finds the baseline, compares with pixelmatch, posts a check run per platform and a single PR comment with expected, captured and diff images, and on pushes to `main` publishes the captures as a new baseline. It runs once Test All completes, including when unrelated jobs fail, so the report arrives with the rest of CI rather than as soon as the captures are ready. - **Visual Approve RNTester** (`pull_request_target`, on the `visual-change-approved` label) records a maintainer's approval of an intentional change. The approval is tied to the exact set of differences, so a later push that changes them removes the label and needs a new review. Comparison is strict by default: any changed pixel is a difference. A test can be loosened in `visual-tests/config.json`, with evidence. Captures have been byte-identical across separate builds and runners so far, so I'd expect that to be rare. ### What defines the baseline A baseline is the full set of captures from a commit on `main`, keyed by commit and by device environment. Every push to `main` publishes one, and nothing is ever overwritten. A PR is compared with the baseline for the commit it's based on (the first parent of the merge commit CI tests), or the nearest earlier commit on `main` that has one, and the report says which. So a PR based on last week's `main` is compared with last week's rendering, not with whatever has landed since. Approving a PR doesn't promote its images - `main` is captured again after merge and that becomes the baseline. The environment key covers everything that affects rendering (OS runtime, device, screen size, density, font scale, locale, appearance, animations, system image and, on Android, the emulator version, which is pinned). If the CI image drifts, the key changes and a fresh baseline is published rather than every PR showing a spurious diff. ### Settings Storage is any GCS bucket, configured through repository variables. Writes use GitHub's OIDC token, so there's no stored secret, and only the two trusted workflows on `main` can authenticate. They need create-only access (`roles/storage.objectCreator`). Reads are public. | Variable | | |---|---| | `VISUAL_BUCKET` | Bucket to upload to | | `VISUAL_PUBLIC_BASE_URL` | Public URL for reads and report images - the bucket, or a CDN in front of it | | `VISUAL_GCP_PROJECT` | GCP project for authentication | | `VISUAL_WORKLOAD_IDENTITY_PROVIDER` | Workload identity provider trusting this repo's `visual-compare-rntester.yml` and `visual-approve-rntester.yml` on `main` | Expo can host the bucket and a CDN for this. Reports and approvals expire after 30 days by lifecycle rule. Baselines are small (8-175 KB per image), and at the current rate of pushes to `main` the cost is cents a month. `packages/rn-tester/visual-tests/README.md` covers adding a test and handling a failure. This starts with two text tests and runs non-blocking. The plan is to port high-value jest-e2e text and rendering tests, then make the checks required once they've proven reliable. ## Changelog: [INTERNAL] [ADDED] - Visual regression testing for RNTester in CI Pull Request resolved: react#59115 Test Plan: Ran the full flow in a fork (https://github.com/robhogan/react-native), with Test All trimmed to the jobs this depends on, against a personal GCS bucket: - A PR with no visual change passes on both platforms (robhogan#3). - A deliberate text layout change fails both platforms, with expected, captured and diff images on the checks page and in the PR comment (robhogan#4). - Applying `visual-change-approved` passes the approval check. Pushing a further, different change removed the label with a comment and failed the check again. - Captures were byte-identical across 5 Android and 4 iOS runs, from 3 separate builds on fresh runners. - When an unrelated Test All job fails, the comparison still runs. Run against a Test All run whose Android capture had failed, it reports Android as having no capture and compares iOS as usual. - Pixel Launcher raised an ANR dialog during emulator boot in 3 of 8 probe runs. After the capture's setup step, none of the 8 had a dialog showing. - Running all flows in one Maestro invocation brought the iOS capture step to 2m20s for two tests, from 3m49s for one test when each flow started Maestro separately. Failed flows are retried individually. - New unit tests for artifact validation (path traversal, symlinks, hash and size checks), comparison, storage keys, approval and report rendering. Not yet tested: a PR from a third-party fork, to confirm it can't reach storage credentials. ### Internal CI repair after the V11 import - Rebased on `remote/master` (`de108a6bc86ad106de94b84aa819e58b23cfc797`). Removed the dependency on abandoned D126750229; no Gauntlet framework files are included. All ten child suite patches are unchanged. Rob's PR react#59115 remains linked. - `xplat/yarn-validation/bin/yarn-validation`: passed for the root and RN OSS workspace, including the offline mirror check. - ARVR `HelixHTestImages-test`, `HelixHTest-test`, and `xplat-js-versions-aligned-test`: 3 suites and 486 tests passed. - `js1 test` selecting `rn-oss`: all 3 visual workflow suites and 51 tests passed. - OSS ESLint with `--no-ignore --max-warnings=0` on all 15 workflow JavaScript files: passed. `arc f` applied formatting; `arc lint` found no new issues. The Record helper missing-test warning also occurs on the unchanged `remote/master` file. - The exact four-root ARVR installer check exits 1 solely for existing Gauntlet `electron-chromedriver/METADATA.bzl` package URL drift. Restored that unrelated file and excluded the abandoned prerequisite change. - Device/Maestro captures were not repeated for this dependency and lint repair; Rob's Android/iOS evidence above and the child suite test plans are retained. Fresh CI results remain pending. Reviewed By: Abbondanzo, cipolleschi Differential Revision: D126719236 Pulled By: cortinico
meta-codesync Bot
pushed a commit
to react/react-native
that referenced
this pull request
Oct 22, 2026
Summary: RNTester's rendering coverage, particularly text, currently lives in Meta's internal jest-e2e suite. To move it into public CI we need screenshot comparison that works for fork PRs and doesn't add PNGs to the repo. This adds that, as a separate set of visual flows alongside the existing Maestro E2E. ### How it works Visual tests are Maestro flows in `packages/rn-tester/visual-tests/flows/`. Each one deep-links to an RNTester example, waits for it to be ready and captures a screenshot cropped to a `testID`. Maestro only captures - it never decides pass or fail. The work is split across a trust boundary, so a fork PR never runs with credentials: - **Visual capture** runs as two jobs in Test All, reusing its RNTester Release builds: `test_visual_android_rntester` after `build_android` and `test_visual_ios_rntester` after `test_ios_rntester`. Each runs every flow in one Maestro invocation, on a Pixel 7 API 35 emulator (1080x2400, 420dpi) or an iPhone 17 Pro on iOS 26.2, and uploads the PNGs and a manifest recording the commit and device environment as an artifact. They have no secrets. - **Visual Compare RNTester** (`workflow_run` on Test All, from `main`) treats that artifact as data - it's validated before any image is decoded and nothing in it is executed. It finds the baseline, compares with pixelmatch, posts a check run per platform and a single PR comment with expected, captured and diff images, and on pushes to `main` publishes the captures as a new baseline. It runs once Test All completes, including when unrelated jobs fail, so the report arrives with the rest of CI rather than as soon as the captures are ready. - **Visual Approve RNTester** (`pull_request_target`, on the `visual-change-approved` label) records a maintainer's approval of an intentional change. The approval is tied to the exact set of differences, so a later push that changes them removes the label and needs a new review. Comparison is strict by default: any changed pixel is a difference. A test can be loosened in `visual-tests/config.json`, with evidence. Captures have been byte-identical across separate builds and runners so far, so I'd expect that to be rare. ### What defines the baseline A baseline is the full set of captures from a commit on `main`, keyed by commit and by device environment. Every push to `main` publishes one, and nothing is ever overwritten. A PR is compared with the baseline for the commit it's based on (the first parent of the merge commit CI tests), or the nearest earlier commit on `main` that has one, and the report says which. So a PR based on last week's `main` is compared with last week's rendering, not with whatever has landed since. Approving a PR doesn't promote its images - `main` is captured again after merge and that becomes the baseline. The environment key covers everything that affects rendering (OS runtime, device, screen size, density, font scale, locale, appearance, animations, system image and, on Android, the emulator version, which is pinned). If the CI image drifts, the key changes and a fresh baseline is published rather than every PR showing a spurious diff. ### Settings Storage is any GCS bucket, configured through repository variables. Writes use GitHub's OIDC token, so there's no stored secret, and only the two trusted workflows on `main` can authenticate. They need create-only access (`roles/storage.objectCreator`). Reads are public. | Variable | | |---|---| | `VISUAL_BUCKET` | Bucket to upload to | | `VISUAL_PUBLIC_BASE_URL` | Public URL for reads and report images - the bucket, or a CDN in front of it | | `VISUAL_GCP_PROJECT` | GCP project for authentication | | `VISUAL_WORKLOAD_IDENTITY_PROVIDER` | Workload identity provider trusting this repo's `visual-compare-rntester.yml` and `visual-approve-rntester.yml` on `main` | Expo can host the bucket and a CDN for this. Reports and approvals expire after 30 days by lifecycle rule. Baselines are small (8-175 KB per image), and at the current rate of pushes to `main` the cost is cents a month. `packages/rn-tester/visual-tests/README.md` covers adding a test and handling a failure. This starts with two text tests and runs non-blocking. The plan is to port high-value jest-e2e text and rendering tests, then make the checks required once they've proven reliable. ## Changelog: [INTERNAL] [ADDED] - Visual regression testing for RNTester in CI Pull Request resolved: #59115 Test Plan: Ran the full flow in a fork (https://github.com/robhogan/react-native), with Test All trimmed to the jobs this depends on, against a personal GCS bucket: - A PR with no visual change passes on both platforms (robhogan#3). - A deliberate text layout change fails both platforms, with expected, captured and diff images on the checks page and in the PR comment (robhogan#4). - Applying `visual-change-approved` passes the approval check. Pushing a further, different change removed the label with a comment and failed the check again. - Captures were byte-identical across 5 Android and 4 iOS runs, from 3 separate builds on fresh runners. - When an unrelated Test All job fails, the comparison still runs. Run against a Test All run whose Android capture had failed, it reports Android as having no capture and compares iOS as usual. - Pixel Launcher raised an ANR dialog during emulator boot in 3 of 8 probe runs. After the capture's setup step, none of the 8 had a dialog showing. - Running all flows in one Maestro invocation brought the iOS capture step to 2m20s for two tests, from 3m49s for one test when each flow started Maestro separately. Failed flows are retried individually. - New unit tests for artifact validation (path traversal, symlinks, hash and size checks), comparison, storage keys, approval and report rendering. Not yet tested: a PR from a third-party fork, to confirm it can't reach storage credentials. ### Internal CI repair after the V11 import - Rebased on `remote/master` (`de108a6bc86ad106de94b84aa819e58b23cfc797`). Removed the dependency on abandoned D126750229; no Gauntlet framework files are included. All ten child suite patches are unchanged. Rob's PR #59115 remains linked. - `xplat/yarn-validation/bin/yarn-validation`: passed for the root and RN OSS workspace, including the offline mirror check. - ARVR `HelixHTestImages-test`, `HelixHTest-test`, and `xplat-js-versions-aligned-test`: 3 suites and 486 tests passed. - `js1 test` selecting `rn-oss`: all 3 visual workflow suites and 51 tests passed. - OSS ESLint with `--no-ignore --max-warnings=0` on all 15 workflow JavaScript files: passed. `arc f` applied formatting; `arc lint` found no new issues. The Record helper missing-test warning also occurs on the unchanged `remote/master` file. - The exact four-root ARVR installer check exits 1 solely for existing Gauntlet `electron-chromedriver/METADATA.bzl` package URL drift. Restored that unrelated file and excluded the abandoned prerequisite change. - Device/Maestro captures were not repeated for this dependency and lint repair; Rob's Android/iOS evidence above and the child suite test plans are retained. Fresh CI results remain pending. Reviewed By: Abbondanzo, cipolleschi Differential Revision: D126719236 Pulled By: cortinico fbshipit-source-id: 385b06100f6d57f456a00631b62ead0835fc4089
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fork test PR for react#59115 (capture chained on Test All builds).