|
| 1 | +# iOS snapshot convergence evidence |
| 2 | + |
| 3 | +This harness owns the reproducible evidence contract for [#2189](https://github.com/callstack/agent-device/issues/2189), which supplies measurements to [#2188](https://github.com/callstack/agent-device/issues/2188). It does not change daemon, runner, snapshot, or package runtime behavior. |
| 4 | + |
| 5 | +## Prepare a real target |
| 6 | + |
| 7 | +Use a dedicated iOS Simulator and the checked-in test app. Build the CLI and app before measuring: |
| 8 | + |
| 9 | +```sh |
| 10 | +pnpm install --frozen-lockfile |
| 11 | +pnpm build |
| 12 | +pnpm test-app:install |
| 13 | +pnpm test-app:ios -- --device "bench-golden-v1" |
| 14 | +``` |
| 15 | + |
| 16 | +The app build must succeed on the host. If signing, Xcode, XCTest, simulator, runner, or daemon setup cannot be made deterministic, stop and attach the exact command, target, toolchain, and diagnostic; do not report fixture-only timings. |
| 17 | + |
| 18 | +## Local state matrix |
| 19 | + |
| 20 | +Replace `SIMULATOR_UDID` with the dedicated simulator UDID. The default screen set is quiet, list, nested-scroll, alert, system-surface, and xctest-stress. Cold cells require at least 10 samples; warm and relaunch cells require at least 20. |
| 21 | + |
| 22 | +```sh |
| 23 | +pnpm bench:ios-snapshot -- \ |
| 24 | + --mode local \ |
| 25 | + --udid SIMULATOR_UDID \ |
| 26 | + --state cold-cold,cold,warm,relaunch \ |
| 27 | + --screen quiet,list,nested-scroll,alert,system-surface,xctest-stress \ |
| 28 | + --samples 20 \ |
| 29 | + --out .tmp/ios-snapshot-convergence.v1.json |
| 30 | +``` |
| 31 | + |
| 32 | +The cells mean: |
| 33 | + |
| 34 | +- `cold-cold`: simulator off, daemon off, derived runner data cleared before each sample. |
| 35 | +- `cold`: simulator booted, daemon stopped, and app terminated before each sample. |
| 36 | +- `warm`: app, daemon, runner, and target are prepared once; each sample is a fresh CLI snapshot. |
| 37 | +- `relaunch`: the same prepared tooling is retained while each sample launches a new app process. |
| 38 | + |
| 39 | +Every sample keeps daemon duration and fresh-process wall time separately, the first-tree status, response bytes, target generation, and typed failure details. Each raw result also records the typed host model, model identifier, CPU, and core count needed to compare performance baselines. The raw JSON is validated against `raw-result.schema.v1.json`; the adjacent Markdown is a human-readable summary. |
| 40 | + |
| 41 | +Each local cell is admitted only after the simulator state, verified daemon identity, app process generation, and exact fixture anchor are checked. A mismatch stops the run with a typed cell-state or fixture-anchor reason. Derived data is cleared only below the benchmark-owned state directory. |
| 42 | + |
| 43 | +When `--state-dir` is omitted, the harness allocates a fresh marker-owned root under the host temporary directory. A caller-supplied state directory must already be a real, marker-owned directory; the CLI never initializes ownership for an existing path. Any explicit `--derived-path` must remain below that root. |
| 44 | + |
| 45 | +## Proxy matrix |
| 46 | + |
| 47 | +The proxy mode starts the repository proxy, then inserts a local deterministic conditioner in front of it. It runs both a persistent Node client and a fresh-process CLI at RTT 0, 20, and 80 ms. Request and response body bytes, failures, bandwidth, packet-loss rate, and seed are retained in the raw result. |
| 48 | + |
| 49 | +```sh |
| 50 | +pnpm bench:ios-snapshot -- \ |
| 51 | + --mode proxy \ |
| 52 | + --udid SIMULATOR_UDID \ |
| 53 | + --screen quiet \ |
| 54 | + --rtt 0,20,80 \ |
| 55 | + --samples 20 \ |
| 56 | + --bandwidth-kbps unlimited \ |
| 57 | + --packet-loss 0 \ |
| 58 | + --out .tmp/ios-snapshot-proxy.v1.json |
| 59 | +``` |
| 60 | + |
| 61 | +The conditioner is semantics-preserving at zero packet loss. Non-zero loss is an explicit failure experiment, not a successful baseline. |
| 62 | + |
| 63 | +The complete exact-head corpus from `bench-golden-v2` (iPhone 17 Pro, iOS 27.0) is retained under |
| 64 | +[`evidence/`](./evidence/) from revision `71fb2483f30d90e615e949601c836aeebbf450c5`: |
| 65 | + |
| 66 | +- [`ios-snapshot-cold-local-71fb2483f.json`](./evidence/ios-snapshot-cold-local-71fb2483f.json) |
| 67 | + covers cold-cold and cold lifecycle cells across all six screens with 10 samples per cell. |
| 68 | +- [`ios-snapshot-warm-relaunch-local-71fb2483f.json`](./evidence/ios-snapshot-warm-relaunch-local-71fb2483f.json) |
| 69 | + covers warm and relaunch lifecycle cells across all six screens with 20 samples per cell and |
| 70 | + includes package-size measurements. |
| 71 | +- [`ios-snapshot-proxy-71fb2483f.json`](./evidence/ios-snapshot-proxy-71fb2483f.json) covers |
| 72 | + persistent-client and fresh-process CLI cells at RTT 0, 20, and 80 ms with 20 samples per cell. |
| 73 | + |
| 74 | +Each JSON file is the schema-validated raw result from the commit named in its `revision` field; |
| 75 | +each has an adjacent Markdown summary. Superseded pre-admission captures are not part of the |
| 76 | +published corpus. |
| 77 | + |
| 78 | +## Package-size evidence |
| 79 | + |
| 80 | +`pnpm size --json .tmp/size.json --markdown .tmp/size.md` prepares the same Apple and Android |
| 81 | +publish assets used by `pnpm build:package`, then measures bundled JavaScript, packed tarball, |
| 82 | +packed unpacked tree, and the package tree after a clean `npm install` into an isolated consumer. |
| 83 | +The report rejects packages missing either Android helper or containing benchmark/build scripts. |
| 84 | +The iOS harness includes those three package measurements unless `--skip-package-size` is supplied. |
| 85 | + |
| 86 | +## Permanent deep-button control |
| 87 | + |
| 88 | +This is an implementation-independent control for [#1626](https://github.com/callstack/agent-device/issues/1626): the checked-in `deep-button-fixture.v1.json` artifact contains a 72-level ancestor chain and independently recorded shallow/full outputs. The changed leaf is intentionally omitted by the shallow observation, so a no-effect assertion must fail. The full observation includes the leaf and passes. |
| 89 | + |
| 90 | +```sh |
| 91 | +pnpm bench:ios-snapshot:deep-button -- --rule invalid-shallow |
| 92 | +# expected exit 1: |
| 93 | +# AssertionError: changed descendant was omitted by shallow observation; no-effect claim is invalid. |
| 94 | +pnpm bench:ios-snapshot:deep-button -- --rule safe-full |
| 95 | +# expected exit 0 |
| 96 | +``` |
| 97 | + |
| 98 | +The schema links [#1571](https://github.com/callstack/agent-device/issues/1571) so an unreadable or empty first tree remains distinct from bridge, runner, timeout, stale-generation, packet-loss, and upstream failures. This harness does not modify either issue’s runtime behavior. |
0 commit comments