README.md

Nitro benchmark app

A dedicated Release/Hermes app for measuring JS ↔ native boundary performance. It uses the real C++, Swift/Kotlin, and generated bindings from the Nitro test packages. It has no Harness or navigation dependencies. Correctness tests and demos remain in tests/react-native/apps/example.

The suite runs automatically when a host receiver is present. Launching the app alone reports a controller connection error; it does not silently substitute a Debug benchmark or publish anything to Bencher.

Build

From the repository root:

bash scripts/performance/build-android.sh "$PWD" com.margelo.nitrobenchmark.head
bash scripts/performance/build-ios.sh "$PWD" com.margelo.nitrobenchmark.head

Both platforms use the normal Release configuration, an embedded optimized Hermes bundle, and no debugger or sanitizers. Android enables R8 and uses the development signing key because this app is not distributed. Its network policy permits cleartext only to 127.0.0.1 and localhost for the host receiver.

Run locally

For an already booted Android API 36 emulator, compare four pairs of fresh launches of the same built APK to check measurement variation:

bun scripts/performance/run-sequence.ts \
  --platform android \
  --base-app tests/react-native/apps/benchmark/android/app/build/outputs/apk/release/app-release.apk \
  --head-app tests/react-native/apps/benchmark/android/app/build/outputs/apk/release/app-release.apk \
  --base-root "$PWD" \
  --head-root "$PWD" \
  --output-directory /tmp/nitro-benchmark \
  --device-id "$(adb get-serialno)" \
  --base-sha "$(git rev-parse HEAD)" \
  --head-sha "$(git rev-parse HEAD)" \
  --device 'Local emulator' \
  --os-version 'Android 16 / API 36' \
  --architecture x86_64 \
  --toolchain 'Local JDK 17 / NDK 29.0.14206865'

Adjust device metadata to match the target. The host starts the receiver, installs and launches the app, validates the result, and terminates the app. The host installs each binary once, then launches a fresh process for each case and assembles their results. This releases Nitro's runtime-scoped JSI reference bookkeeping between cases; GC alone cannot clear that cache. Each process posts one result only after its timing is complete. Per-case raw results are kept beside the combined base-1.json through base-4.json and matching head outputs, in base-1-cases/ through base-4-cases/ and matching head directories. At each position in the apps' suite order, launches run AB, BA, BA, AB (A = base, B = head) before advancing to the next position. The report matches results by ID even when cases move, appear or disappear. With the same case order, each function runs back to back. Identical SHAs reuse one installed binary. Startup, transport, and process restarts are not timed. Startup waits for two animation frames, with no fixed one-second sleep. For iOS, use --platform ios, a simulator UDID for --device-id, and the built NitroBenchmark.app for --base-app and --head-app, with matching simulator/toolchain metadata. To compare different revisions, build base from its own checkout using the default app ID (omit the second build-script argument), and build head with the .head ID shown above. Pass the corresponding app paths, source roots, and commit SHAs. Local runs do not upload results.

Each metric has fixed Android/iOS counts in iterations.ts, seeded from the existing GitHub-runner results with roughly 150 ms of timed work per sample. There is no calibration process. Each measurement process performs five warmup batches and twenty samples, for eighty measured samples across the four processes per revision and case. Each revision uses its own checked-in count and chunk size for each case. Slow samples are retained without shortening the work.

Adding a case requires an explicit count. Revisit the counts when changing the case or device, using raw durations from representative runs. The counts remain fixed during CI, even if a revision is slower. Changing a count or test definition does not disable comparisons; the report uses per-operation timings, and the PR author interprets any measurement changes.

The eight primitive-only control, method and numeric property cases skip per-batch GC and frame waits. They still run in fresh processes with the same warmup and sample counts.

Allocation-heavy cases split a sample into bounded chunks, collecting garbage after each chunk, outside the timer. iOS buffer copies use synchronous GC without frame waits; their wrappers release the owned native storage. Other allocating cases also yield for native cleanup after at most four chunks. Kotlin buffer-copy and Promise cases also collect Java's heap between chunks through a synchronous, benchmark-only TurboModule helper; Hermes GC alone cannot reclaim Java-backed direct buffers. Cleanup is excluded from timing. Each sample divides its accumulated timed duration by the total operation count; the memory limit no longer caps the sample duration. Hermes gc() is required. Raw results include iterations and chunkIterations; each sample's total timed milliseconds is samplesNsPerOp[i] * iterations / 1e6. These are operation-cost measurements with explicit inter-chunk cleanup excluded, not sustained allocation/GC throughput. Natural GC during an operation is timed. The measured cost includes the operation, marshaling, and JS loop bookkeeping. Input setup, checksum validation, logging, statistics, and transport are outside the timed batch. Operation-induced allocations remain inside it.

CI and reporting

See performance CI for observed comparisons, process variability, raw artifacts, trusted reporting, and Bencher publishing. CI retains exact app artifacts for measurement-only reruns. It builds both revisions even when definitions change, and reuses one binary for identical SHAs. New and removed cases stay visible in the table without a percentage comparison. Post @nitro-modules-bot please test performance on an open PR to start CI. Each request posts a new report; rerunning the same workflow updates its report. Local same-revision runs show baseline variation. Performance remains report-only.

The example's former benchmark screen and TurboModule control have moved here. No public Nitro API changes are needed. App dependency versions initially match the example; automated version-alignment enforcement is a separate follow-up.