benchmark compiled agent installations

[kevinjosethomas]

The Bun migration benchmarks currently finish with incomplete results because the harness builds only npm packages and the new installer rejects its local HTTP downloads.

This PR builds a Linux x64 compiled archive alongside the npm fallback packages when the tested revision supports it, then lets the installer choose its normal default. It preserves checksums for both formats, enables the installer's existing loopback test exception, and rejects a compiled candidate that falls back to Node. This must land on main before rerunning the Bun stack's benchmarks, because those jobs use the trusted default-branch harness.

Tested the build and install changes in two fresh Prime sandboxes: main used Node, while the Bun candidate used the compiled executable. Both completed installation, three cold/warm startup trials, memory measurements, and three Python runtime trials without failed samples. All 41 harness tests, Python lint/format checks, and npm run check passed, including the final guard against Node fallback.

ENG-6119 · Related Bun stack: #2140 → #2165 → #2166.

Note

Benchmark compiled agent installations with native Linux x64 release archives

• worker.prepare now validates the four npm tarballs, optionally builds a Linux x64 native archive via a pinned Bun 1.4.0 toolchain, and reports combined archive size with SHA256SUMS
• prepare_native_artifact detects compiled-release support from the presence of both build scripts; incomplete support or a missing output archive raises an error
• verify_installation_format reads the installed launcher's ELF header and rejects Node-script launchers when native artifacts were selected, while npm-only trials still record npm format
• install passes the loopback insecure-download test flag to the installer environment without overriding the installer's default artifact-format selection
• Risk: verify_installation_format raises when a compiled candidate installs through Node — any benchmark trial expecting native execution must ensure the native build succeeds or it will fail rather than silently benchmark Node

^{Macroscope summarized e561a39.}

---

Note

Low Risk
Changes are confined to the benchmark harness and documentation; they do not alter the product installer or runtime outside CI/sandbox measurement paths.

Overview
Extends the PR benchmark harness so revisions with compiled release support are measured with a Linux x64 native archive in addition to the four npm tarballs, instead of npm-only artifacts that made Bun migration comparisons incomplete.

During untimed prepare, the worker optionally installs pinned Bun, runs the binary build and archive assembly, writes SHA256SUMS over all packaged archives, and records artifact_format (npm-tarballs vs npm-tarballs+linux-x64-native). Partial native toolchains now fail preparation rather than silently benchmarking npm-only.

Install trials set PRIME_AGENT_ALLOW_INSECURE_HTTP_FOR_TESTS for loopback release downloads (without forcing install method), then verify_installation_format checks whether prime-agent is an ELF binary; compiled candidates that still install the Node shim are treated as failed installs. README and new test_worker_release.py tests document and lock in this behavior; an existing install test now stubs a prime-agent binary so format verification does not break.

^{Reviewed by Cursor Bugbot for commit e561a39. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

Prime Agent performance — completed

PR e561a399 compared with main 5255a6b8.

Overall: 0 regressed · 0 improved · 17 no clear change.

Metric	Main	This PR	Change
Cold startup	2,937.9 ms	2,808.8 ms	≈ -129.1 ms (-4.40%)
Warm startup	1,704.0 ms	1,640.0 ms	≈ -64.0 ms (-3.75%)
Installation	25.62 s	27.17 s	≈ +1.56 s (+6.07%)
Compressed release artifacts	11.13 MB	11.13 MB	≈ -0.00047 MB (-0.00%)
Installed footprint	597.55 MB	597.55 MB	≈ +0.0018 MB (+0.00%)
Idle memory, summed RSS	1,028.04 MB	1,043.90 MB	≈ +15.85 MB (+1.54%)

Python runtime

Metric	Main	This PR	Change
Python kernel startup	121.4 ms	118.4 ms	≈ -2.9 ms (-2.40%)
Python cell round trip	0.472 ms	0.498 ms	≈ +0.026 ms (+5.51%)
Empty bash command	10.6 ms	10.1 ms	≈ -0.4 ms (-4.06%)
Bash git status	15.6 ms	14.8 ms	≈ -0.8 ms (-4.99%)
Bash 32 KiB output	11.1 ms	10.5 ms	≈ -0.6 ms (-5.56%)
35 cells / 9 shell calls	153.6 ms	149.1 ms	≈ -4.4 ms (-2.89%)
Python interrupt to done	1.440 ms	1.467 ms	≈ +0.027 ms (+1.86%)
Python state snapshot	24.9 ms	24.0 ms	≈ -0.9 ms (-3.73%)
Python state restore	339.4 ms	334.0 ms	≈ -5.4 ms (-1.59%)
Python idle RSS	36.18 MB	33.39 MB	≈ -2.79 MB (-7.70%)
Python RSS after pandas workload	98.32 MB	97.35 MB	≈ -0.97 MB (-0.99%)

Sandbox cost: ~$0.0874 — no inference calls.
Run, logs, and downloadable raw results

Methodology and samples

Main resolved at 2026-09-11T22:56:20.146671+00:00. Harness 5255a6b8.
Linux x64, 4 vCPU, 8 GB RAM, 20 GB disk; region us.
Image: node:24-bookworm@sha256:be23f54a88d34e8824c741b19b91064094f92c1c97b194144bfc8b50d67258e2.
Stock tools, skills, daemon, and Python bootstrap enabled; fresh homes and a fixed Git fixture.
Onboarding is dismissed; the editor starts without a selected model or submitted prompt.
Medians shown. Arrows require a 20% timing/memory change plus absolute floors and IQR.
These practical noise floors are not a statistical significance test.
Cold means stopped Prime processes; OS filesystem caches are not flushed.
No model requests or credentials. Installation excludes build/setup time.
Installer tarballs use loopback; npm/Python downloads use the network with fresh caches.
Artifact size counts release tarballs; footprint after first use includes registry packages.
MB is decimal. Summed RSS can double-count shared pages; PSS is recorded when available.
Provisioning, setup, and build durations are recorded separately in the raw results.
Kernel probes use the installed JSONL runtime, outside the TUI/TypeScript host.
Per trial: 50 Python cells, 5 calls per shell case, and one 35-cell mix (9 git status calls).
Cell/shell values are batch means; other runtime timings are single operations.
State fixture: a 10,000-row × 8-column integer DataFrame and a 10,000-integer list.
Restore runs in a fresh kernel, including pandas imports; kernel startup is excluded.
Kernel RSS covers the isolated Python process; loaded RSS follows the pandas workload.
Costs estimate full sandbox lifetimes at configured rates, including setup and build.
Budget target: $1; not a billing cap. Checks are informational.

Metric	Main successful/attempted	PR successful/attempted	Main spread	PR spread
Cold startup	10/10	10/10	IQR 91.5 ms	IQR 53.5 ms
Warm startup	10/10	10/10	IQR 96.3 ms	IQR 33.5 ms
Installation	3/3	3/3	range 2.39 s	range 1.31 s
Compressed release artifacts	1/1	1/1	—	—
Installed footprint	1/1	1/1	—	—
Idle memory, summed RSS	10/10	10/10	IQR 29.32 MB	IQR 21.93 MB
Python kernel startup	10/10	10/10	IQR 8.0 ms	IQR 3.1 ms
Python cell round trip	10/10	10/10	IQR 0.057 ms	IQR 0.046 ms
Empty bash command	10/10	10/10	IQR 0.2 ms	IQR 0.5 ms
Bash git status	10/10	10/10	IQR 0.5 ms	IQR 1.0 ms
Bash 32 KiB output	10/10	10/10	IQR 0.8 ms	IQR 0.3 ms
35 cells / 9 shell calls	10/10	10/10	IQR 8.2 ms	IQR 6.2 ms
Python interrupt to done	10/10	10/10	IQR 0.111 ms	IQR 0.147 ms
Python state snapshot	10/10	10/10	IQR 2.2 ms	IQR 0.8 ms
Python state restore	10/10	10/10	IQR 24.6 ms	IQR 14.8 ms
Python idle RSS	10/10	10/10	IQR 4.37 MB	IQR 5.53 MB
Python RSS after pandas workload	10/10	10/10	IQR 4.40 MB	IQR 5.42 MB