colour benchmark changes below the noise threshold

[kevinjosethomas]

• fixes eng-6043: small measured deltas now receive muted red or green, with stronger colours for larger percentages.
• preserves the noise threshold for arrows and overall assessments; exact zero and incomplete results remain neutral.
• verified with 30 benchmark tests, repository checks, and github’s comment preview.

---

Note

Low Risk
Reporting-only change in the benchmark harness; noise thresholds and outcome aggregation are unchanged.

Overview
PR benchmark comments now show direction for small deltas while keeping the noise gate for arrows and headline counts.

In comparison, changes below the existing absolute/relative/spread threshold still count as no clear change and keep the ≈ prefix, but nonzero deltas are rendered with muted red or green MathJax text whose intensity scales with percentage. ↑/↓ and improved/regressed outcomes apply only when the delta exceeds the threshold; exact zero, incomplete, and unavailable rows stay uncolored.

The benchmarks README documents this split (color vs arrows vs summary). Tests cover sub-threshold coloring in full renders, colored small bundle deltas, and neutral exact-zero cells.

^{Reviewed by Cursor Bugbot for commit f69921c. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Color nonzero benchmark changes below the noise threshold in report.py

Updates the benchmark comparison helper in report.py so that nonzero complete changes are always passed through color rendering, regardless of whether they exceed the threshold. The threshold now controls only the displayed symbol (≈ vs directional arrow) and the outcome classification (no clear change vs regressed/improved). Exact-zero, incomplete, and unavailable comparisons remain uncolored. Tests in test_benchmarks.py and docs in README.md are updated to match.

• Behavioral Change: within-threshold nonzero changes now render with direction-specific color markup where they were previously uncolored; exact-zero changes remain uncolored.

^{Macroscope summarized f69921c.}

[github-actions]

Prime Agent performance — completed

PR f69921cb compared with main d4bc773d.

Overall: 0 regressed · 0 improved · 17 no clear change.

Metric	Main	This PR	Change
Cold startup	3,019.0 ms	3,016.8 ms	≈ -2.2 ms (-0.07%)
Warm startup	1,754.6 ms	1,753.5 ms	≈ -1.1 ms (-0.06%)
Installation	25.21 s	25.71 s	≈ +0.50 s (+1.99%)
Compressed release artifacts	11.10 MB	11.10 MB	≈ +0.00079 MB (+0.01%)
Installed footprint	597.35 MB	597.35 MB	≈ +2e-05 MB (+0.00%)
Idle memory, summed RSS	1,102.47 MB	1,102.78 MB	≈ +0.31 MB (+0.03%)

Python runtime

Metric	Main	This PR	Change
Python kernel startup	133.9 ms	130.8 ms	≈ -3.0 ms (-2.25%)
Python cell round trip	0.517 ms	0.511 ms	≈ -0.007 ms (-1.26%)
Empty bash command	12.4 ms	12.1 ms	≈ -0.3 ms (-2.25%)
Bash git status	17.5 ms	18.0 ms	≈ +0.5 ms (+2.92%)
Bash 32 KiB output	12.6 ms	12.4 ms	≈ -0.2 ms (-1.80%)
35 cells / 9 shell calls	176.9 ms	180.8 ms	≈ +3.9 ms (+2.22%)
Python interrupt to done	1.561 ms	1.583 ms	≈ +0.022 ms (+1.41%)
Python state snapshot	27.5 ms	26.7 ms	≈ -0.7 ms (-2.69%)
Python state restore	375.0 ms	382.3 ms	≈ +7.3 ms (+1.96%)
Python idle RSS	36.31 MB	36.50 MB	≈ +0.19 MB (+0.52%)
Python RSS after pandas workload	98.03 MB	98.49 MB	≈ +0.46 MB (+0.47%)

Sandbox cost: ~$0.0848 — no inference calls.
Run, logs, and downloadable raw results

Methodology and samples

Main resolved at 2026-09-10T05:40:05.677813+00:00. Harness d4bc773d.
Linux x64, 4 vCPU, 8 GB RAM, 20 GB disk; region us.
Image: node:24-bookworm@sha256:be23f54a88d34e8824c741b19b91064094f92c1c97b194144bfc8b50d67258e2.
Stock tools, skills, daemon, and Python bootstrap enabled; fresh homes and a fixed Git fixture.
Onboarding is dismissed; the editor starts without a selected model or submitted prompt.
Medians shown. Arrows require a 20% timing/memory change plus absolute floors and IQR.
These practical noise floors are not a statistical significance test.
Cold means stopped Prime processes; OS filesystem caches are not flushed.
No model requests or credentials. Installation excludes build/setup time.
Installer tarballs use loopback; npm/Python downloads use the network with fresh caches.
Artifact size counts release tarballs; footprint after first use includes registry packages.
MB is decimal. Summed RSS can double-count shared pages; PSS is recorded when available.
Provisioning, setup, and build durations are recorded separately in the raw results.
Kernel probes use the installed JSONL runtime, outside the TUI/TypeScript host.
Per trial: 50 Python cells, 5 calls per shell case, and one 35-cell mix (9 git status calls).
Cell/shell values are batch means; other runtime timings are single operations.
State fixture: a 10,000-row × 8-column integer DataFrame and a 10,000-integer list.
Restore runs in a fresh kernel, including pandas imports; kernel startup is excluded.
Kernel RSS covers the isolated Python process; loaded RSS follows the pandas workload.
Costs estimate full sandbox lifetimes at configured rates, including setup and build.
Budget target: $1; not a billing cap. Checks are informational.

Metric	Main successful/attempted	PR successful/attempted	Main spread	PR spread
Cold startup	10/10	10/10	IQR 136.3 ms	IQR 130.7 ms
Warm startup	10/10	10/10	IQR 28.0 ms	IQR 55.4 ms
Installation	3/3	3/3	range 0.70 s	range 1.36 s
Compressed release artifacts	1/1	1/1	—	—
Installed footprint	1/1	1/1	—	—
Idle memory, summed RSS	10/10	10/10	IQR 163.14 MB	IQR 32.47 MB
Python kernel startup	10/10	10/10	IQR 5.9 ms	IQR 7.5 ms
Python cell round trip	10/10	10/10	IQR 0.094 ms	IQR 0.051 ms
Empty bash command	10/10	10/10	IQR 0.7 ms	IQR 0.6 ms
Bash git status	10/10	10/10	IQR 1.4 ms	IQR 1.1 ms
Bash 32 KiB output	10/10	10/10	IQR 1.4 ms	IQR 0.6 ms
35 cells / 9 shell calls	10/10	10/10	IQR 9.3 ms	IQR 15.6 ms
Python interrupt to done	10/10	10/10	IQR 0.093 ms	IQR 0.110 ms
Python state snapshot	10/10	10/10	IQR 3.5 ms	IQR 1.1 ms
Python state restore	10/10	10/10	IQR 26.9 ms	IQR 9.8 ms
Python idle RSS	10/10	10/10	IQR 2.64 MB	IQR 3.12 MB
Python RSS after pandas workload	10/10	10/10	IQR 2.61 MB	IQR 2.20 MB