scripts/benchmark.py: cross-commit benchmark harness

[tony]

Summary

Adds the cross-commit benchmark harness and refreshes several docs surfaces. Three categories of change in one PR — happy to split if you'd prefer separate reviews, but they're tightly related (the harness IS the dev-tooling page that motivated the Development section).

Benchmark harness (scripts/benchmark.py)

• New PEP 723 self-contained harness (typer + rich + pydantic) for hyperfine-driven benchmarks across HEAD, trunk, ranges, lookback, from-trunk-back, tags, or explicit commit lists. Subcommands: `run`, `compare`, `list-commits`, `list-commands`, `show-config`.
• Output formats: rich table (default), JSON, NDJSON, Markdown, CSV. Raw samples preserved in machine-readable outputs so downstream tooling can compute its own percentiles.
• TOML config layering — `scripts/benchmark.toml` ships the agentgrep defaults; `scripts/benchmark.local.toml` (gitignored) carries per-machine overrides; CLI flags trump both.
• Safety guards: dirty-tree refusal (`--allow-dirty` to override), atexit + signal trap restoring HEAD on every exit path (incl. SIGINT), pre-flight `--output` path validation, pydantic-validated `Settings` bounds (`runs >= 1`, `warmup >= 0`).
• Hyperfine preferred with `-N` (no shell). Auto-fallback to pure-Python `time.perf_counter` when hyperfine isn't on PATH or `--no-hyperfine` is passed.

The script surfaced a real +7% cold-start regression at the query-language landing commit on the cli-normalization branch — that's what motivated promoting the prototype `/tmp/bench-*.sh` to a first-class repo tool.

Docs reorganization

• New `docs/dev/` section for contributing/internals references. `scripts/benchmark.py` is dev-only (not part of the published wheel), so its reference page lives under Development. `docs/storage-catalog.md` relocated to `docs/dev/` alongside it — it's an internals reference for adapter authors.
• `docs/history.md` now uses the standard `{include} ../CHANGES` pattern to render the changelog inline; previously it was a stale "no releases yet" stub even though four releases had shipped.
• Sidebar flattened to the libtmux pattern: one entry per section with child pages expanded under it, instead of seven caption-grouped flat lists.
• Top four landing pages (Quickstart, Installation, MCP Clients, Configuration) collapsed into one Getting Started section. The quickstart page is now `docs/getting-started/index.md` (the section landing); installation / clients / configuration are children.

Performance

• Preload `IBM Plex Mono 400 italic` in the page ``. Furo's Pygments style renders comment tokens (`.c`, `.c1`, `.cm`) italic, so every code block with a `# comment` was lazy-loading the face. Fixed locally on agentgrep; an upstream PR (git-pull/gp-sphinx#43) adds the same preload to `DEFAULT_SPHINX_FONT_PRELOAD` so every gp-sphinx consumer benefits.

Test plan

• `uv run ruff check .` passes
• `uv run ruff format --check .` passes
• `uv run ty check` passes
• `uv run pytest --reruns 0` passes (272 tests, including 26 new `tests/test_benchmark_script.py` cases)
• `just build-docs` passes; redirects render for `/benchmark/`, `/storage-catalog/`, `/quickstart/`, `/installation/`, `/clients/`, `/configuration/`
• Benchmark harness verified across 5 sweeps and ~160 distinct invocations covering every subcommand, target selector, output format, real timing path, and error/safety path
• Sidebar verified locally — top-level sections expand to children; redirects from old slugs land on new pages

Notes

• The CHANGES entry for these additions will land via the `cli-normalization` branch's merge, where the `0.1.0a5` placeholder already exists. Happy to add a standalone CHANGES update on master if you'd prefer it lands in this PR.
• The font preload override in `docs/conf.py` becomes redundant once gp-sphinx#43 merges and a new gp-sphinx version is pinned. Easy follow-up to remove the local override at that point.

[tony]

Code review

Found 2 issues:

1. Module docstring directs readers to docs/library/benchmark.md, which doesn't exist on this branch (the file is at docs/dev/benchmark.md after the move-to-Project commit). Anyone reading --help output or the source header and following the link will hit a 404.

agentgrep/scripts/benchmark.py

Lines 15 to 20 in 8bfedfd

	be overridden per-machine via ``scripts/benchmark.local.toml`` (gitignored).
	Run ``uv run scripts/benchmark.py --help`` for the subcommand list, or see
	``docs/library/benchmark.md`` for invocation recipes.
	"""

2. Status literal advertises a "sync_fail" value, but no code path can ever produce it. _maybe_sync calls subprocess.run(..., check=False) and returns None; the caller in _run_one_commit discards the result. A failing uv sync is silently absorbed and the benchmark proceeds against a potentially broken venv — the user never sees the failure marked as sync_fail even though the Status type promises it.

agentgrep/scripts/benchmark.py

Lines 48 to 52 in 8bfedfd

	LOCAL_CONFIG = REPO_ROOT / "scripts" / "benchmark.local.toml"
	Status = t.Literal["ok", "checkout_fail", "sync_fail", "command_missing", "bench_fail"]
	OutputFormat = t.Literal["rich", "json", "ndjson", "md", "csv"]
	PERCENTILE_LABELS: tuple[str, ...] = ("min", "max", "avg", "p50", "p90", "p95", "p99")

agentgrep/scripts/benchmark.py

Lines 640 to 652 in 8bfedfd

	def _maybe_sync(settings: Settings, repo: pathlib.Path) -> None:
	if not settings.sync_command.strip():
	return
	subprocess.run(
	shlex.split(settings.sync_command),
	cwd=repo,
	check=False,
	capture_output=True,
	text=True,
	)

agentgrep/scripts/benchmark.py

Lines 708 to 713 in 8bfedfd

	return results
	if not no_sync and not dry_run:
	_maybe_sync(config.settings, repo)
	venv = (repo / config.settings.venv).resolve()

🤖 Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}