Keep state correctness, policy compliance, workflow effort, and time/cost separate. Execution status is a separate prerequisite: a CLI error cannot pass just because the repository happens to be correct. There is no blended performance score. Compare the same task/model/configuration, use successful runs for efficiency claims, and keep cold adoption separate from warm work.
result.json.performance contains:
| Dimension | Recorded evidence |
|---|---|
| Execution | Completion, runtime failure, exit status, timeout |
| Correctness | Each state invariant; passed/total checks; protected-work checks |
| Compliance | Observed protocol violations, independent of state correctness |
| Effort | Task VC reads/writes/errors; model tool calls; parser-call heuristic; repeated identical reads; selector failures; VC and model-visible output bytes |
| Time/cost | Wall time, first task command, first successful mutation, summed and occupied VC runtime, API time/turns/cost when exposed, provider-specific token categories |
First task action means the first task VC command in the wrapper trace. Model events without timestamps provide order, not per-turn latency. A successful mutation means command exit zero, not proof of progress. Review tasks legitimately have no mutation. Protected-work checks and the complete named invariants remain available; a fraction of passing checks is a diagnostic, not a calibrated quality score.
Both adapters collect structured CLI output: Codex exec --json, Claude --output-format stream-json --verbose. agent-stdout.txt retains the raw stream, agent-events.json normalizes tool calls/results and final messages, and agent-output.json retains the terminal result. Intermediate lifecycle updates do not multiply tool-call counts. Missing terminals, malformed data, incomplete tool results, and unsupported event types have explicit coverage flags.
The Git/But/jj wrappers stream command stdout/stderr unchanged and also save them in command-output/. Trace v3 retains exact argv, cwd, timestamps, exit status, signal, byte counts and artifact paths. trace-buckets.json records task/platform/tool-internal classification. Native tool subprocesses remain separate from agent work. These files are included in the existing scanned evidence export.
-
The legacy
task_errors/ failed-command counter counts nonzero VC exits. Expected probes such asgit diff --quietreturning 1 for a dirty file are included; inspect the command before calling an exit an agent mistake or CLI error. It is not the number of failed model tool calls. -
Read/mutation counts classify command forms; they do not measure actual writes. A zero exit does not prove progress, and some preview forms still fall into the mutation class.
-
Parser calls count observed shell tool invocations of common parsing/filtering programs. This is a labelled heuristic: a parser can be the appropriate tool. It counts calls, not every program in a pipeline.
-
Repeated reads mean consecutive standalone unquoted VC commands with identical full arguments and cwd. Simple CLI shell wrappers are decoded statically. Every other tool call resets the window; compound commands are excluded. This is a conservative observed repetition count, not a judgement that a query was unnecessary.
-
Selector failures recognize selector-related error messages in captured stderr, not words in the attempted command. Expected negative probes can also match; inspect the retained error before interpreting the count. Historical runs without that output are unknown.
-
VC stdout/stderr bytes measure wrapper output. Model tool-output bytes measure visible tool results, including non-VC calls. Structured output uses serialized UTF-8 bytes. Outputs may be truncated by an agent CLI; neither field measures hidden context or reasoning.
-
Explicit skill reads can be attributed by their tool-call path. Implicit instruction loading remains outside that counter.
-
Summed VC runtime can overlap. Occupied VC runtime merges overlapping intervals. Neither can identify all model latency by subtraction: file reads, tests, startup and network waits also occupy wall time.
-
Claude exposes API time, turns, cost and cache/input/output usage. Codex exposes its usage categories; unavailable cost/API-time/turn fields remain null. Codex JSONL currently does not expose an observed model ID, so configured and observed model fields remain distinct.
Missing measurements are null, never fabricated zeros. The former hardcoded parser count, argv-based selector heuristic, and coarse repeated-query counter are retired. Measurement schema 2 retires the old total_bytes/warm_estimated_total_bytes transcript proxies; actual transport, visible event and tool-output sizes have explicit names. Historical Claude final-answer JSON cannot reconstruct intermediate events. Its final-answer bytes are not substituted for context consumption.
Matrix runs automatically write performance/report.md and performance/report.json alongside the legacy matrix report. An explicit selection can also be analyzed without running agents:
npm run performance:report -- \
--runs tmp/pilot-runs/measurement-smoke-codex,tmp/pilot-runs/measurement-smoke-claude \
--out tmp/pilot-runs/measurement-scorecard-smokesFor a larger selection, use --manifest selection.json containing {"runs":["relative/run-dir"]}. Paths are relative to the manifest. Do not glob all historical runs: that mixes smokes, invalidated attempts and reference solutions.
The reporter recomputes diagnostics from available evidence using current code and records hashes of the measurement sources. It preserves original results. A corrected-verifier.json is applied only after checking the original response and verifier-source hashes; it cannot pardon a protocol or execution failure. The raw and effective grade remain distinguishable.
npm run runner:check: guard streaming/binary roundtrip, event parsing, full adapter stubs, incomplete/error results, output caps, descendant cleanup, scorecard/correction integrity and matrix integration pass.- The final seven-task reference smoke passed across Git/But/jj in about 10 seconds.
- Native But synchronization and integration references passed in 2.6s and 2.1s, checking that piped native Git operations still work.
- Real Codex and Claude cleanup smokes passed in 21.9s and 15.0s. Both expose three model tool calls and one parser call. The current scorecard recomputes those counters from the captured events; the smoke processes started while the counter implementation was being finalized.
- Both smoke evidence exports succeeded, including intermediate events and command outputs. No publishing was performed.
- The explicit historical 14-run scorecard preserves unavailable fields and the corrected Claude review grade. No full matrix was rerun.
Local evidence: current smoke scorecard, historical scorecard.
For optimization, change one tool/skill behavior, then run a small matched before/after batch on the affected task. Keep fixture, task prompt, model and other settings fixed. Use several repetitions to distinguish a repeatable improvement from a fast single sample; k=1 remains useful for finding setup problems.