Roadmap: the order we are working in, and why

[leggetter]

This issue is the roadmap. It carries the order and the reasoning; the milestones
carry the contents. Nothing is listed twice, so nothing can disagree with itself.

• Milestones — what is in each phase
• .plans/delivery-plan.md — why the project is
  shaped this way
• LOOPS.md — what the benchmark found and whether fixing it
  worked

Where we are, 2 September

The first clean full matrix under the current prompt ran on 1 September. 114 cells,
six experiments, nineteen scenarios, all executed the same day — nothing carried, nothing
mixed, which is what #66 asks for before publishing resumes. Claude Code 19/19, GPT-5.6
18/19 in both arms, the weak pair 15 and 14. Skills delta +2 for Claude, 0 for GPT-5.6,
-1 for the weak model, so the weak model's figure has now read -3, -2 and -1 on
successive clean measurements and nobody should report that as a trend yet.

Two things came out of reading it rather than scoring it. verification-001-stripe-express
failed in four of six arms, three on the same check, which is the shape a scorer defect
makes and wants a triage read before the numbers go public. And the delta does not live
anywhere stable: 9 of 57 paired cells differ on that run, but 14 of 19 scenarios have
disagreed at some point across the published history — so the disagreements move, and a
targeted re-run of "the cells where skills matter" cannot be cheaper than the full matrix
because there is no such stable set.

The page is live. /evals and /evals.md publish from a release rather than from
whatever ran last, so an automated run can no longer move the public numbers.

Three releases cut. v0.1.0 the baseline, v0.2.0 a correction: eight of the twenty
failures published in v0.1.0 turned out to be our own scoring, in two scenarios, both
cases of a scorer asserting how an outcome was reached rather than whether it was.
Corrected, frontier agents came out level at 3 failures of 30 each — which closed the
"nearly every failure is Codex" finding as not confirmed.

The leak is contained and the page is live. Publishing is not finished: the release
helper is unbuilt. The two gaps that shipped knowingly — the page never saying what the
paired columns mean, and the weaker model published without a caption — were closed as
not planned on 29 August. The page carries the scoreboard; anyone wanting method comes
here. #29 and #30 carry the reasoning.

Publishing is held right now, and the next thing anyone does should account for it.
#62 put "Do not ask clarifying questions" into the base prompt on 27 August — the one
string every cell shares — so every row measured before then was measured under a
different treatment. EVALS_PUBLISH=false until a snapshot exists that was measured end
to end under the current prompt. #66 carries the reason and the condition for clearing
it; do not clear it from memory of this paragraph.

Three days of instrument work sit behind that. The order they happened in matters,
because each one found the next:

1. Six issues fixed on 25 August had stayed open — both PRs referenced them in headings
   rather than closing keywords, so milestone 4 read as eleven items when it was five.
2. The weak-model skills delta was −2, not −3. Both −3 readings are from 13 August,
   the credit-outage day, and no run since reproduced them.
3. Chasing an apparent skills win in verification-002 found it was one execution
   republished into seven snapshots. That produced A published snapshot mixes execution dates, so counting snapshots overcounts measurements #60.
4. Chasing that found the arms reach documentation differently — which produced The -no-skills arm reaches documentation differently, so part of the skills delta is not skills #61,
   and then a correction to The -no-skills arm reaches documentation differently, so part of the skills delta is not skills #61: split per experiment, both Claude arms sit at 0% empty
   searches and every Codex arm between 26% and 83%. The dominant variable is the agent,
   not the arm, and a pooled aggregate hid it.
5. An agent that stops to ask a question scores zero, and we do not track how often #57 was decided by surveying six benchmarks giving five different answers, then
   measured: the detector went 4 → 0 for about $3.

The through-line is that every one of those was an instrument defect wearing the costume
of a result, and each was found by looking at a row rather than at a rate. That is the
same lesson as Loops 1 and 2 and it has now happened often enough to expect.

The order, and why it is this order

Reordered on 20 August. The page is live and the numbers behind it are trustworthy enough
to publish, so the work that matters now is what the benchmark is for: covering the
product properly, and settling whether the skills earn their place. Trust and publishing
work continues behind those.

1. Cover the product

Outpost went from one scenario to five, published as v0.3.0, and the runs against them
showed the coverage was the easy half. Console still has none, and no stage but build has
more than four.

Rescoped on 1 September: this was never an Outpost problem. scenario-criteria
(#72) counts what CONTRIBUTING.md has asked for since before the first scenario existed —
a cited motivation, and an observed failure before a scenario joins the published
benchmark. Neither had ever been checked. Five scenarios have never been failed by any
agent, three of them Event Gateway, and no motivation carries a citation. #47 now covers
all five.

The five are being sorted from transcripts rather than from argument, which is the only
route that has produced discriminating scenarios here. dedupe-001 is the clearest case:
agents split between keying deduplication on the full request body and on body.id, and
both pass because the scorer sends byte-identical copies — so half of them built something
that silently does nothing against a provider that varies a field between retries.

First because the page is live and the Product pivot names a per-product score. The number
most likely to be wrong in a way that matters is no longer "a product scored from one
test" — it is a product scored from tests that pass regardless.

2. Prove the skills earn their place

Skills must improve the product or there is no point having them.

Currently measured at +1 for Claude, 0 for GPT-5.6 and −2 for the deliberately weak
model — so they help where the agent is strong and hurt where it is not, and nobody knows
why. This paragraph said −3 until 26 August. Both −3 readings are from 13 August, the
credit-outage day, and the eight runs since read −2 (one read 0), so the sign is
replicated and the magnitude was not. #2 carries the recomputation.

The composition matters more than the number and has changed completely.
transform-001-reshape-payload is worse with skills in all ten published runs, which no
other scenario manages. The other three that made up the original −3 dissolved into no
difference, and verification-002 — the scenario the shared-cause theory was built on —
inverted to a skills win that has held for eight consecutive runs, with no skill change
to explain it (the submodule pin has not moved since 12 August).

The one controlled measurement since went the other way from what its first run claimed. A
24-cell Outpost run put the delta at 2 cells in 24, not the 9–0 an earlier run
suggested — because that earlier run was measuring a harness omission of ours, and a
+skills arm that had never been given the Outpost skill. Expect that shape again: most
of an apparent skills win is our own instrument until proven otherwise.

Settling it means both arms at three attempts on the current scorers — about $64, and the
run this milestone exists for. Worth saying in advance: if the answer is that skills are
neutral for frontier agents and negative for weaker ones, that is a finding and it gets
published. The alternative is shipping something because we built it.

Three things stood between that run and an interpretable answer, all found after this
milestone was written. One is now decided; two remain. Running before they are settled
spends the better part of $100 on a number that cannot be read:

• The -no-skills arm reaches documentation differently, so part of the skills delta is not skills #61 — the arms do not read documentation the same way. The baseline makes 3.6× the
  documentation calls, 77% of them web_search, and 58% return nothing. -no-skills is
  supposed to differ from its twin by skills and nothing else; it also differs by whether
  the agent fetches a docs URL or queries a search index. Part of the delta is that.
• An agent that stops to ask a question scores zero, and we do not track how often #57 — an agent that stops to ask scores zero. Decided 27 August, in Tell agents not to ask clarifying questions #62. The
  base prompt now carries Clerk's line — "Do not ask clarifying questions. Complete the
  task with the information provided." — and the ASKED_AND_STOPPED detector stays, so
  the instruction removes the confound and the detector reports whether it worked. Clerk
  instructs without measuring and upstream does neither; doing both is available to us
  because triage: flag an agent that stopped to ask a question #58 shipped the detector first. The decision was taken after surveying six
  benchmarks giving five different answers, recorded on An agent that stops to ask a question scores zero, and we do not track how often #57. It is not yet known whether
  the instruction changes behaviour — the four known cells get re-run before the matrix
  does.
• A published snapshot mixes execution dates, so counting snapshots overcounts measurements #60 — a snapshot is not a run. Rows are carried forward, so the frontier models'
  -no-skills arm in the current snapshot is from 13 August — before the CLI bump, the
  polling fixes and several scorer corrections. A three-attempt run compared against
  carried-forward baselines answers nothing.

The run itself is now costed at $90–$110 rather than $64: nineteen scenarios rather
than fifteen, and --runs 3 at about 1.11× because attempts stop at the first pass.
Preferring more attempts to more instruction is now a convention in AGENTS.md — it is the
cheaper correction and it does not touch the one string every cell shares.

#61 and #60 both came out of chasing a single scenario in #2 that looked like a clean
skills win and was one execution republished seven times. That is the third time an
apparent skills result has dissolved under a control, which is the reason this milestone
is worth doing properly rather than quickly.

3. Finish publishing

The page is live and publishes from a release rather than from whatever ran last. The
release helper is half built (#13), and publishing is held until a snapshot exists that was
measured end to end under the current base prompt (#66).

The two explanatory gaps are closed as not planned. #29 was described here as a
prerequisite for reading the milestone-2 result, and it is not. The redesign labels the
arms Baseline and With skills, clearer than -no-skills/+skills was; a reader
wanting to know how the arms differ comes to this repository. #30 went the same way for a
sharper reason: the case for captioning the weaker model rested on it outranking a frontier
agent, which turned out to be arithmetic on the website rather than anything a caption
would have fixed.

What replaced them is a question about the numbers rather than the copy. The redesign
briefly scored the page on checks passed instead of scenarios completed. Our scorers stop
as soon as there is nothing left to check, so a total failure returns 0/1 where a
near-miss returns 4/5 — failing worse is cheaper than failing partially. On the 25 August
snapshot that ranked the deliberately weaker model above a frontier agent, 97% to 95%,
where by scenarios they tie at 89%. The definition now lives in AGENTS.md rather than only
in the consumer that renders it.

4. Trust the numbers

Substantially done, and more so than the milestone showed until 26 August: six
issues fixed by #55 and #56 on 25 August stayed open because both PRs referenced
them in headings rather than closing keywords, so the phase read as eleven items
when it was five.

Substantially done. Three independent causes of moving results were found and handled:
fixed sleeps against asynchronous ingestion, a rubric that contradicted its own scenario,
and configuration readable before it is enforced.

Substantially done for the three causes found before 20 August. Three more surfaced in
the Outpost runs and none of them is theoretical: a killed container is still scored as an
agent's answer, an agent can wipe shared deployment config for every later run, and
nothing detects a run that reads this repository. A project reset that adopts leftovers as
pristine and a silently drifting submodule pin remain.

5. Sustain

Cadence and budget, durable transcripts, cost reporting, and the product findings this
repository cannot close itself (#25).

What this order is not

It is not a schedule and has no dates. The later phases can move in parallel where
they do not collide.

It is also not a commitment to do everything. An issue that stops being worth doing should
be closed with the reason — #3 was closed as a duplicate of #19, and #4 as not confirmed.

Keeping this current

Update this issue when the reasoning changes: a phase reordered, dropped, or finished.
Do not list individual issues here; that is what milestones are for, and a list kept in
two places goes stale in one of them. This repository has learned that three times.

[leggetter]

28 August, later: the page is frozen while it is redesigned

The website team is iterating on the /evals UI, so changes to the published page are held until that work lands. That has a consequence for the order above, and it is not a small one.

#29 is named in "3. Finish publishing" as a prerequisite for anyone reading the milestone-2 result: a reader who does not know what +skills and -no-skills are cannot read a skills delta at all. Both #29 and #30 are now written and verified — https://github.com/hookdeck/website/pull/771 — and held as a draft rather than merged.

So one of three things has to happen before the milestone-2 result is published, and it is worth choosing rather than discovering:

1. the redesign carries the pairing explanation and the weaker-model marker itself;
2. the held PR merges after the redesign lands, before the result publishes;
3. the result publishes without them, which is the option that gets screenshotted.

Nothing about this blocks the run. It blocks reading it in public.

Also today

• Five benchmark scenarios have never been failed by any agent #47, first piece. outpost-001 was passing on a receiver of the agent's own choosing — the ticket named no endpoint, so six cells used six different receivers and the single failure was a localtunnel that died with its container, not a weak model. Now scored against an endpoint the ticket names: Score outpost-001 against the customer's endpoint, not one the agent picked #67. Verified with score-only on three paths rather than an agent run. This makes "does 001 discriminate" askable; it does not answer it.
• pnpm check has been broken since d69e572 and nobody noticed, because CI runs its steps directly rather than through it. Four scripts pointed at deleted files. Stop pnpm check from running scripts that no longer exist #68.
• The plan and the page brief were audited against the repository and six stale claims corrected, including a −3 skills delta that has read −2 since 26 August and a weekly costing that assumed twelve scenarios and arms we do not run. Bring the plan's status, counts and costs back to what is true #69.

[leggetter]

Correction to yesterday's note: the page freeze is not a blocker, and #29 is not a prerequisite

I wrote that #29 — the page never saying what the arms are — had to be settled before the milestone-2 skills result could be published, and listed three ways to get there. That is no longer the position.

Decided: the page carries the scoreboard and nothing else. Anyone wanting to know how the two arms differ, or why a weaker model is in the matrix, comes to this repository. Both issues closed as not planned, with the reasoning on each. The redesign relabels the arms Baseline / With skills, which is clearer than -no-skills/+skills was and is judged sufficient on its own.

So milestone 3 loses two of its four open items and the page redesign (https://github.com/hookdeck/website/pull/778) is going in as it stands.

What replaces it as the thing to settle before publishing is narrower and is about the numbers rather than the copy. The redesign changes the headline from scenarios passed to checks passed, and on the baseline arm that inverts the agent ranking: GPT-5.4-mini reads 97% against Claude Code's 95%, where by scenarios they tie at 89%. The mechanism is that a scorer which finds nothing to score returns one failing check and stops, so a total failure costs 0/1 while a near-miss costs 4/5 — failing worse is cheaper than failing partially.

supabase/evals computes its rate as passed / results.length, scenarios not checks, which is the shape ours had until now. Options and evidence are on the website PR.

[leggetter]

Decisions taken 29 August, and where each is recorded

Four calls were made on the page and the numbers behind it. Recording them together because two of them reverse earlier positions of mine on this issue, and one is being made for the second time.

Decision	Why, in short	Recorded in
The page carries the scoreboard and no explanatory prose	Deeper information belongs in this repository; the redesign's Baseline / With skills labels are clearer than -no-skills/+skills and enough on their own	#29 (closed), https://github.com/hookdeck/website/pull/779 (closed)
The deliberately weaker model is not captioned	The model is called GPT-5.4-mini, the same vendor holds first place as well as last, and supabase/evals captions nothing	#30 (closed), https://github.com/hookdeck/website/pull/781, and a comment in scoreboard.ts next to the column ordering
The published score counts scenarios, not checks	A scorer stops when there is nothing left to check, so a total failure returns 0/1 and a near-miss 4/5 — failing worse became cheaper than failing partially, and the weaker model outranked a frontier agent 97% to 95% where by scenarios they tie	#71 (AGENTS.md), https://github.com/hookdeck/website/pull/780
Ranked ties break by name, on the unrounded share	Two ties are live; arrival order changes between snapshots, so agents would swap places with no result behind the move	https://github.com/hookdeck/website/pull/780

The second one is why the third is recorded here rather than only on the website. #30 existed because a previous change removed a visible marker and left isWeakerModel, a stylesheet rule and a docstring describing a badge — so the next reader found a feature that was computed, styled, documented and absent, and filed it as a bug. A decision recorded in one place, in code that outlives it, gets re-derived. The scoring definition belongs to this repository because this is where the number is produced; the website is a consumer.

Correcting myself on this issue twice over. I wrote that #29 gated publishing the milestone-2 skills result: it does not, it is closed. And I proposed captioning the weaker model as a fairness fix: the case for it rested on the weaker model outranking a frontier agent, which turned out to be the check-level arithmetic rather than anything a caption would have fixed.