Skip to content

Review prompts: swap the lens list for enumerable moves and an absence pass - #21

Closed
latekvo wants to merge 1 commit into
mainfrom
review-moves
Closed

Review prompts: swap the lens list for enumerable moves and an absence pass#21
latekvo wants to merge 1 commit into
mainfrom
review-moves

Conversation

@latekvo

@latekvo latekvo commented Jul 28, 2026

Copy link
Copy Markdown
Owner

Problem

The depth fragments fanned the swarm out over kinds of bug - "correctness, scope/simplification, edge cases (nulls, boundaries, errors, concurrency) and ripple effects on callers/docs/tests". That list is open-ended, and its coverage cannot be audited: an agent can report a lens swept and still have missed everything in it.

It did. On argent#574 a max-depth sweep returned two test-scaffolding notes plus a body claiming the concurrency, ripple/call-site and device-scoping lenses "held up throughout". A human reviewing identical source an hour later (the only intervening commit touched one test file) found three real defects, all in those lenses, all accepted and fixed: a writer-only lock whose readers can observe a truncated flow file and report a top-level PASS having replayed zero steps; a second device to services matcher disagreeing with the existing one on case-sensitivity and namespace set; and an unmatched field reporting real device ids as typos.

Approach

The fragments now fan out over review moves - acts a reviewer performs, one agent per move - matching the rewritten gates in CLAUDE.md:

claims vs code · nearest twin · non-happy paths · inputs · reachability both ways · what outlives the call

plus a separate absence pass, because roughly half of all real findings are things that are missing and you cannot grep for what does not exist. Absence only shows against a reference, so the pass names the four that work: a sibling that has it, prose that promises it, symmetry (spawn/reap, open/close, write/delete, set/reset), and mutation.

The moves are not invented. They were induced from a corpus of 231 real findings across 27 argent PRs: two agents given only the corpus, with no sight of any candidate framework, produced near-identical taxonomies. Three adversarial passes then broke the first draft, which scored only ~40-50% coverage and had no hook for absence at all. Mutation alone accounts for 33-41 of the 231 and had no home in the old lens list.

Naming one agent per move also keeps the swarm inside CLAUDE.md's sizing limits instead of leaving the count implicit.

Also fixed

Each of these was found by running this change against its own gates.

  • softApprove's example was the bug. It suggested posting "Ran full E2E sweep across the review lenses. Returned perfectly clean." - the exact unearned all-clear that fired on #574 - while also contradicting CLAUDE.md's no-LGTM rule it claims to be the one sanctioned exception to. It now has to state the moves covered and the reproductions run, and names bare impressions as banned.
  • finalPass asked for "massive swarms", contradicting the hard cap of 14 in CLAUDE.md. Defers to those limits now.
  • leaveReviews' no-duplicate rule had no floor, so a PR body pre-listing known limitations read as prior coverage. An acknowledged limitation is now explicitly not a resolved finding.
  • The standard and deep blurbs still advertised "review lenses" after the fragments stopped using them.

Verification

  • DIPLOMAT_GOLDEN_WRITE=1 swift run DiplomatCoreSmoke to regenerate; the three affected goldens are in the diff. Re-run without the env var: SMOKE OK, golden-prompt assertions pass across all 10 modes.
  • swift build --product diplomat-core clean.
  • pytest linux/tests with DIPLOMAT_CORE_BIN and QT_QPA_PLATFORM=offscreen: 195 passed, including test_golden_prompts.py (byte-for-byte Swift/Python parity) and test_review_author.py. The mesh suite is excluded locally - it blocks on getaddrinfo in a sandbox with no DNS - and is left to CI.
  • Read the regenerated prompts end to end to confirm assembly order and that no stale "lens" wording survives anywhere in review.json or the goldens.

Notes for reviewers

  • Prompt text only. No Swift or Python logic changes, so the assembly layer is untouched and the goldens are the whole behavioral surface.
  • The moves are 10 items where the lenses were 4. That is deliberate: they sit under two headings instead of four unrelated categories, they are acts rather than categories, and the corpus says the old four were not covering the ground.
  • Known limits, from the adversarial passes: about a quarter of the corpus needs judgement no enumeration supplies - a move aims you at the file, you still have to recognise the defect. Four findings in 231 (two style opinions, a reviewer retraction, a self-answered question) are reachable by no method at all.
  • The moves overlap, so "I enumerated set X" is not an auditable claim. That is why the reporting rule asks what you ran and compared against rather than which set you covered.
  • Diplomat.app/Contents/Resources/core/review.json is a build artifact copied by scripts/build-app.sh and is untracked; only core/review.json is edited here.

…e pass

The depth fragments told the swarm to cover "the standard lenses -
correctness, scope/simplification, edge cases and ripple effects". Those
name kinds of bug, so an agent could report a lens swept and still have
missed everything in it. On argent#574 a full sweep at max depth returned
two test-scaffolding notes and a body claiming the concurrency, ripple and
device-scoping lenses "held up throughout"; a human reviewing the same
source an hour later found three real defects in exactly those lenses.

The fragments now fan out over review moves - acts, not categories -
matching the rewritten CLAUDE.md: claims vs code, nearest twin, non-happy
paths, inputs, reachability both ways, and what outlives the call, plus a
separate absence pass. The moves were induced from 231 real findings across
27 argent PRs, and one agent per move keeps the swarm inside the sizing
limits rather than leaving the count implicit.

Also fixed, all found while checking this change against its own gates:

- softApprove's example was "Ran full E2E sweep across the review lenses.
  Returned perfectly clean." That is the exact unearned all-clear that
  fired on #574, and it contradicted CLAUDE.md's no-LGTM rule while
  claiming to be its one sanctioned exception. It now has to state the
  moves covered and the reproductions run, and bare impressions are named
  and banned.
- finalPass asked for "massive swarms", contradicting the hard cap of 14
  in CLAUDE.md. It now defers to those limits.
- leaveReviews' no-duplicate rule had no floor, so a PR body that
  pre-lists known limitations read as prior coverage. An acknowledged
  limitation is now explicitly not a resolved finding.
- The standard/deep blurbs still advertised "review lenses".

Golden prompts regenerated with DIPLOMAT_GOLDEN_WRITE=1 swift run
DiplomatCoreSmoke.
@latekvo
latekvo marked this pull request as ready for review July 28, 2026 16:53
@latekvo
latekvo marked this pull request as draft July 28, 2026 17:10
@latekvo

latekvo commented Jul 30, 2026

Copy link
Copy Markdown
Owner Author

bollocks

@latekvo latekvo closed this Jul 30, 2026
@latekvo
latekvo deleted the review-moves branch July 31, 2026 14:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant