Roadmap: Behavioral A/B benchmark (stock vs lobotomized-CC)

[dividedby]

Parent

PRD: release-adoption control plane — #2 (slice 5, roadmap)

What to build

The value-proving track (ADR 0002): a Behavioral A/B benchmark comparing stock CC vs lobotomized-CC — same version/model/effort/prompt — scored on the behavioral axes the Lobotomy targets via an LLM judge (paired, randomized order), with a Correctness guardrail. Evidence, not a gate; it feeds the Behavioral A/B field of the Adoption record.

Hybrid home: ~/repos/bench exposes its run/judge/aggregate as library primitives (a leaf refactor in bench); the tweakcc-specific behavior-bait fixtures, the behavioral rubric, and the A/B driver live in tweakcc-maint. Driver seam mirrors the gate: the LLM judge sits behind a stubbable port so pairing/randomization/aggregation are tested without real model calls.

Tracking placeholder — needs its own design pass (and the bench refactor) before AFK-ready.

Acceptance criteria

• Triaged: scope the bench library refactor + the behavioral rubric/fixtures.

Blocked by

None - roadmap, needs triage.

🤖 Generated with Claude Code

[dividedby]

This was generated by AI during triage.

Human Brief

Category: enhancement
Summary: Build the Behavioral A/B benchmark (stock CC vs lobotomized-CC) that proves the fork's value claim (ADR 0002) and feeds the Adoption record's Behavioral A/B field.

Why not an AFK agent: the load-bearing prerequisite is a leaf refactor in ~/repos/bench (exposing its run/judge/aggregate as library primitives) — a separate repo — plus a design pass on the behavioral rubric/fixtures and a live LLM judge. Cross-repo leaf work + evaluation design + real model calls put this beyond a clean AFK slice until the design is settled.

Current behavior:
No value-proving track exists; the AdoptionRecord has no Behavioral A/B field populated. ~/repos/bench has run/judge/aggregate scripts but not reusable library primitives.

Desired behavior:
A paired, order-randomized A/B comparing stock vs lobotomized-CC at the same version/model/effort/prompt, scored by an LLM judge on the axes the Lobotomy targets, with a Correctness guardrail. The bench primitives are reused as a library; the tweakcc-specific fixtures, rubric, and A/B driver live here. The judge sits behind a stubbable port so pairing/randomization/aggregation are testable without real model calls. Output feeds the Adoption record's Behavioral A/B field — evidence, not a gate.

Human decisions to make:

• Scope the ~/repos/bench library refactor (coordinate as a leaf change).
• Design the behavioral rubric and behavior-bait fixtures (judgment-heavy; the heart of ADR 0002).
• Decide the judge model/config and the Correctness-guardrail criteria.

Acceptance criteria:

• Bench library refactor scoped (and prepared as a leaf change).
• Behavioral rubric + fixtures designed; A/B driver with a stubbable judge port specified.
• A follow-on agent slice (the driver + aggregation behind the stub) can be split out once the above are settled.

Out of scope:

• Making this a pass/fail gate (it's evidence; the Four-zeros bar is the gate).
• Implementing the driver before the bench refactor + rubric design are settled.

[dividedby]

Burn-down order 4 — decision session (2026-06-10). Parked since filing; the burn-down forces an explicit fate instead of a zombie row. Maintainer opens a session and says "work #11 with me".

Pickup protocol for the agent:

1. Read ADR 0002 first (the vs-vanilla benchmark measures TARGETED behavior, not general quality) — the design stance is already recorded; don't re-derive it.
2. Ask the maintainer the fate question with a recommendation: (a) activate now → run /grill-with-docs against CONTEXT.md/ADR 0002 to scope a minimal v1 (e.g. one targeted behavior, one A/B pair on the live install), then /to-prd + /to-issues; (b) keep parked with a named revisit trigger (e.g. after Post-backlog: complete refactor plan from a fallow static-analysis pass #102's refactor plan); (c) wontfix and close. Note honestly: activating SPAWNS issues — it grows the backlog before it shrinks it.
3. Execute whichever branch is chosen in the same session; update the roadmap row to match.