Quality gate certifies downgrades against an unmeasured baseline (no SE-padding); a cheaper tier is admitted with zero baseline observations

[scarson]

Summary

The conservative gate in engine.recommend() admits a downgrade when the cheaper tier's Wilson lower bound clears mu_star − q_tol, where mu_star is the baseline tier's posterior mean. On any cell where the baseline tier has never been dispatched, mu_star is just the cold-start prior (Beta(8,2) = 0.80), so the gate certifies the cheaper tier against 0.75 (Moderate) rather than against the baseline's real quality. Nothing in the decision path conditions on how many times the baseline was actually observed, so the gate can't distinguish "baseline unknown" from "baseline known-mediocre" — and it will return a "quality preserved vs the baseline" recommendation with zero baseline evidence.

Found this while studying the design (it's genuinely nice work). Flagging it because the pack is up for a community-pack PR, where an installed gate that makes unsafe certifications is worth catching first.

Where

• modeladvisor/engine.py:199 — mu_star = store.pooled(...).mean (a point mean; at cold start this is the baseline_a/baseline_b = 0.80 prior).
• modeladvisor/engine.py:201 — gate_threshold = mu_star - klass.q_tol (no term reflecting baseline n or uncertainty).
• modeladvisor/engine.py:282 — admission is exactly q_lo >= gate_threshold.

The paper this implements (Thm 5.3) states the guarantee Pr[θ_cheap ≥ θ_baseline − τ] ≥ 1−α, and notes it relies on approximating mu* ≈ θ_baseline, kept safe at small n by padding — "the production implementation conservatively pads µ* by one posterior standard error at small n*." That padding isn't in this engine (engine.py:201). I can only check the public pack; if the padding lives in a non-public build, treat this as "the public pack is missing it."

Reproduction

Run python repro.py from the pack root (the script below, attached):

"""Repro: the quality gate certifies downgrades against an unmeasured baseline.
Run from the model-advisor pack root:  python repro.py"""
from modeladvisor import config as C, engine as E, store as S
from modeladvisor.store import cell_key

cfg = C.default_config()                 # haiku/sonnet/opus, baseline opus
prov = cfg.default_provider

def feed(st, tier, s, f):
    key = cell_key(prov, "polecat", "implement", tier)
    st.apply_records(
        [{"kind": "quality", "cell_key": key, "q": 1, "channel": "close"} for _ in range(s)]
        + [{"kind": "quality", "cell_key": key, "q": 0, "channel": "close"} for _ in range(f)]
    )

def show(label, st):
    rec = E.recommend("polecat", "implement", cfg, st, tol_class="Moderate")
    r = rec["reasons"]
    opus_n = next(c for c in r["candidates"] if c["tier_id"] == "opus")["n"]
    print(f"{label}: recommend -> {rec['tier_id']:<6} "
          f"mu*={r['baseline_mean']:.3f}  threshold={r['gate_threshold']:.3f}  opus_obs={opus_n}")

# (1) opus NEVER observed; cheaper tier (sonnet) observed at 0.85 -> admitted vs the prior
st1 = S.CellStore.cold_start(cfg)
feed(st1, "sonnet", 85, 15)
show("opus unmeasured, sonnet 85/100", st1)

# (2) opus MEASURED but mediocre (0.80): IDENTICAL threshold -> gate can't tell the two apart
st2 = S.CellStore.cold_start(cfg)
feed(st2, "opus", 80, 20)
show("opus measured mediocre 80/100", st2)

Output:

opus unmeasured, sonnet 85/100: recommend -> sonnet mu*=0.800  threshold=0.750  opus_obs=0
opus measured mediocre 80/100: recommend -> opus   mu*=0.800  threshold=0.750  opus_obs=100

Case (1): a sonnet observed at 0.85 is admitted as "quality-preserving vs opus" with 0 opus observations. If opus performs near the frontier it's assumed to (~0.97), that is a ~12-point regression certified as within the 5-point Moderate tolerance. Case (2): measuring opus at a mediocre 80/100 yields the same 0.75 threshold — confirming the gate conditions on mu*'s point value, not on whether opus was ever observed.

Why pooling doesn't save it

store.pooled() caps injected sibling mass at pool_lambda × own_mass = 0.5 × 10 = 5 pseudocounts (store.py:486). So even a fully-measured opus sibling lifts the threshold only from 0.75 to ~0.81 — still well below the ~0.90 a true-opus 0.95 would require. eval_flag and inspect's drop-CI both fire on this cell but are advisory-only (engine.py:24-26, 344), so neither blocks the recommendation.

Observed in the wild?

Not in the published 6-day trace — there the one measured cheaper tier (Builder/Bu1 sonnet) ran at 293/293, so no regression occurred. This is a latent hazard: it goes live the first time a genuinely-mediocre downgrade candidate lands on a cell whose baseline was never dispatched (the normal state, since the static default routes many cells away from opus).

Possible fixes (any one closes it)

1. Gate on baseline n — refuse to certify a downgrade when the baseline cell's n is below a floor (treat an unmeasured baseline like a thin cheaper tier: keep baseline). Smallest change, a couple of lines in recommend().
2. Pad mu_star as the paper describes — subtract a posterior SE, or use the baseline's lower credible bound, so an uncertain/unmeasured baseline raises the bar rather than lowering it.
3. Make Layer-4 enforce — have eval_flag (or inspect's drop-CI exceeding tolerance) block the downgrade until resolved, instead of only annotating it.

Happy to send a PR for (1) or (2) if that's useful.

---

Analysis done with Claude (Claude Code, Opus 4.8); the repro is deterministic and self-contained. Glad to be corrected if I've misread the decision path.

repro.py

verify_pooling_cap.py