Fail fast on dataset bundle drift instead of silently swapping the eval mixture

[Shumatsurontek]

Bug

• npy_sources._load_bundle_defaults() caught every exception on the manifest fetch, warned, and substituted a hardcoded 11-source list
• delta_threshold is a fixed 0.0015 nats — change the mixture and you move the bar every challenger is judged against, while the verdict still reads accepted: true
• One timeout scored challengers against a different distribution and wrote it into the king chain permanently
• The fallback was never equivalent: live bundle serves eu-central-1/us-east-1, the hardcoded list pointed at s3
• Manifest was also unauthenticated and unpinned — whoever serves that URL controls every validator's eval distribution

Fix

• No fallback mixture: 3 bounded retries with backoff, then raise
• Pin by sha256 over raw manifest bytes in chain.toml [dataset].bundle_digest
• Malformed pin raises at import, so a typo can't read as "unpinned" and disable the check
• Resolution moved out of import side effect into explicit ensure_bundle_resolved() at lifespan — no network I/O on import
• {url, digest, pinned, origin} stamped into verdict.dataset.bundle and /health, so a decision can be replayed against the exact source list
• scripts/pin_dataset_bundle.py prints digest + weighted sources; --check exits non-zero on drift

Trade accepted

• Unreachable manifest now aborts eval-server startup → validator burns weights. Honest and reversible; a verdict against an unknown mixture is neither. PM2 restarts, so transient outages self-heal.

Pin ships empty

• Fix works unpinned: no fallback, digest recorded, loud UNPINNED warning at startup
• Enabling it makes drift fatal, so it's a deployment call. Live digest ready to paste:

sha256:d2a08b90f5971b067b858df1a6bf55721008622b186b97ac09e9a89f027b5c95

Tests

First in the repo — 55, under 1s, no GPU or network.

• test_dataset_bundle.py (20) — fetch exhaustion raises, digest mismatch refuses, empty mixture refuses, retry-then-succeed, env override skips network, in-place mutation
• test_verdict.py (9) — identical rejected, clearly-better accepted, better-but-under-δ rejected, accepted == (lcb > delta), seed determinism
• test_reveal.py (18) — v3/v4 round-trips, malformed payloads, digest shape rejection
• test_custom_code_policy.py (8) — _validate_quasar_auto_map rejects hub-qualified and traversal module paths

The in-place-mutation test guards a live trap: eval_server_two_sources imports DEFAULT_MANIFEST_URLS by value, so resolution mutates rather than rebinds — rebinding would leave /health reporting a stale empty list.

Flagged, not fixed

• fla is an undeclared dep — archs/quasar/modeling_quasar.py:23 imports flash-linear-attention, absent from pyproject.toml, so uv sync yields an install where import validator fails. Stubbed in tests/conftest.py only when missing, so GPU hosts hit the real arch module
• Ruff 280 → 282; both new are RUF100 on # noqa: E402, identical to validator.py:31-32 and miner.py:30-31
• uv.lock not included — separate call