fix(evals): bind HTTP resumes to experiment identity (#4707)

* fix(evals): bind HTTP resumes to experiment identity

* fix(evals): publish only validated complete HTTP samples

Serialize each result and raw sample before writing them. Stage samples
in the destination directory and publish them without overwriting existing
captures, so malformed responses or failed writes cannot leave a poisoned
sample that prevents a later valid resume. Cover invalid probabilities,
non-finite raw extras, successful retry, and existing sample preservation.

* fix(evals): refuse dangling samples and honor file umask

Reject unusable existing sample destinations before inference and after
an atomic-link race. Create staged files with ordinary exclusive-open
permissions instead of mkstemp's forced 0600; preserve the current umask
without modifying process-global settings. Test dangling links, clean
staging after refusal, and normal shared-readable sample modes.

* fix(evals): score sparse subsets without fabricated coverage

* fix(evals): validate resumed metrics before publishing results

Use one row admission path for fresh responses and resumed rows. Reject
nonfinite JSON, invalid timings and malformed token counts before inference,
partial-log appends, samples or final-output creation. Preserve absent server
timing and valid zero counts.

The new regressions fail on the previous implementation. All 23 HTTP runner
tests and Python compilation pass locally on hash-verified source; hosted
Bazel and the full platform matrix remain required.

* fix(evals): validate calibrated replay evidence

Require calibrated partial rows to retain the exact raw probabilities
that produce their reported calibrated values. Refuse fabricated raw
probabilities on uncalibrated runs, before HTTP calls or file writes.

Add behavioral resume regressions for missing, changed and unexpected
raw values, tampered calibrated values and successful exact replay.

* fix(evals): preserve unknown usage in resumed reports

Keep input, state and dropped-token counts distinct. Missing server
telemetry stays null; reported zeroes remain measured zeroes. Score
observed values with explicit coverage and refuse full-cohort truncation
claims when some cases are unmeasured.

Advance HTTP experiment identity to version 2 and refuse ambiguous older
partial and completed results instead of relabeling their metrics.

All 43 Python runner and scoring tests pass, including interrupted
resume and CLI reporting with controlled HTTP responses. Baseline source
reproduced the telemetry failures. Python parsing and whitespace checks
pass; Bazel execution and live model qualification were not performed.

* fix(evals): put intent engine modules on the test import path

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Author: ScriptedAlchemy

evals/intent_engine/BUILD.bazel

@@ -0,0 +1,27 @@
+load("@rules_python//python:defs.bzl", "py_test")
+
+py_test(
+    name = "http_runner_test",
+    size = "small",
+    srcs = [
+        "common.py",
+        "run_http.py",
+        "score.py",
+        "test_run_http.py",
+    ],
+    imports = ["."],
+    main = "test_run_http.py",
+)
+
+py_test(
+    name = "scoring_test",
+    size = "small",
+    srcs = [
+        "common.py",
+        "compare.py",
+        "score.py",
+        "test_score.py",
+    ],
+    imports = ["."],
+    main = "test_score.py",
+)

evals/intent_engine/HTTP_RUNNER.md

@@ -0,0 +1,74 @@
+# Resumable HTTP evaluations
+
+`run_http.py` requires a serving declaration in addition to the endpoint and tag:
+
+```sh
+python evals/intent_engine/run_http.py \
+  --url http://127.0.0.1:8009/v1/systemone \
+  --tag pinned-run \
+  --server-config /path/to/serving.json \
+  --send-model kev-latest \
+  B_noul_text
+```
+
+The JSON declaration must contain nonempty `checkpoint`, `server_revision`,
+`backend`, and `dtype` strings, and the actual finite, nonnegative server
+`temperature`. Include other settings that affect inference as additional keys.
+Record immutable checkpoint/server revisions, not only a moving model alias.
+Do not put credentials in this file. The declaration is operator-supplied
+provenance, **not remote attestation**: update it whenever the deployment changes.
+The optional `--noul-temperature` is a separate client-side calibration step;
+it must be finite and strictly positive.
+
+The existing `.partial` log starts with the experiment identity. This binds the
+whole dataset hash, selected case IDs and order, the exact rendered HTTP request
+bytes for each case, endpoint hash, requested model, client calibration, and the
+serving declaration. A completed result retains the same identity and each row's
+request fingerprint. The endpoint itself is not copied into the identity.
+
+An interrupted run resumes only with matching identity. Changed questions,
+question order, renderer output, labels, subset contents, or serving settings
+refuse before any HTTP request or append. Headerless old logs, incomplete rows,
+duplicate/foreign cases, and malformed probability maps also refuse; no migration
+or silent salvage is performed. Choose a new tag for another experiment or an
+old unidentifiable log. Completed output files are never overwritten.
+
+Use only one writer per tag/configuration. Rows are flushed after each response;
+this is not a crash-durable database or a concurrent-writer protocol. If final
+output publication itself is interrupted, the partial log remains, but an
+incomplete final file requires operator inspection before retrying that tag.
+
+## Token and truncation evidence
+
+Each result records `usage.input_tokens` as `input_tokens`,
+`usage.state_tokens` as `state_tokens`, and `usage.state_tokens_dropped` as
+`dropped`. These are distinct server measurements; total input tokens are never
+substituted for state tokens. Omitted or null telemetry stays JSON `null`.
+Reported zeroes stay zero, and reported counts must be nonnegative integers.
+
+`score.py` computes token statistics only from measured cases and reports
+`input_tokens_measured_cases`, `state_tokens_measured_cases`, and
+`truncation_measured_cases` alongside `total_cases`. The full-cohort
+`cases_truncated` is unknown unless every case reports a dropped-token count;
+`observed_cases_truncated` counts only confirmed truncation in measured cases.
+No measurements means unknown token statistics, not a zero-token result.
+
+The experiment identity is version 2. Version-1 partial logs are refused before
+inference or append: their default-zero truncation and mislabeled input counts
+cannot be recovered as measurements. Use a new tag; do not relabel old logs.
+Version-1 completed HTTP results are also refused by the scorer. Historical
+reports are not rewritten or retrospectively qualified.
+
+The runner still uses the historical nine-category evaluation in `common.py`.
+Runtime-renderer parity, qualified labels, held-out quality, and independent
+verification of server-reported telemetry remain separate qualification work.
+
+Run the no-inference persistence regressions with:
+
+```sh
+bazel test //evals/intent_engine:http_runner_test //evals/intent_engine:scoring_test --test_output=errors
+```
+
+The tests exercise the actual runner and filesystem with substituted HTTP
+responses. They do not download models, spend inference tokens, or qualify a
+model's quality or performance.

evals/intent_engine/compare.py

@@ -1,15 +1,18 @@
 """Head-to-head tables from summary.json for the report. usage: compare.py CONFIG [CONFIG ...]"""
-import json, os, sys, collections
+import json, os, sys
 HERE = os.path.dirname(os.path.abspath(__file__)); sys.path.insert(0, HERE)
-from common import CATS, load
+from common import CATS
 S = json.load(open(os.path.join(HERE, "summary.json")))
 cfgs = [c for c in sys.argv[1:] if c in S]
-ds = load(os.path.join(HERE, "dataset.json"))
-cnt = collections.Counter(l for c in ds for l in c["labels"])
-print("n=%d none=%d ctx=%d" % (len(ds), sum(not c["labels"] for c in ds), sum(c["context_needed"] for c in ds)), dict(cnt))
+for c in cfgs:
+    sample = S[c]["t0.5"]
+    print("%s: n=%d none=%d ctx=%d" % (c, sample["all"]["n"], sample["all"]["none_n"], sample["ctx"]["n"]))
 f = lambda x: "-" if x is None else "%.2f" % x
 print("\n| metric | " + " | ".join(cfgs) + " |\n|---|" + "---|" * len(cfgs))
 rows = [("macro AUC", lambda s: s["macro_auc"]), ("macro AUC (ctx subset)", lambda s: s["macro_auc_ctx"]),
+        ("AUC eligible categories", lambda s: s["macro_auc_n_categories"]),
+        ("AUC eligible categories (ctx subset)", lambda s: s["macro_auc_ctx_n_categories"]),
+        ("LOO evaluated cases", lambda s: s["loo_evaluated_cases"]),
         ("macro F1 @ best t (in-sample)", lambda s: s["best_t_in_sample"]["macro_f1"]),
         ("macro F1 @ LOO-chosen t", lambda s: s["macro_loo_f1"]),
         ("macro F1 @ 0.6", lambda s: s["t0.6"]["macro_f1"]),
@@ -24,14 +27,14 @@
         ("median ms / call", lambda s: s["latency"]["median_ms"])]
 for name, fn in rows:
     print("| %s | %s |" % (name, " | ".join(f(fn(S[c])) if name != "median ms / call" else str(fn(S[c])) for c in cfgs)))
-print("\nper-category F1 at best t (t) / F1 at 0.6 / AUC")
-print("| category (n) | " + " | ".join(cfgs) + " |\n|---|" + "---|" * len(cfgs))
+print("\nper-category positive count; F1 at best t (t) / F1 at 0.6 / AUC")
+print("| category | " + " | ".join(cfgs) + " |\n|---|" + "---|" * len(cfgs))
 for k in CATS:
     cells = []
     for c in cfgs:
         p = S[c]["per_cat"][k]
-        cells.append("%s (%.2g) / %s / %s" % (f(p["best_f1"]), p["best_t"], f(p["sweep"]["0.6"]["f1"]), f(p["auc"])))
-    print("| %s (%d) | %s |" % (k, cnt[k], " | ".join(cells)))
-print("\nrecommended thresholds (in-sample best-F1 t; LOO F1)")
+        cells.append("n=%d; %s (%.2g) / %s / %s" % (p["n_pos"], f(p["best_f1"]), p["best_t"], f(p["sweep"]["0.6"]["f1"]), f(p["auc"])))
+    print("| %s | %s |" % (k, " | ".join(cells)))
+print("\nin-sample threshold selections (not held-out qualification; LOO F1)")
 for c in cfgs:
-    print(c, {k: (S[c]["per_cat"][k]["best_t"], round(S[c]["per_cat"][k]["loo_f1"] or 0, 2)) for k in CATS})
+    print(c, {k: (S[c]["per_cat"][k]["best_t"], None if S[c]["per_cat"][k]["loo_f1"] is None else round(S[c]["per_cat"][k]["loo_f1"], 2)) for k in CATS})