Repository navigation
fix(evals): bind HTTP resumes to experiment identity - #4707
Conversation
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Serialize each result and raw sample before writing them. Stage samples in the destination directory and publish them without overwriting existing captures, so malformed responses or failed writes cannot leave a poisoned sample that prevents a later valid resume. Cover invalid probabilities, non-finite raw extras, successful retry, and existing sample preservation.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Reject unusable existing sample destinations before inference and after an atomic-link race. Create staged files with ordinary exclusive-open permissions instead of mkstemp's forced 0600; preserve the current umask without modifying process-global settings. Test dangling links, clean staging after refusal, and normal shared-readable sample modes.
|
Added The scorer now reports null metrics for empty groups, averages AUC only across categories with both classes while exposing that denominator, and reports no leave-one-out result for a singleton with no training fold. Incomplete Choice coverage and entirely empty result configurations are explicitly refused. Comparison tables use each scored configuration's actual total, context, no-intent, and per-category positive counts rather than opening Added nine behavioral regressions under Validation: AST parsing of all three changed/new Python sources and whitespace/final-newline checks passed. Their published Git blob hashes exactly match the inspected staged sources; the GitHub comparison confirms only score.py, compare.py, test_score.py, and the additive Bazel target changed. The new behavioral tests were not executed: Bazel is unavailable here, and no direct Python test run or Cargo build was substituted. Hosted CI must validate this exact new head; earlier local test results in the PR body do not validate this addition. Published with an expected-head, non-force update from |
Use one row admission path for fresh responses and resumed rows. Reject nonfinite JSON, invalid timings and malformed token counts before inference, partial-log appends, samples or final-output creation. Preserve absent server timing and valid zero counts. The new regressions fail on the previous implementation. All 23 HTTP runner tests and Python compilation pass locally on hash-verified source; hosted Bazel and the full platform matrix remain required.
Require calibrated partial rows to retain the exact raw probabilities that produce their reported calibrated values. Refuse fabricated raw probabilities on uncalibrated runs, before HTTP calls or file writes. Add behavioral resume regressions for missing, changed and unexpected raw values, tampered calibrated values and successful exact replay.
Keep input, state and dropped-token counts distinct. Missing server telemetry stays null; reported zeroes remain measured zeroes. Score observed values with explicit coverage and refuse full-cohort truncation claims when some cases are unmeasured. Advance HTTP experiment identity to version 2 and refuse ambiguous older partial and completed results instead of relabeling their metrics. All 43 Python runner and scoring tests pass, including interrupted resume and CLI reporting with controlled HTTP responses. Baseline source reproduced the telemetry failures. Python parsing and whitespace checks pass; Bazel execution and live model qualification were not performed.
|
Pushed b36e130 after reviewing the current head and preserving its existing resume/sample/calibration repairs. Removed two misleading telemetry defaults: total The HTTP experiment identity is now version 2. Old partial logs refuse before inference/writes, and the scorer refuses old completed HTTP results rather than relabeling fabricated/defaulted metrics. No existing result files are migrated or rewritten. Validation: all 43 Python runner/scoring tests pass with Refs #4535; runtime renderer parity, held-out qualification, and independent verification of server-reported telemetry remain outside this patch. |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem
The HTTP evaluator resumed by case ID alone, so a reused tag could mix incompatible experiments. It also labeled total input tokens as state tokens and reported missing token/truncation telemetry as measured zeroes.
Changes
The partial log and completed result share a versioned identity covering dataset bytes, ordered cases, exact request bytes, endpoint, requested model, calibration, and the operator's serving declaration. Resume validates the complete log before network calls or writes. Malformed rows, contradictory calibration evidence, duplicate cases, and changed identities refuse without modifying the log. Raw samples are validated before exclusive publication; completed outputs are not overwritten.
Input tokens, state tokens, and dropped-state tokens now retain their distinct reported meanings. Missing telemetry stays null; actual zeroes stay zero. Scoring reports measured-case coverage and leaves the total truncation count unknown when coverage is incomplete. Version-1 HTTP logs/results are refused rather than silently relabeled as version-2 measurements.
Validation
At
b36e130035d2fd8acca3e139666e29508212d6b8, 43 Python runner and scoring tests passed:python -m unittest discover -s evals/intent_engine -p 'test_*.py' -vThese tests execute the actual runner, filesystem persistence/resume, request encoding, scorer, and report CLI with controlled HTTP responses. The added telemetry regressions reproduced the previous failures. Python parse and whitespace checks passed, and all six publication blobs match the tested local files.
Bazel was unavailable locally. Hosted CI must execute
//evals/intent_engine:http_runner_testand//evals/intent_engine:scoring_test. No live model inference, model-quality result, or performance measurement is claimed.Scope and limits
Refs #4535; not a complete resolution. The historical nine-category corpus is unchanged. Runtime-renderer parity, held-out qualification, and independent verification of server-reported telemetry remain open. The serving declaration is operator provenance, not remote attestation. The log supports one writer per tag/configuration; final-publication interruption limits are documented in
HTTP_RUNNER.md. Existing historical result files are not rewritten.