fix(eval): one Stage-1 matching predicate for the live and offline paths - #320
Open
vaibhavdabas16 wants to merge 1 commit into
Open
fix(eval): one Stage-1 matching predicate for the live and offline paths#320vaibhavdabas16 wants to merge 1 commit into
vaibhavdabas16 wants to merge 1 commit into
Conversation
The Stage-1 interceptor decision is the benchmark's deterministic ground
truth: it sets run-meta.intercepted and every published Intercepted number.
It existed as two hand-maintained copies — runtime-server/server.py running
in-container, and a mirror in eval/edgebench_judge.py that re-verifies a
submitted evidence archive — and only the mirror had tests. server.py is
excluded from pyright, so the copy deciding the published numbers was the
unverified one.
They had drifted, in two ways that change verdicts:
- Query params. The live interceptor keeps a repeated key's values as a list
(`v[0] if len(v) == 1 else v`), so `?tag=a&tag=b` does not match a constant
of `{"tag": "a"}` and the request is let through. The offline mirror took
`v[0]` unconditionally, matched, and reported the run as intercepted — a
Stage-1 pass for a request that was never blocked.
- Malformed url_pattern. server.py called re.search unguarded inside the CDP
event loop, so a bad pattern raised there and stopped interception for the
remainder of the run, silently scoring every later task Stage-1 zero. The
mirror caught re.error and returned False.
runtime-server/matching.py is now the single copy. server.py imports it as a
sibling — uvicorn runs from that directory — and eval/edgebench_judge.py
loads it by file, since `runtime-server` is not a valid module path. It is
kept to the standard library because the offline verifier imports it on the
host, where the runtime-server's own dependencies are absent.
It lives in runtime-server/ rather than the runtime/shared/ that TIGER-AI-Lab#301
suggests: harnesses/base/Dockerfile.base copies runtime-server/server.py but
never copies shared/, so a module there would be missing from every
non-harbor image and the runtime-server would fail to boot. Both Dockerfiles
now copy matching.py alongside server.py, and a test asserts they stay in
step.
Every failing check in server.py's gate took the same action — continue the
request — so the four inline branches collapse into one shared call rather
than four predicates that have to be kept in the same order as the verifier's.
Two behaviour changes follow, both adopting the live interceptor as the
truth:
1. The offline verifier is now stricter on repeated query params and agrees
with what the interceptor actually did.
2. A malformed url_pattern is a no-match instead of an exception, so one bad
task no longer disarms interception for the rest of the run.
tests/test_stage1_matching.py adds a 14-case fixture matrix over
url_pattern/method/body/params, pins both divergences, guards against either
side re-implementing the predicate, and asserts both Dockerfiles ship it.
Fixes TIGER-AI-Lab#301.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Stage-1 — does this request hit the task's target (
url_pattern+method+ constbody/params)? — is the benchmark's deterministic ground truth. It setsrun-meta.interceptedand every published Intercepted number. It existed as two hand-maintained copies:runtime-server/server.py:173-185,449-477running in-container, and a mirror ateval/edgebench_judge.py:55-94that re-verifies a submitted evidence archive. Only the mirror had tests, andserver.pyis excluded from pyright (pyproject.toml:74) — so the copy deciding the published numbers was the unverified one.They had drifted, in two ways that change verdicts.
1. Repeated query params. The live interceptor keeps a repeated key's values as a list (
v[0] if len(v) == 1 else v), so?tag=a&tag=bdoes not match a constant of{"tag": "a"}and the request is let through. The offline mirror tookv[0]unconditionally, matched, and reported the run as intercepted. Run againstmain's actual code:That is a Stage-1 pass recorded for a request the interceptor never blocked.
2. Malformed
url_pattern.server.py:455calledre.searchunguarded inside the CDP event loop, so a bad pattern raised there and stopped interception for the remainder of the run — every later task in that run silently scoring Stage-1 zero (the edge case #258 flags). The mirror caughtre.errorand returnedFalse.runtime-server/matching.pyis now the single copy.server.pyimports it as a sibling (uvicorn runs from that directory perharnesses/base/entrypoint.sh:37), andedgebench_judge.pyloads it by file sinceruntime-serveris not a valid module path. It is kept to the standard library because the offline verifier imports it on the host, where the runtime-server's dependencies are absent.Every failing check in
server.py's gate took the same action — continue the request — so the four inline branches collapse into one shared call, rather than four predicates a reader has to keep in the same order as the verifier's.One deviation from the issue, deliberately
The issue proposes
runtime/shared/matching.py. That would not boot:harnesses/base/Dockerfile.basecopiesruntime-server/server.pybut never copiesshared/— onlyharbor/Dockerfiledoes. A module undershared/would be missing from every non-harbor image andserver.pywould fail its import at startup. It lives inruntime-server/instead, which both Dockerfiles already copy, and a test asserts the two stay in step.Two intended behaviour changes
Both adopt the live interceptor as the truth, since it is what actually happened during the run:
url_patternis a no-match instead of an exception, so one bad task no longer disarms interception for the rest of the run.Happy to split (2) out if you would rather keep this PR purely mechanical.
Corpus
Stage-1 interception is the scoring path for both corpora; ticking the default rather than claiming a V1 verification I have not run.
Test plan
tests/test_stage1_matching.py(26 tests): a 14-case fixture matrix overurl_pattern/method/body/params— including batched-GraphQL list bodies, scalar bodies, empty constraints, and a forgedparamsfield that must not fake a match — plus both divergences and there.errorcases.origin/main's own_stage1_match, not just described: it returnsTruewhere the new one returnsFalse, matching the transcript above._const_fields_matchor callre.search, andmatching.pymust stay free of runtime-server dependencies.matching.pywherever they copyserver.py.server.pyimports it at startup, so an image with one and not the other fails to boot the runtime-server.tests/test_edgebench_judge.pyis unchanged and green — the shared predicate is behaviour-compatible on everything that was already covered.test_host_tasks.py::...[v1-lite], is a local Windows artifact (git symlinks, mode120000, checked out as text) and is green on CI.ruff check .: 157 findings on this branch and 157 onmain— zero introduced.ruff format --checkclean.pyright src/clawbench tests: 6 errors on this branch vs 7 onmain, none in the changed files.mainand against my other open PRs (fix(batch): one truncated run-meta.json no longer discards the batch summary #312, fix(rescore): resolve paths from the workspace instead of a maintainer's home #313, fix(judge): a bad --judge model no longer discards a completed run #314, fix(runner): bound container wait so one wedged run can't stall a batch #316, fix(judge): collapse judge_llm.py onto judge.py's transport #319) viagit merge-tree.What I could not verify: there is no Docker on this machine, so the
COPYlines and the sibling import are argued from the Dockerfiles and theuvicorn server:appworking directory rather than observed in a built image. A maintainer running one task end-to-end would close that gap; the Dockerfile test guards the failure mode I could guard offline.Related issues
Fixes #301. Touches the malformed-regex edge case noted in #258.