Skip to content

fix(eval): one Stage-1 matching predicate for the live and offline paths - #320

Open
vaibhavdabas16 wants to merge 1 commit into
TIGER-AI-Lab:mainfrom
vaibhavdabas16:fix/shared-stage1-matching
Open

fix(eval): one Stage-1 matching predicate for the live and offline paths#320
vaibhavdabas16 wants to merge 1 commit into
TIGER-AI-Lab:mainfrom
vaibhavdabas16:fix/shared-stage1-matching

Conversation

@vaibhavdabas16

Copy link
Copy Markdown

What does this PR do?

Stage-1 — does this request hit the task's target (url_pattern + method + const body/params)? — is the benchmark's deterministic ground truth. It sets run-meta.intercepted and every published Intercepted number. It existed as two hand-maintained copies: runtime-server/server.py:173-185,449-477 running in-container, and a mirror at eval/edgebench_judge.py:55-94 that re-verifies a submitted evidence archive. Only the mirror had tests, and server.py is excluded from pyright (pyproject.toml:74) — so the copy deciding the published numbers was the unverified one.

They had drifted, in two ways that change verdicts.

1. Repeated query params. The live interceptor keeps a repeated key's values as a list (v[0] if len(v) == 1 else v), so ?tag=a&tag=b does not match a constant of {"tag": "a"} and the request is let through. The offline mirror took v[0] unconditionally, matched, and reported the run as intercepted. Run against main's actual code:

OLD offline verifier (origin/main) says intercepted: True
NEW offline verifier            says intercepted: False
live interceptor params: {'tag': ['a', 'b']}

That is a Stage-1 pass recorded for a request the interceptor never blocked.

2. Malformed url_pattern. server.py:455 called re.search unguarded inside the CDP event loop, so a bad pattern raised there and stopped interception for the remainder of the run — every later task in that run silently scoring Stage-1 zero (the edge case #258 flags). The mirror caught re.error and returned False.

runtime-server/matching.py is now the single copy. server.py imports it as a sibling (uvicorn runs from that directory per harnesses/base/entrypoint.sh:37), and edgebench_judge.py loads it by file since runtime-server is not a valid module path. It is kept to the standard library because the offline verifier imports it on the host, where the runtime-server's dependencies are absent.

Every failing check in server.py's gate took the same action — continue the request — so the four inline branches collapse into one shared call, rather than four predicates a reader has to keep in the same order as the verifier's.

One deviation from the issue, deliberately

The issue proposes runtime/shared/matching.py. That would not boot: harnesses/base/Dockerfile.base copies runtime-server/server.py but never copies shared/ — only harbor/Dockerfile does. A module under shared/ would be missing from every non-harbor image and server.py would fail its import at startup. It lives in runtime-server/ instead, which both Dockerfiles already copy, and a test asserts the two stay in step.

Two intended behaviour changes

Both adopt the live interceptor as the truth, since it is what actually happened during the run:

  1. The offline verifier is stricter on repeated query params and now agrees with the interceptor.
  2. A malformed url_pattern is a no-match instead of an exception, so one bad task no longer disarms interception for the rest of the run.

Happy to split (2) out if you would rather keep this PR purely mechanical.

Corpus

  • v2
  • v1
  • both
  • not applicable

Stage-1 interception is the scoring path for both corpora; ticking the default rather than claiming a V1 verification I have not run.

Test plan

What I could not verify: there is no Docker on this machine, so the COPY lines and the sibling import are argued from the Dockerfiles and the uvicorn server:app working directory rather than observed in a built image. A maintainer running one task end-to-end would close that gap; the Dockerfile test guards the failure mode I could guard offline.

Related issues

Fixes #301. Touches the malformed-regex edge case noted in #258.

The Stage-1 interceptor decision is the benchmark's deterministic ground
truth: it sets run-meta.intercepted and every published Intercepted number.
It existed as two hand-maintained copies — runtime-server/server.py running
in-container, and a mirror in eval/edgebench_judge.py that re-verifies a
submitted evidence archive — and only the mirror had tests. server.py is
excluded from pyright, so the copy deciding the published numbers was the
unverified one.

They had drifted, in two ways that change verdicts:

- Query params. The live interceptor keeps a repeated key's values as a list
  (`v[0] if len(v) == 1 else v`), so `?tag=a&tag=b` does not match a constant
  of `{"tag": "a"}` and the request is let through. The offline mirror took
  `v[0]` unconditionally, matched, and reported the run as intercepted — a
  Stage-1 pass for a request that was never blocked.
- Malformed url_pattern. server.py called re.search unguarded inside the CDP
  event loop, so a bad pattern raised there and stopped interception for the
  remainder of the run, silently scoring every later task Stage-1 zero. The
  mirror caught re.error and returned False.

runtime-server/matching.py is now the single copy. server.py imports it as a
sibling — uvicorn runs from that directory — and eval/edgebench_judge.py
loads it by file, since `runtime-server` is not a valid module path. It is
kept to the standard library because the offline verifier imports it on the
host, where the runtime-server's own dependencies are absent.

It lives in runtime-server/ rather than the runtime/shared/ that TIGER-AI-Lab#301
suggests: harnesses/base/Dockerfile.base copies runtime-server/server.py but
never copies shared/, so a module there would be missing from every
non-harbor image and the runtime-server would fail to boot. Both Dockerfiles
now copy matching.py alongside server.py, and a test asserts they stay in
step.

Every failing check in server.py's gate took the same action — continue the
request — so the four inline branches collapse into one shared call rather
than four predicates that have to be kept in the same order as the verifier's.

Two behaviour changes follow, both adopting the live interceptor as the
truth:

1. The offline verifier is now stricter on repeated query params and agrees
   with what the interceptor actually did.
2. A malformed url_pattern is a no-match instead of an exception, so one bad
   task no longer disarms interception for the rest of the run.

tests/test_stage1_matching.py adds a 14-case fixture matrix over
url_pattern/method/body/params, pins both divergences, guards against either
side re-implementing the predicate, and asserts both Dockerfiles ship it.

Fixes TIGER-AI-Lab#301.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stage-1 interceptor matching is duplicated between runtime-server and edgebench_judge, and the live copy has zero tests

1 participant