Add live e2e tests for dual-era stateless proxy behavior

[jhrozek]

Summary

Dual-era operation — serving both Legacy (2025-11-25, session-based) and Modern (2026-07-28, stateless) MCP through the transport proxies — shipped in #5829/#5830/#5831. But that path is a concurrency-sensitive confidentiality boundary: a routing regression can deliver one client's response to another. Single-handler unit tests can't observe that — it only surfaces when real traffic races through a running proxy. This adds the live end-to-end coverage the design calls for (docs/arch/stateless-transport-proxies-design.md §Test plan), across four tiers:

• Harness — a hand-rolled, stdlib-only raw MCP HTTP client (byte-level control to emit the spoofed/malformed/colliding traffic a conforming SDK never would), plus a golden go-sdk v1.7 request fixture as an independent encoding oracle. Backend is yardstick v1.2.0, whose BACKEND_MODE=echo|barrier|hang|crash + --stateless upstreamed exactly what these tests need — so no bespoke backend image.
• Tier 2 — adversarial/concurrency: crown-jewel cross-delivery stress (barrier-forced collisions, conservation assertions + a positive control that the barrier actually engaged), hostile-input suite (-32020/-32021/-32022/-32602, foreign-session 404, oversized/truncated), dual-era concurrent mixing, and a mutation-verified guard that the streamable Modern path never reuses a client Mcp-Session-Id as its routing token.
• Tier 1 — functional: a real independent SDK client (yardstick-client) driving Modern and Legacy round-trips through both proxies (fidelity leg), the server/discover authz behavior, and OTLP span-attribute assertions (mcp.protocol.version, mcp.client.name).
• Tier 3 — resilience: no-state-accumulation under sustained Modern load (integration test, mutation-verified) + fault injection (backend hang/crash; Legacy recovers, Modern degrades cleanly).
• Kubernetes multi-replica: operator-deployed dual-era over Redis with SessionAffinity: None, pod-eviction self-heal, and Redis-unavailable → 503 (authored to run in test-e2e-lifecycle.yml).

Closes #5837.

Type of change

• Other (describe): Test-only — live e2e/integration coverage. No production behavior change; the only non-test edits are a 9-line doc comment + a guard test in pkg/transport/proxy/streamable, and CI/go.mod/images.go wiring for the yardstick 1.2.0 test dependency.

Test plan

• E2E tests (task test-e2e) — ran LABEL_FILTER=dual-era task test-e2e locally on the current main: 11/11 pass.
• Manual testing (described below)

The streamable-package tests (session-id-reuse guard, no-accumulation) pass under go test -race; golangci-lint is clean on every touched file. The Kubernetes tier is authored for CI (verified by compile + go vet + kubernetes-expert review; it runs in test-e2e-lifecycle.yml, not locally — a local kind run would have required deleting a shared cluster with unrelated namespaces). Note for local runs: thv run needs TOOLHIVE_SKIP_DESKTOP_CHECK=1 if ToolHive Desktop is installed.

API Compatibility

• This PR does not break the v1beta1 API. (Test-only; it adds a spec under test/e2e/thv-operator and bumps a workflow env — no CRD/API surface change.)

Changes

File(s)	Change
test/e2e/mcp_raw_client.go, mcp_raw_client_test.go	Raw, stdlib-only adversarial MCP HTTP client
test/e2e/testdata/golden_modern_request.json, mcp_raw_client_golden_test.go	Independent go-sdk v1.7 wire oracle for the client
test/e2e/dual_era_proxy_test.go	Tier-2 crown-jewel cross-delivery stress
test/e2e/hostile_input_proxy_test.go	Tier-2 hostile-input suite
test/e2e/dual_era_mixing_test.go	Tier-2 dual-era concurrent mixing
pkg/transport/proxy/streamable/streamable_proxy_modern_test.go (+9-line comment in streamable_proxy.go)	Modern session-id-reuse guard
test/e2e/dual_era_functional_test.go	Tier-1 functional round-trips (SDK client) + server/discover authz
test/e2e/dual_era_observability_test.go	Tier-1 OTLP span-attribute assertions
pkg/transport/proxy/streamable/streamable_proxy_integration_test.go	Tier-3 no-state-accumulation
test/e2e/dual_era_resilience_test.go	Tier-3 fault injection
test/e2e/thv-operator/acceptance_tests/dual_era_k8s_test.go	k8s multi-replica dual-era
.github/workflows/e2e-tests.yml, test-e2e-lifecycle.yml, go.mod, test/e2e/images/images.go	Wire the yardstick 1.2.0 test dependency (pre-pull, kind-load, image tag)

Does this introduce a user-facing change?

No.

Implementation plan

Approved implementation plan

Four stacked tiers (test-only; the behavior under test already merged in #5829/#5830/#5831). Modern traffic driven by an in-repo raw client for adversarial/concurrency tiers and by a real out-of-process go-sdk v1.7 peer (yardstick-client) for the fidelity tier. Hard constraint: never import go-sdk v1.7 into the root module (peers run out-of-process; yardstick is a separate module/image).

Ground truths the tests respect: streamable proxy = --transport stdio; transparent proxy = remote HTTP, Modern driven by the MCP-Protocol-Version header + _meta; Modern signal = header 2026-07-28 OR a reserved io.modelcontextprotocol/* _meta key (clientInfo optional; clientCapabilities must be an object); -32020 covers only header/body version mismatch; GET/DELETE 405 is stateless-mode-driven.

• Tier 2 (Commit group 1): raw client → golden oracle → crown-jewel cross-delivery (barrier backend forces the collision; conservation + false-green guard) → streamable session-reuse guard → hostile inputs → dual-era mixing.
• Tier 1 (Commit group 2): yardstick-client Modern+Legacy round-trips + server/discover authz + OTLP observability (in-process OTLP receiver).
• Tier 3 (Commit group 3): no-accumulation (white-box integration) + fault injection.
• k8s (Commit group 4): multi-replica MCPServer + Redis + SessionAffinity:None, eviction, Redis-503.

Every tier passed an adversarial multi-reviewer gate before landing.

Special notes for reviewers

• Follow-up issues filed from findings during this work (all pre-existing, not fixed here): Transparent proxy: session-not-found 404 does not echo the request JSON-RPC id #5945 (session-not-found 404 doesn't echo the request id), Authz-denied responses emit non-conformant JSON-RPC (capitalized keys, no jsonrpc field) #5950 (authz-denied responses emit non-conformant JSON-RPC — capitalized keys, no jsonrpc field), Streamable proxy drops/fails requests under high concurrent load (fixed 100-buffer channels, no backpressure) #5952 (streamable proxy drops/fails requests under >~100 concurrent in-flight — fixed-100-buffer channels, no backpressure).
• Deliberate behaviors encoded (verified correct-by-design, not bugs): the transparent proxy 404s a foreign Mcp-Session-Id on a Modern request because its session guard keys on header presence, not the client-forgeable revision (a forged-Modern signal must not bypass session validation); and server/discover is now always-allowed under authz as of Serve MCP 2026-07-28 Modern stateless requests through vMCP #5953 (the authz spec asserts that — it also means Modern-through-authz no longer downgrades).
• Security nuance to weigh (from Serve MCP 2026-07-28 Modern stateless requests through vMCP #5953): that PR lets server/discover bypass the response-filter (a restricted client can enumerate descriptors), judged "equally safe here." Flagged in case tool-visibility is a boundary you care about on this path.
• k8s "any-pod routing": the distinct-backend-pod-count assertion was intentionally not included — kube-proxy is L4/per-connection and both proxy hops pool connections, so pod distribution isn't reliably observable through the test. The tier asserts SessionAffinity: None is set + Modern works across replicas instead (documented in-file).

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 72.06%. Comparing base (cfa2b40) to head (a415ac9).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5981      +/-   ##
==========================================
- Coverage   72.10%   72.06%   -0.05%     
==========================================
  Files         718      718              
  Lines       74465    74465              
==========================================
- Hits        53695    53663      -32     
- Misses      16921    16968      +47     
+ Partials     3849     3834      -15     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[JAORMX]

The test design here is genuinely strong — the barrier-backed positive control in the crown jewel, the mutation-verified session-id-reuse guard, the precise hostile-input error codes, and the independent go-sdk golden oracle are exactly the anti-false-green rigor the PR promises. One blocker before merge, though, introduced by the near-final commit 01dea165 ("Send required Accept header on raw MCP requests").

Blocker — the default Accept header pushes streamable-proxy responses onto the SSE path the raw client can't parse.

SendRaw now unconditionally sets Accept: application/json, text/event-stream (mcp_raw_client.go:335). The streamable proxy branches to its SSE response path for any Accept containing text/event-stream (streamable_proxy.go:610, handlePost → handleSingleRequestSSE), emitting data: {…}\n\n. But populateEnvelope does a plain json.Unmarshal of the whole body with no SSE-frame unwrapping (mcp_raw_client.go:382), so on the streamable-proxy specs resp.Result stays nil and extractNonce returns "".

Impact on the --transport stdio (streamable-proxy) tier:

• dual_era_proxy_test.go (crown jewel): assertCorrelated's gotNonce == nonce fails for every request — the confidentiality boundary this PR exists to guard is never actually exercised.
• dual_era_mixing_test.go: same nonce failure. Its own file comment (lines ~51-58) still documents the opposite premise — "Deliberately NOT setting Accept … Without the header the proxy returns a plain JSON response, which is what every other spec in this suite relies on" — which 01dea165 silently invalidated.
• dual_era_resilience_test.go: the hang context expects 504 and the crash context >= 500, but the SSE path returns an in-band 200 with an error frame.

Everything else is unaffected and sound: the transparent-proxy specs, the hostile-input suite (rejected 400/404 before any streaming), the golden oracle, the routing-token guard, and the unit/integration tests.

Suggested fix (either — you documented both in the mixing-test comment):

1. Teach populateEnvelope to unwrap a single SSE data: frame when Content-Type: text/event-stream, or
2. Have the streamable-proxy specs send Accept: application/json only.

Note the k8s tier legitimately needs the full Accept (the real go-sdk backend 400s without it), so it can't just be dropped globally — option 1 is probably the cleaner single fix. Please also update the now-stale dual_era_mixing_test.go comment.

(FWIW the earlier "11/11 pass" run predates 01dea165 — the commit that added the default header is second from the tip.)

Happy to re-review as soon as the streamable-proxy specs are parsing real responses again.

[jhrozek]

Good catch, and thanks for the precise pointer to streamable_proxy.go:610 — that's exactly it. The default Accept in 01dea165 walked right over the deliberate no-Accept decision the mixing test's own comment documents.

I went with a variant of your option 2 rather than option 1 (SSE unwrapping), for the reason you flagged yourself: the resilience specs assert HTTP 504 (hang) and >= 500 (crash), and the SSE path answers 200 with an in-band error frame — so teaching populateEnvelope to unwrap SSE would fix the nonce parsing but still leave resilience red, and I'd rather not weaken those to in-band assertions. Keeping the proxy tiers on the plain-JSON path preserves what all four specs were designed around.

Concretely (f9d8baa8):

• Reverted the global Accept default in SendRaw, so proxy requests get a plain JSON body again. This also makes the dual_era_mixing_test.go comment accurate again, so no rewrite needed there.
• Added RawRequest.WithStreamableAccept() and call it only on the two k8s Modern requests — that tier's real go-sdk backend 400s without the header and only checks status, so it's the one place that genuinely needs it.

Verified locally this time (which I should have done on the header commit): LABEL_FILTER=dual-era task test-e2e → 11/11 pass, including cross-delivery, mixing, and both resilience specs. Also rebased onto main to clear the go.mod conflict. CI re-running now — happy to get your re-review once it's green.

[JAORMX]

The blocker is resolved by d800074 ("Revert global Accept default; opt in only for real backends") — thanks for the quick turnaround. Verified:

• SendRaw no longer sets a default Accept, so the streamable proxy returns a plain-JSON body again and populateEnvelope parses it — the cross-delivery crown jewel, mixing, and resilience specs get real Result/nonce values back (and the resilience 504/>=500 assertions see real HTTP status, not an in-band SSE 200).
• The real-backend need is handled with an explicit opt-in, RawRequest.WithStreamableAccept(), called only on the two k8s-tier Modern requests (dual_era_k8s_test.go:240,271) — no proxy spec sets Accept.
• The dual_era_mixing_test.go comment is accurate again (it documents exactly why Accept must not be set here), and the unit test now asserts no-Accept-by-default + opt-in.

Everything I flagged as sound before (barrier positive-control, mutation-verified session-id guard, precise hostile-input codes, independent golden oracle) is unchanged. Approving.

One heads-up unrelated to this PR: main is currently red from a nil-pointer panic in pkg/vmcp/client (a struct-literal test from #5990 hits the new resolveAuthStrategy path added by #5980), so your Go-test check may show that failure until main is fixed — it's not caused by anything in this PR.

[JAORMX]

Re-approving after the rebase onto the now-green main (head 637c0c9d, base b928cc31). This is a clean replay — identical changeset to what I approved earlier (17 commits, +3053/−6, 18 files, no content change), just picking up the #5999 CI fix so this can get a green run. My earlier review stands: the Accept/SSE blocker fix (d800074) is intact and the test design (barrier positive-control, mutation-verified session-id guard, precise hostile-input codes, independent golden oracle) is unchanged. LGTM.

[JAORMX]

CI note after re-running the failed jobs twice on the current head (637c0c9d): the two red checks are pre-existing vMCP e2e suites flaking on server/backend bring-up — not this PR's dual-era specs.

• E2E Tests / E2E Tests Core (vmcp) — fails in vmcp_cli_test.go / vmcp_cli_features_test.go / vmcp_infra_features_test.go / vmcp_optimizer_test.go (starts vMCP and exposes backend tools, binds only to 127.0.0.1, serve --config, prefix-conflict, ExcludeAllTools, JWT/OIDC, Redis session, Tier-1 optimizer). The failing-spec count swung 1 → 12 → 12 across three runs of the identical commit (first run: only the composite-tool 60s timeout; both re-runs: 12/16), i.e. nondeterministic cascading vMCP-server startup failures, not a deterministic defect.
• E2E Test Lifecycle (all 3 kind nodes) — the operator test/e2e/thv-operator/virtualmcp suite: Global ExcludeAllTools … empty tools list from all backends (helpers.go:87, BeforeAll) failed and interrupted the Redis-session / horizontal-scaling / circuit-breaker specs.

Both are the same signature (vMCP server/backend bring-up + tool aggregation) and both are pre-existing suites unrelated to this test-only PR — its dual-era proxy specs and the rest of the e2e matrix (proxy, conformance, operator, core, unit, lint) are green, and the k8s dual-era spec added here is not among the failures. So this blocks a clean green here but isn't caused by the change, and it almost certainly affects other PRs too. Worth a deflake pass on the vMCP e2e bring-up (core vmcp + operator virtualmcp). cc @jhrozek