flaky: test_v1_responses_http_bridge_reconnects_after_clean_upstream_close intermittently fails with 409 on CI

[Soju06]

tests/integration/test_http_responses_bridge.py::test_v1_responses_http_bridge_reconnects_after_clean_upstream_close has failed with assert 409 == 200 on the second request in three unrelated CI runs over the past week (observed on PRs #1615, #1653, and one more takeover branch), each time passing on a plain rerun and passing locally (3/3 and 56/56 in the containing module).

The 409 suggests the reconnect race occasionally resolves to a conflict (the bridge still considers the previous owner live when the second request lands) under CI runner timing. Likely needs either a deterministic synchronization point in the test (wait for the close to be fully settled before the second request) or a widened acceptance if 409-then-retry is legitimate behavior for this window.

Filing so the next flake doesn't get papered over with reruns — happy to take it in a future triage round if nobody claims it.

[Soju06]

Narrowed the candidate set: the only 409 producers reachable from POST /v1/responses are the three bridge_instance_mismatch sites in http_bridge/mixin.py (turn-state scope conflict ~1169, generated-turn-state continuity ~1188, durable claim rejection ~1800). The test client sends no turn-state header, so the durable claim at ~1800 is the prime suspect.

Proposed mechanism: the fake upstream queues the clean close(1000) behind response.completed, so the reader teardown (sets closed, pops the session via _detach_http_bridge_session_locked, then a scheduled _close_http_bridge_session releases the durable lease) races the second request's acquisition. That path can detach the retiring session (registry_detach, which sets force_durable_takeover_after_detach) and immediately claim the durable row while release_live_session is mid-flight. claim_session's read-modify-write in durable_bridge_repository.py leans on with_for_update(), which is a no-op on SQLite (CI), so an interleaved release/claim can return a lookup that fails the owner check and surfaces the 409.

Two notes before picking a fix: (1) capturing the 409 body in the test assertion (or bridge event logs in CI) would confirm which site fires; (2) since a single-instance deployment can 409 a real client at this boundary, widening the test to accept 409-then-retry would mask a product bug — the settlement ordering (or a same-instance claim retry after settle) looks like the right fix, with a deterministic test barrier (wait for the retiring session to leave the registry before the second POST) as an interim de-flake. Likely shares a root boundary with #1304.

[Soju06]

Another occurrence of the 409 bridge_instance_mismatch flake, this time on #1618's integration-bridge shard (run 31579913656, 2026-08-12T08:55Z): POST /v1/responses returned the durable-claim conflict on a single-instance test run — consistent with the durable claim vs release race hypothesis above (SQLite with_for_update no-op path). Telemetry-only diff, so unrelated to the change under test.

[Soju06]

Third occurrence: #1659's integration-bridge shard (run 31573964831, 2026-08-12T21:20Z), same 409 bridge_instance_mismatch on POST /v1/responses in a single-instance test run. That's three hits across three unrelated PRs (#1677 telemetry-adjacent rerun, #1618, #1659) in two days — this flake is now the top source of red CI on otherwise-green PRs; raising priority on the durable claim/release race fix.

[Soju06]

Fourth occurrence: #1716's integration-bridge shard (run 31767534257, 2026-08-14T03:44Z), test_v1_responses_http_bridge_classifies_responses_lite_developer_interleaved_full_resend[lite-bundle-not-at-prefix-start], same 409 bridge_instance_mismatch. Four unrelated PRs in three days now.

[Soju06]

Fixed by #1751 (auto-closed). The hypothesis in the thread above was right about the location — the durable claim vs release race on the SQLite with_for_update no-op path — and reproducing it deterministically turned up two distinct defects there:

1. A same-owner reclaim kept the owner epoch, so the retiring session's fenced release still matched after the successor's claim and closed the row beneath it. The epoch was historically kept because reused sessions re-claimed; today a reused session renews instead, so nothing needed epoch stability across claims.
2. The claim's write was not authoritative. It mutated ORM attributes, and SQLAlchemy omits fields whose values match the transaction's read — so on a single instance (owner "unchanged") a release committing between the claim's SELECT and its write survived, and the post-commit refresh handed the claimant a closed, ownerless row. That is the 409 you saw.

Both have deterministic repros that fail on the pre-fix code: a repository-level one that injects the release exactly between the SELECT and the write, and a route-level one through POST /v1/responses that holds the predecessor's release captive while the successor claims.

Review then walked the rest of that machinery, which added: an epoch CAS so racing claims serialize, revalidated takeover permission after losing a race (a loser was stealing the winner's live lease), a guard so a superseded creator never claims at all, epoch handoff to a registered winner, adoption of a same-process epoch advance on renewal instead of evicting the owner, and refusal to carry ownership across a process incarnation or an account change.

One shape is deliberately not fixed here and is filed as #1758: eviction does not cancel the creator it evicts, so multiple creators can race one key. That is a structural change to the hottest bridge path and wants its own design; the residual outcome is fail-closed (a retryable 409 plus a row the abandoned purge reclaims), not corruption.

Four occurrences across four unrelated PRs in three days should stop now — thanks for the reproduction data in the thread.