You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
bug: anchor-poison threshold (default 7) is unreachable because the retry circuit trips at a hardcoded 2 — dead previous_response_id is never abandoned on 1.24.0-beta.3 #1852
Pro (pool of 2 accounts, both status=active throughout)
Model(s) involved
gpt-5.6-sol
What happened?
Summary: http_responses_session_bridge_anchor_poison_failure_threshold defaults to 7, but the retry circuit that gates the same counter trips at a hardcoded 2. The poison threshold is therefore effectively unreachable, so a dead previous_response_id is never abandoned and the bridge re-injects it indefinitely.
Both values are compared against the same counter. In app/modules/proxy/_service/http_bridge/upstream_events.py:
Once the circuit opens at consecutive_failures == 2, subsequent requests on that bridge key are refused before reaching upstream (continuity_fail_closed reason=retry_circuit_cooldown_continuity_bound), so they do not record circuit failures. The counter can then only advance once per half-open probe, i.e. roughly once per cooldown, with the backoff escalating from 60s toward the 600s ceiling. Reaching 7 would take approximately five further open→cooldown→half-open cycles spanning on the order of 25 minutes. No interactive client waits that long, so in practice the anchor is never poisoned and the same dead anchor is retried forever.
The observed cascade, from one incident:
The stored anchor goes stale. Upstream reports previous_response_not_found.
The bridge re-injects the same dead anchor and discards the client's real context: original_items=248 trimmed_to=6, with fresh_replay_available=false. The client had supplied the entire conversation.
Second consecutive failure opens the circuit: event=opened failures=2 cooldown_seconds=60.0 detail=stream_incomplete.
The client emitted 10 requests in 10 seconds, every one answered 503 ... cooling down after repeated upstream timeouts; retry shortly. Because the turn is continuity-bound it cannot fail over to the second, healthy account. The client aborts the session long before the 60s cooldown elapses.
Net effect: the message says "retry shortly", but the shortest possible cooldown (60s) is far longer than the client's retry budget (~10-16s observed), and because the anchor is never poisoned, waiting out the cooldown just re-wedges on the next turn.
What did you expect to happen?
The anchor-poison path should be reachable at its documented default. Either the poison threshold should be less than or equal to the circuit threshold, or poisoning should be evaluated independently of the circuit gate.
A dead previous_response_id should be abandoned rather than re-injected once upstream has rejected it with previous_response_not_found.
When the bridge has just trimmed a full client history down to an anchor, and that anchor is rejected with zero response events, failing closed discards recoverable information the client already sent.
Steps to reproduce
Run uvx --from "codex-lb==1.24.0b3" codex-lb with a 2-account Pro pool, defaults otherwise.
Drive a long Codex Desktop conversation through http://127.0.0.1:2455/backend-api/codex until the HTTP responses session bridge holds a latest_response_id anchor.
Cause that anchor to become unknown upstream (observed here after an abrupt upstream websocket drop mid-turn; upstream then answers previous_response_not_found).
Continue the same conversation.
Observe two consecutive terminal_error detail=stream_incomplete, the circuit opening at failures=2, and a burst of 503 ... cooling down. Observe that http_bridge_retry_circuits.consecutive_failures stalls at 2 and never approaches 7, so durable_anchor_poisoned never fires.
Relevant logs
Identifiers truncated; response ids shortened.
15:34:25 INFO http_bridge_event event=fresh_reattach_anchor_injected bridge_key=sha256:99f1afc14b46 detail=response_id=resp_0f07fa9a...
15:34:25 WARN continuity_fail_closed surface=websocket_stream reason=previous_response_not_found
upstream_error_code=previous_response_not_found
diagnostics=previous_response_source=proxy_injected fresh_replay_available=false
15:34:25 WARN http_bridge_event event=terminal_error bridge_key=sha256:99f1afc14b46 detail=stream_incomplete
15:34:25 INFO http_bridge_event event=previous_response_recover_local detail=outcome=local_rebind_after_local_error
15:34:25 INFO http_bridge_retry_circuit event=half_open bridge_key=sha256:99f1afc14b46 failures=1
15:34:30 WARN continuity_fail_closed surface=websocket_stream reason=previous_response_not_found
upstream_error_code=previous_response_not_found
diagnostics=previous_response_source=client_supplied fresh_replay_available=false
15:34:30 WARN http_bridge_event event=terminal_error bridge_key=sha256:99f1afc14b46 detail=stream_incomplete
15:34:31 INFO session_anchor_injected response_id=resp_0f07fa9a... <-- same dead anchor re-injected
15:34:31 INFO store_context_input_trimmed original_items=248 trimmed_to=6 previous_response_id=resp_0f07fa9a...
15:34:33 WARN http_bridge_event event=reader_failure bridge_key=sha256:99f1afc14b46 pending=1 detail=stream_incomplete
error_message=Upstream websocket closed before response.completed: no close frame received or sent
response_events_seen=0 transport_classification=websocket_transport_error
15:34:33 ERROR proxy_error_response status=502 code="stream_incomplete"
15:34:33 WARN http_bridge_retry_circuit event=opened bridge_key=sha256:99f1afc14b46 failures=2 cooldown_seconds=60.0 detail=stream_incomplete
15:34:39 WARN continuity_fail_closed surface=http_bridge reason=retry_circuit_cooldown_continuity_bound
15:34:39 ERROR proxy_error_response status=503 code="upstream_request_timeout"
message="HTTP responses session bridge is cooling down after repeated upstream timeouts; retry shortly."
... 10 x 503 between 15:34:39 and 15:34:49 ...
Persisted circuit state, showing the counter stalled at the circuit threshold:
http_bridge_retry_circuits:
session_key_kind = thread_header
consecutive_failures = 2 <-- never advances toward the poison threshold of 7
cooldown_until_epoch = updated_at_epoch + 60.0
last_detail = stream_incomplete
Configuration / environment
All defaults; no .env.local. Windows 11, SQLite store. Two Pro accounts, both active, neither rate-limited or blocked.
Additional context
Workaround that appears to address it, for anyone hitting this before a fix:
This aligns the poison threshold with the hardcoded circuit threshold so the dead anchor is abandoned on the same failure that opens the circuit. The cost is more eager anchor discard, hence weaker prompt-cache reuse. It is a mitigation, not a fix — the underlying coupling remains.
Note also that a restart does not clear an already-wedged session, because the anchor is persisted in http_bridge_sessions.latest_response_id; the http_bridge_* tables have to be cleared.
Two smaller observations from the same investigation, offered in case they are useful:
store_context_input_trimmed original_items=248 trimmed_to=6 immediately before a fresh_replay_available=false failure looks like a missed recovery opportunity: the client had just supplied the full history that the bridge then reported as unavailable. http_responses_session_bridge_ambiguous_continuation_recovery_mode=client_full_history_once exists for this, but is documented as at-least-once, which is a strong deterrent for a default.
On this deployment the abrupt-drop signature (no close frame received or sent) accounted for 105 of 132 stream_incomplete events on 1.23.0 and continued at a similar daily rate on b3, which is what keeps feeding step 1 of the cascade.
Happy to supply fuller logs, the raw http_bridge_* table dump, or to test a patch.
Pre-flight checklist
Closest prior art, and why this is distinct:
1.24.0-beta.3) — same 503cooling downsurface. This report reproduces on b3, with fix(http-bridge): classify recovery error frames and poison same-anchor eventless failures #1841 in the build. fix(http-bridge): classify recovery error frames and poison same-anchor eventless failures #1841 bounded the wedge (it now self-clears after a cooldown instead of persisting) but did not remove the mechanism, and the misclassification it targeted is still observable (see step 2 below).response.createdsilence watchdog. That is a different timer from the retry-circuit backoff described here; the two are complementary.previous_response_idcontinuation loops on 1.23.0, about recognising the terse upstream rejection. This report is about threshold arithmetic, not classification recognition.anchor_poison_failure_threshold.codex-lb version
1.24.0-beta.3(confirmed viax-app-versionresponse header)Deployment method
uvx (codex-lb)
Client used against codex-lb
Codex app (desktop / web) —
Codex Desktop/0.148.0-alpha.15 (Windows 10.0.26200; x86_64)ChatGPT account plan(s) involved
Pro (pool of 2 accounts, both
status=activethroughout)Model(s) involved
gpt-5.6-solWhat happened?
Summary:
http_responses_session_bridge_anchor_poison_failure_thresholddefaults to7, but the retry circuit that gates the same counter trips at a hardcoded2. The poison threshold is therefore effectively unreachable, so a deadprevious_response_idis never abandoned and the bridge re-injects it indefinitely.Both values are compared against the same counter. In
app/modules/proxy/_service/http_bridge/upstream_events.py:and in
app/modules/proxy/_service/http_bridge/retry_circuit.py:Once the circuit opens at
consecutive_failures == 2, subsequent requests on that bridge key are refused before reaching upstream (continuity_fail_closed reason=retry_circuit_cooldown_continuity_bound), so they do not record circuit failures. The counter can then only advance once per half-open probe, i.e. roughly once per cooldown, with the backoff escalating from 60s toward the 600s ceiling. Reaching7would take approximately five further open→cooldown→half-open cycles spanning on the order of 25 minutes. No interactive client waits that long, so in practice the anchor is never poisoned and the same dead anchor is retried forever.The observed cascade, from one incident:
previous_response_not_found.detail=stream_incomplete. Both appear in the same second for the same bridge key — the websocket surface classifies correctly while the bridge path does not. This is the residual bug: HTTP bridge misclassifies upstream error frames as stream_incomplete; retry circuit + operation ledger wedge sessions permanently ("cooling down" 503 loop) #1830 described, still present with fix(http-bridge): classify recovery error frames and poison same-anchor eventless failures #1841 in the build.original_items=248 trimmed_to=6, withfresh_replay_available=false. The client had supplied the entire conversation.event=opened failures=2 cooldown_seconds=60.0 detail=stream_incomplete.503 ... cooling down after repeated upstream timeouts; retry shortly. Because the turn is continuity-bound it cannot fail over to the second, healthy account. The client aborts the session long before the 60s cooldown elapses.Net effect: the message says "retry shortly", but the shortest possible cooldown (60s) is far longer than the client's retry budget (~10-16s observed), and because the anchor is never poisoned, waiting out the cooldown just re-wedges on the next turn.
What did you expect to happen?
previous_response_idshould be abandoned rather than re-injected once upstream has rejected it withprevious_response_not_found._HTTP_BRIDGE_RETRY_CIRCUIT_FAILURE_THRESHOLD/_BASE_BACKOFF_SECONDSshould be settings-derived, so the cooldown can be aligned with real client retry budgets (this is the same class of fix as fix(proxy): name the pre-response eventless timeout honestly and derive its budget from settings #1633).Steps to reproduce
uvx --from "codex-lb==1.24.0b3" codex-lbwith a 2-account Pro pool, defaults otherwise.http://127.0.0.1:2455/backend-api/codexuntil the HTTP responses session bridge holds alatest_response_idanchor.previous_response_not_found).terminal_error detail=stream_incomplete, the circuit opening atfailures=2, and a burst of503 ... cooling down. Observe thathttp_bridge_retry_circuits.consecutive_failuresstalls at2and never approaches7, sodurable_anchor_poisonednever fires.Relevant logs
Identifiers truncated; response ids shortened.
Persisted circuit state, showing the counter stalled at the circuit threshold:
Configuration / environment
All defaults; no
.env.local. Windows 11, SQLite store. Two Pro accounts, both active, neither rate-limited or blocked.Additional context
Workaround that appears to address it, for anyone hitting this before a fix:
This aligns the poison threshold with the hardcoded circuit threshold so the dead anchor is abandoned on the same failure that opens the circuit. The cost is more eager anchor discard, hence weaker prompt-cache reuse. It is a mitigation, not a fix — the underlying coupling remains.
Note also that a restart does not clear an already-wedged session, because the anchor is persisted in
http_bridge_sessions.latest_response_id; thehttp_bridge_*tables have to be cleared.Two smaller observations from the same investigation, offered in case they are useful:
store_context_input_trimmed original_items=248 trimmed_to=6immediately before afresh_replay_available=falsefailure looks like a missed recovery opportunity: the client had just supplied the full history that the bridge then reported as unavailable.http_responses_session_bridge_ambiguous_continuation_recovery_mode=client_full_history_onceexists for this, but is documented as at-least-once, which is a strong deterrent for a default.no close frame received or sent) accounted for 105 of 132stream_incompleteevents on1.23.0and continued at a similar daily rate on b3, which is what keeps feeding step 1 of the cascade.Happy to supply fuller logs, the raw
http_bridge_*table dump, or to test a patch.