Skip to content

bug: anchor-poison threshold (default 7) is unreachable because the retry circuit trips at a hardcoded 2 — dead previous_response_id is never abandoned on 1.24.0-beta.3 #1852

Description

@nickwilson159

Pre-flight checklist

  • I searched open and closed issues and confirmed this is not a duplicate.
  • I am running a supported codex-lb version.
  • I can reproduce the problem with the steps below.

Closest prior art, and why this is distinct:

codex-lb version

1.24.0-beta.3 (confirmed via x-app-version response header)

Deployment method

uvx (codex-lb)

Client used against codex-lb

Codex app (desktop / web) — Codex Desktop/0.148.0-alpha.15 (Windows 10.0.26200; x86_64)

ChatGPT account plan(s) involved

Pro (pool of 2 accounts, both status=active throughout)

Model(s) involved

gpt-5.6-sol

What happened?

Summary: http_responses_session_bridge_anchor_poison_failure_threshold defaults to 7, but the retry circuit that gates the same counter trips at a hardcoded 2. The poison threshold is therefore effectively unreachable, so a dead previous_response_id is never abandoned and the bridge re-injects it indefinitely.

Both values are compared against the same counter. In app/modules/proxy/_service/http_bridge/upstream_events.py:

consecutive_failures = await self._record_http_bridge_retry_circuit_failure_for_attempt_selection(
    session, detail=retry_circuit_detail, selection=retry_circuit_attempt_selection,
)
poison_candidate_detail = _http_bridge_anchor_poison_detail(retry_circuit_detail)
if (
    poison_candidate_detail is not None
    and observed_response_events == 0
    and consecutive_failures is not None
    and consecutive_failures
    >= _service_get_settings().http_responses_session_bridge_anchor_poison_failure_threshold
):
    poison_detail = poison_candidate_detail

and in app/modules/proxy/_service/http_bridge/retry_circuit.py:

_HTTP_BRIDGE_RETRY_CIRCUIT_FAILURE_THRESHOLD = 2
_HTTP_BRIDGE_RETRY_CIRCUIT_BASE_BACKOFF_SECONDS = 60.0
_HTTP_BRIDGE_RETRY_CIRCUIT_MAX_BACKOFF_SECONDS = 600.0

Once the circuit opens at consecutive_failures == 2, subsequent requests on that bridge key are refused before reaching upstream (continuity_fail_closed reason=retry_circuit_cooldown_continuity_bound), so they do not record circuit failures. The counter can then only advance once per half-open probe, i.e. roughly once per cooldown, with the backoff escalating from 60s toward the 600s ceiling. Reaching 7 would take approximately five further open→cooldown→half-open cycles spanning on the order of 25 minutes. No interactive client waits that long, so in practice the anchor is never poisoned and the same dead anchor is retried forever.

The observed cascade, from one incident:

  1. The stored anchor goes stale. Upstream reports previous_response_not_found.
  2. The bridge records it as detail=stream_incomplete. Both appear in the same second for the same bridge key — the websocket surface classifies correctly while the bridge path does not. This is the residual bug: HTTP bridge misclassifies upstream error frames as stream_incomplete; retry circuit + operation ledger wedge sessions permanently ("cooling down" 503 loop) #1830 described, still present with fix(http-bridge): classify recovery error frames and poison same-anchor eventless failures #1841 in the build.
  3. The bridge re-injects the same dead anchor and discards the client's real context: original_items=248 trimmed_to=6, with fresh_replay_available=false. The client had supplied the entire conversation.
  4. Second consecutive failure opens the circuit: event=opened failures=2 cooldown_seconds=60.0 detail=stream_incomplete.
  5. The client emitted 10 requests in 10 seconds, every one answered 503 ... cooling down after repeated upstream timeouts; retry shortly. Because the turn is continuity-bound it cannot fail over to the second, healthy account. The client aborts the session long before the 60s cooldown elapses.

Net effect: the message says "retry shortly", but the shortest possible cooldown (60s) is far longer than the client's retry budget (~10-16s observed), and because the anchor is never poisoned, waiting out the cooldown just re-wedges on the next turn.

What did you expect to happen?

  • The anchor-poison path should be reachable at its documented default. Either the poison threshold should be less than or equal to the circuit threshold, or poisoning should be evaluated independently of the circuit gate.
  • A dead previous_response_id should be abandoned rather than re-injected once upstream has rejected it with previous_response_not_found.
  • _HTTP_BRIDGE_RETRY_CIRCUIT_FAILURE_THRESHOLD / _BASE_BACKOFF_SECONDS should be settings-derived, so the cooldown can be aligned with real client retry budgets (this is the same class of fix as fix(proxy): name the pre-response eventless timeout honestly and derive its budget from settings #1633).
  • When the bridge has just trimmed a full client history down to an anchor, and that anchor is rejected with zero response events, failing closed discards recoverable information the client already sent.

Steps to reproduce

  1. Run uvx --from "codex-lb==1.24.0b3" codex-lb with a 2-account Pro pool, defaults otherwise.
  2. Drive a long Codex Desktop conversation through http://127.0.0.1:2455/backend-api/codex until the HTTP responses session bridge holds a latest_response_id anchor.
  3. Cause that anchor to become unknown upstream (observed here after an abrupt upstream websocket drop mid-turn; upstream then answers previous_response_not_found).
  4. Continue the same conversation.
  5. Observe two consecutive terminal_error detail=stream_incomplete, the circuit opening at failures=2, and a burst of 503 ... cooling down. Observe that http_bridge_retry_circuits.consecutive_failures stalls at 2 and never approaches 7, so durable_anchor_poisoned never fires.

Relevant logs

Identifiers truncated; response ids shortened.

15:34:25 INFO  http_bridge_event event=fresh_reattach_anchor_injected bridge_key=sha256:99f1afc14b46 detail=response_id=resp_0f07fa9a...
15:34:25 WARN  continuity_fail_closed surface=websocket_stream reason=previous_response_not_found
               upstream_error_code=previous_response_not_found
               diagnostics=previous_response_source=proxy_injected fresh_replay_available=false
15:34:25 WARN  http_bridge_event event=terminal_error bridge_key=sha256:99f1afc14b46 detail=stream_incomplete
15:34:25 INFO  http_bridge_event event=previous_response_recover_local detail=outcome=local_rebind_after_local_error
15:34:25 INFO  http_bridge_retry_circuit event=half_open bridge_key=sha256:99f1afc14b46 failures=1

15:34:30 WARN  continuity_fail_closed surface=websocket_stream reason=previous_response_not_found
               upstream_error_code=previous_response_not_found
               diagnostics=previous_response_source=client_supplied fresh_replay_available=false
15:34:30 WARN  http_bridge_event event=terminal_error bridge_key=sha256:99f1afc14b46 detail=stream_incomplete

15:34:31 INFO  session_anchor_injected response_id=resp_0f07fa9a...          <-- same dead anchor re-injected
15:34:31 INFO  store_context_input_trimmed original_items=248 trimmed_to=6 previous_response_id=resp_0f07fa9a...

15:34:33 WARN  http_bridge_event event=reader_failure bridge_key=sha256:99f1afc14b46 pending=1 detail=stream_incomplete
               error_message=Upstream websocket closed before response.completed: no close frame received or sent
               response_events_seen=0 transport_classification=websocket_transport_error
15:34:33 ERROR proxy_error_response status=502 code="stream_incomplete"
15:34:33 WARN  http_bridge_retry_circuit event=opened bridge_key=sha256:99f1afc14b46 failures=2 cooldown_seconds=60.0 detail=stream_incomplete

15:34:39 WARN  continuity_fail_closed surface=http_bridge reason=retry_circuit_cooldown_continuity_bound
15:34:39 ERROR proxy_error_response status=503 code="upstream_request_timeout"
               message="HTTP responses session bridge is cooling down after repeated upstream timeouts; retry shortly."
   ... 10 x 503 between 15:34:39 and 15:34:49 ...

Persisted circuit state, showing the counter stalled at the circuit threshold:

http_bridge_retry_circuits:
  session_key_kind      = thread_header
  consecutive_failures  = 2          <-- never advances toward the poison threshold of 7
  cooldown_until_epoch  = updated_at_epoch + 60.0
  last_detail           = stream_incomplete

Configuration / environment

All defaults; no .env.local. Windows 11, SQLite store. Two Pro accounts, both active, neither rate-limited or blocked.

Additional context

Workaround that appears to address it, for anyone hitting this before a fix:

CODEX_LB_HTTP_RESPONSES_SESSION_BRIDGE_ANCHOR_POISON_FAILURE_THRESHOLD=2

This aligns the poison threshold with the hardcoded circuit threshold so the dead anchor is abandoned on the same failure that opens the circuit. The cost is more eager anchor discard, hence weaker prompt-cache reuse. It is a mitigation, not a fix — the underlying coupling remains.

Note also that a restart does not clear an already-wedged session, because the anchor is persisted in http_bridge_sessions.latest_response_id; the http_bridge_* tables have to be cleared.

Two smaller observations from the same investigation, offered in case they are useful:

  • store_context_input_trimmed original_items=248 trimmed_to=6 immediately before a fresh_replay_available=false failure looks like a missed recovery opportunity: the client had just supplied the full history that the bridge then reported as unavailable. http_responses_session_bridge_ambiguous_continuation_recovery_mode=client_full_history_once exists for this, but is documented as at-least-once, which is a strong deterrent for a default.
  • On this deployment the abrupt-drop signature (no close frame received or sent) accounted for 105 of 132 stream_incomplete events on 1.23.0 and continued at a similar daily rate on b3, which is what keeps feeding step 1 of the cascade.

Happy to supply fuller logs, the raw http_bridge_* table dump, or to test a patch.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions