Skip to content

retryable: true on a capacity failure carries no backoff signal, so a controller can spin through a capacity window #103

Description

@patriyang

Observation

#101 (closing #96) added failureClass / retryable to job records and a single-attempt model fallback. When both the requested model and the resolved backup are at capacity, the job ends as:

{ "failureClass": "capacity",
  "retryable": true,
  "modelFallback": { "from": "gpt-5.5", "to": "gpt-5.6-terra", "reason": "capacity" } }

retryable: true is correct in the sense the field is defined — the turn produced nothing, so repeating it is safe. But it says nothing about when, and the plugin gives a caller nothing else to go on.

Why that can spin

Fallback resolution is deterministic: CODEX_COMPANION_FALLBACK_MODELconfig.fallbackModelmodel/list discovery (the advertised default). None of those change between one dispatch and the next.

So a controller that reads retryable: true and re-dispatches immediately resolves the same pair of models and can fail identically. Capacity windows are transient in time, not in configuration — but nothing in the record communicates that waiting is the thing that helps. During a broad capacity window affecting several models at once (the common shape, since that is when a backup is also busy), a naive retry loop makes no progress while consuming dispatches.

This is most likely to bite the automated callers the field was added for: fix-issue-batch and the issue worker fan out N agents that all dispatch Codex at once, which is exactly the situation that produces correlated capacity rejections in the first place.

Scope

Not a regression — the field is new, and before #101 a controller had to string-match the summary and hand-roll everything anyway, so this is strictly better than what it replaced. The gap is that the contract stops one step short of being actionable.

Surfaced while explaining the double-capacity path after #101 merged; the behavior itself is covered by the all-models-at-capacity test, which asserts the single attempt and the retryable classification but says nothing about pacing.

Non-prescriptive notes

Several defensible shapes, and the choice is a contract decision rather than a patch:

  1. Carry a hint in the record — a retryAfterMs / notBefore alongside retryable, so the signal is data rather than folklore. Server-provided Retry-After would be ideal if the app-server surfaces one; otherwise a fixed floor.
  2. Document the caller's obligation — leave the field as-is and state in the README and command docs that retryable: true means "safe to repeat after a backoff", with a suggested minimum.
  3. Back off inside the worker — wait and retry the fallback once more before giving up. Cheapest for callers, but it lengthens a failing dispatch and interacts with the turn-stall watchdog and the Give the background Codex flows a wait contract that does not depend on harness re-invocation #94 wait contract, so it needs care.
  4. Widen the fallback instead of retrying the same one — remember which models were tried on this job and let discovery pick a third. Helps only when capacity is model-specific, which is precisely not the case being described here.

Worth settling before more controllers start branching on retryable, since each one that ships its own pacing rule makes the contract harder to change later.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions