fix(proxy): scope backend Codex affinity by thread identity

[leventov]

Summary

Codex now sends three deliberately different identities on backend Responses traffic:

• session-id identifies the root agent tree and is shared by root and child threads;
• thread-id identifies one actual Codex thread and changes for a child or fork while remaining stable across turns, compact, and resume;
• prompt_cache_key intentionally defaults to the shared session-id, so it is a cache hint for the root tree rather than a thread identifier.

codex-lb still treats the first accepted session header as all three things at once. Because session-id has precedence over thread-id, independent root and child threads can share account-affinity, HTTP-bridge, direct-WebSocket replay/tool-continuity, and thread-goal routing scope. The bridge's historical (session header, prompt_cache_key) composition no longer separates those threads because current Codex sends (S, S) for every thread in the tree.

This PR separates those roles. Backend Responses, compact, direct WebSocket continuity, HTTP bridge lanes, and thread-goal requests use a versioned, source-separated internal thread key derived from (process session, thread-id) when thread-id is available. Requests without thread-id retain the existing process-session compatibility behavior. The client-supplied prompt_cache_key continues upstream unchanged, and every exact hard owner remains authoritative.

The new account-affinity row is bounded soft locality. A live or durable bridge using the same logical thread identity is hard continuity for the life of that bridge. Those are deliberately different strengths even though they originate from the same request identity.

Type of change

• fix: — bug fix (no behavior change beyond the bug)
• feat: — new user-facing feature or capability
• refactor: — internal refactor (no behavior change, no API change)
• docs: — documentation only
• chore: / ci: / build: — tooling, CI, packaging
• test: — test-only change
• Breaking change

Linked issue: Fixes #1702

OpenSpec

• This PR includes / updates an OpenSpec change
• Not applicable — bug fix that matches the existing spec
• Not applicable — docs / CI / chore only
• This PR touches a codex-faithful path and preserves upstream-equivalent request identity and prompt-cache behavior

Change directory: openspec/changes/scope-codex-affinity-by-thread/

This change does not introduce a competing affinity capability. Its delta spec modifies the existing backend-session, bare-process-session, and unanchored bridge-concurrency requirements. In particular, it removes the now-false statement that an explicit prompt_cache_key distinguishes Codex agent threads.

Upstream identity contract

The design follows the contract established by upstream Codex rather than inferring identity from subagent markers:

Value	Current meaning	Root/child relationship	Lifecycle
session-id	Root-tree/process session	Shared	Restored on child resume
thread-id	Actual Codex thread	Distinct	Stable across turns, compact, and resume; new on fork
prompt_cache_key	Upstream prompt-cache hint	Shared by default	Defaults to session-id; explicit overrides remain allowed
x-codex-turn-state	Explicit account/session continuation	Thread/turn-specific	Hard when supplied by the client
previous_response_id	Upstream response continuation	Exact object identity	Hard and account-owned

The relevant upstream history is:

• openai/codex#20437 introduced a distinct SessionId specifically so every thread under one root can share a session ID while retaining a distinct thread ID.
• openai/codex#33035 intentionally changed the default prompt_cache_key from thread ID to session ID. Its regression test asserts that root and child have different thread-id values but the same session-based prompt-cache key.
• Upstream session tests restore a persisted parent session-id for a resumed child while retaining the child's own thread-id. Remote compact tests keep the thread identity stable, and fork creates a different thread identity.

Consequently, changing or rewriting the upstream prompt_cache_key to make it thread-specific would be a protocol divergence, not a fix.

Regression origin and relationship to earlier codex-lb work

This is protocol-evolution drift across otherwise valid earlier changes:

• codex-lb#1169 correctly isolated concurrent unanchored bridge requests and composed the process session with the then-thread-specific prompt_cache_key for sequential child isolation. Upstream #33035 landed afterwards and collapsed that composition from (S, thread-cache-key) to (S, S).
• codex-lb#1220 correctly stopped a proxy-generated per-connection turn state from rotating reconnects away from a stable client session. Its specification predates the upstream split between tree-level session identity and thread identity, so current reconnect locality is still too broad.
• codex-lb#1382 separated bare process-session locality from hard turn-state ownership and introduced source-separated soft session keys. This PR reuses that safety model; it does not weaken legacy raw owners or rebind the process-wide row.
• codex-lb#1657 landed while this work was being prepared and added durable recovery for hard bridge operations. This PR is rebased on that result and carries thread_header through hard-kind, owner-forwarding, durable-operation, and account-neutral full-resend recovery paths without weakening its fences.
• codex-lb#1677 landed before publication and keeps eventless idle bridge retirements out of the retry circuit. This PR is rebased on its merged head, does not alter the circuit classifier, and preserves its response-event accounting while changing bridge identity granularity.
• codex-lb#1309 is concurrent, overlapping work around marker-detected subagent TTL and bridge lifecycle. This PR does not adapt that broad schema/settings design: markers prove a role, not identity, and current Codex already supplies authoritative thread-id.
• codex-lb#1679 and codex-lb#1680 are concurrent legacy-owner retirement and bridge-recovery fixes. This PR does not include them; it keeps raw owners hard and exact bridge aliases authoritative so a later rebase can preserve their guarded retirement/restart semantics rather than creating another path around an unavailable hard owner.

Confirmed defects

Session-header parsing hides thread identity

The shared session-header parser currently accepts these names in order: session_id, session-id, x-codex-session-id, x-codex-conversation-id, and thread-id. A normal current Codex request has both session-id and thread-id, so the process session always wins. Callers cannot tell whether the returned string is tree identity or thread identity.

HTTP bridge composition has collapsed

The bridge comments and OpenSpec requirement assume every child has a stable, child-specific explicit prompt_cache_key. Current Codex deliberately sends the shared process session instead. Hashing (session-id, prompt_cache_key) therefore gives the same canonical key to root and children, including sequential requests that do not trigger the concurrent request-scoped fork path.

Direct WebSocket continuity is process-scoped

The direct WebSocket replay/tool-continuity cache is indexed through the same session-header parser. Independent threads under one Codex process can therefore read or replace one process-wide retained continuity state. Fixing only account selection or bridge keying would leave this correctness leak.

Responses and compact account locality are process-scoped

Both paths classify a bare backend session before considering any other soft locality. Every thread in a root tree therefore competes for one process-level mapping. Exact turn state, response, file, and bridge owners are already stronger and remain so.

Thread-goal routing ignores its exact payload identity

/backend-api/codex/thread/goal/{get,set,clear} carries threadId in the payload but currently selects an account using only generic request headers. Once distinct threads can establish different account locality, goal access must use the payload's exact thread identity rather than the broader process session.

Request-log conversation grouping is already correct

Codex request logs use the raw thread-id as conversation_id. This PR does not change that behavior. RequestLog.session_id and internal owner-lookup scope are separate concerns and must not be rewritten merely to make the dashboard grouping look uniform.

Production evidence and its limits

Read-only aggregate analysis of one long-running root-tree session found:

• hundreds of distinct thread-id values sharing one process session;
• successful traffic on four accounts;
• about 48% cached-input reuse on completion-ordered within-thread account changes, versus about 96% when the account stayed the same;
• large-context requests after a change frequently retained only a small fixed cache prefix before rebuilding locality.

This proves that an account change is expensive. It does not prove that every status-line account change is caused by stale identity granularity:

• per-account stream and response-create caps were disabled in the inspected deployment, so cap spillover was not the cause there;
• several changes followed stream_incomplete, websocket_connection_limit_reached, previous_response_not_found, or a client disconnect;
• request logs are ordered by completed requests, so overlapping sockets can look like sequential account transitions even when no live thread was rebound.

The acceptance claim for this PR is therefore bounded: it removes cross-thread identity collisions and makes reconnect/account locality stable per thread. It does not promise that health failover, quota exhaustion, explicit restart, or concurrent completion can never expose another account.

Changes

1. Parse identities independently

• Extract raw process session and raw thread-id into separate typed fields.
• Stop treating thread-id as merely the last alias in a generic session header list.
• Derive a versioned, source-separated opaque key from (process session, thread-id) when both are present.
• When only thread-id is present, derive a separate thread-only key rather than falling back to a client-reproducible raw identity.
• When thread-id is absent, preserve the current process-session behavior.
• Keep raw process-session identity available separately for legacy hard-owner lookups and rolling-upgrade compatibility.

The derivation is namespacing, not secrecy. Its encoded source/version must be structurally unreachable by normalized client headers, matching the safety property introduced for bare process-session keys in #1382.

2. Use bounded thread locality for backend Responses and compact

• Classify the derived thread key as bounded soft affinity for backend Responses and compact.
• Keep the client-supplied prompt_cache_key unchanged in the upstream body.
• Give explicit client turn state, exact previous-response owner, file owner, conversation owner, live/durable bridge owner, replay/reattach state, and every other proven hard owner precedence over the thread-local hint.
• Fail closed on conflicting hard owners exactly as today.
• Keep process-session fallback only for clients that do not send thread-id.

Initial placement and cache-sharing boundary

Upstream deliberately gives sibling threads the same prompt-cache key, so a new thread should not be distributed to another account merely because its thread-local row does not exist yet. Initial thread placement must prefer an eligible existing process-session soft owner, then persist the admitted account under the bounded thread key. If the process-session owner cannot be used and the request is genuinely movable, ordinary safe selection can seed the thread on an alternate account without rewriting the process row.

Current Codex sends thread-id on the first root request, so no earlier bare process request may exist to initialize that preference. A normal first admission therefore inserts the process preference if absent and upserts the initiating thread row in one repository transaction. Insert-if-absent is essential because concurrent siblings may race; atomicity is essential because a process default without its initiating thread is false affinity evidence. A recovery-probe reservation remains reversible until its runtime CAS commits, so it persists only the thread row and does not publish an immutable process default that cannot safely be compensating-deleted after another sibling sees it.

After a thread row exists, that row supplies its reconnect locality. A sibling that later moves because of a justified failure must not move every other thread sharing the process session. This two-level behavior preserves session-wide cache colocation by default while containing necessary divergence to one thread.

The existing selection architecture expresses this without a second policy engine, new setting, schema, or durable child row.

3. Scope direct WebSocket continuity by thread

• Index replay prefix, last completed response, and unresolved tool state by the derived thread identity plus the existing API-key scope.
• Preserve aliases for explicit client turn state and proxy-generated turn state according to the existing reconnect contract.
• Do not fall back from a request with thread-id to the old process-wide in-memory continuity entry: that state may belong to a sibling.
• Keep the cache count-bounded and ensure active turn completion/touch keeps a long-lived thread's locality current; selection-time TTL alone must not let an active multi-hour socket silently expire before reconnect.
• Preserve exact response-ID fallback only where uniqueness and API-key scope are already proven.

No durable migration is needed for this in-memory cache. A process replacement already drops it.

4. Scope HTTP bridge lanes by thread

• Use the derived thread identity as the canonical backend Responses bridge lane for both ordinary Responses and compact forwarding.
• Treat an established local or durable bridge as hard continuity even though the pre-bridge sticky mapping is bounded soft locality.
• Preserve internal request-scoped fork lanes for truly concurrent unanchored requests; thread identity fixes canonical granularity but does not replace fix(proxy): isolate unanchored parallel bridge requests #1169's collision/reservation machinery.
• Preserve exact turn-state and previous-response aliases and their account ownership.
• Preserve forwarded trusted affinity keys verbatim. Do not hash or namespace them a second time on the owner replica.

Existing bridge compatibility

Do not blindly fall back to the old (session-id, prompt_cache_key) canonical key when thread-id is present. With current Codex that key is shared (S, S) and can identify a sibling's bridge.

An old bridge may be recovered only through exact hard evidence already tied to that bridge, such as a client turn-state or previous-response alias. A request with only soft thread identity creates/uses the new canonical lane; unaliased old lanes expire through existing idle/durable cleanup. Mixed-version tests must prove that old and new canonical rows may coexist without one thread attaching to another thread's history, that trusted forwarded keys are not re-derived, and that an exact old alias remains usable until natural retirement.

5. Route thread-goal requests by payload thread identity

• Validate and normalize payload threadId as the exact thread identity for goal get/set/clear account locality.
• Combine it with the process session when available using the same internal derivation as Responses and compact.
• Preserve process-session fallback for legacy payload/header forms only where the endpoint's existing compatibility contract allows it.
• Keep goal protocol fallback/error behavior unchanged.

6. Preserve every hard owner and legacy check

The following remain authoritative and are outside soft affinity mobility:

• explicit client x-codex-turn-state;
• previous-response ownership;
• live file pins and other account-scoped input objects;
• live and durable bridge aliases;
• conversation ownership where it can be proven;
• replay, reattach, unresolved tool, and other transport state;
• API-key account scope, model/security routing, and single-account policy;
• raw legacy codex_session rows that may represent historical turn-state ownership.

Raw legacy process-session lookup must run separately from the new thread selection key. A raw hit wins conservatively; disagreement with another hard owner fails closed. The implementation must not globally repurpose the current _owner_lookup_session_id_from_headers() helper, because its callers mix request-log scope, response-owner lookup, direct-WebSocket continuity, and legacy compatibility. Introduce role-specific typed helpers instead.

Rejected alternatives

Adapt #1309's marker/TTL/schema design

Rejected. It adds a setting, migration, is_subagent state, dashboard surface, and special cleanup lifecycle to infer information now present directly in thread-id. It also applies only to recognized child markers, while root threads, resumes, forks, and clients without those markers still need correct identity. The open PR's review history illustrates the resulting keying, forwarding, reuse, cleanup, and parent-row coupling.

Detect only x-openai-subagent or x-codex-parent-thread-id

Rejected. Those headers describe request role/provenance, not the child's identity. Marker presence can be missing, values can identify the parent, and ordinary root/fork/resume traffic still has thread identity. Marker values must remain compatibility metadata, not affinity keys.

Use x-codex-parent-thread-id as the child key

Rejected. It groups children by their parent and therefore reproduces the same collision one level down. It also changes meaning across nesting depth and cannot identify the root thread.

Treat explicit prompt_cache_key as the thread identifier

Rejected after upstream archaeology. This was true when #1169 was designed, but upstream #33035 intentionally changed the default to shared session-id and tests that root and child use the same cache key.

Derive affinity from (session-id, prompt_cache_key)

Rejected. For current Codex it reduces to (S, S) for every root and child, so it is only a more opaque spelling of the existing process-wide collision.

Rewrite prompt_cache_key to thread-id

Rejected. It contradicts current upstream behavior and discards deliberate root-tree cache sharing. codex-lb must forward an explicit client cache key unchanged.

Use thread-id alone everywhere

Rejected as the general representation. Source/version separation prevents a client-controlled raw value from colliding with historical hard rows, and including process session preserves namespace separation between process trees. Thread-only derivation is only the compatibility fallback when no process session exists.

Use one thread key with one strength and lifetime everywhere

Rejected. Identity and ownership are not the same concept. Sticky account locality is bounded and soft; a live/durable bridge, exact response, file, or turn state is hard. Sharing a derivation helper must not flatten these capabilities.

Keep account/cache locality process-wide and change only bridge/replay state

Rejected as the complete fix, though it is the minimum correctness subset. It would stop cross-thread history/replay contamination but leave every thread reconnect competing for one process-wide soft mapping. The selected design instead seeds new threads from the process preference to preserve default cache colocation, then gives each thread stable bounded locality so a justified divergence does not move its siblings.

If implementation cannot preserve that initial seeding without significant new machinery, this narrower alternative must be reconsidered explicitly; it must not reappear accidentally through a missing-row fallback.

Independently balance every previously unseen thread

Rejected. It would distribute siblings across accounts even while the shared process owner is healthy, defeating the cache-sharing intent behind upstream prompt_cache_key=session-id. A thread key is a containment boundary, not a reason to force immediate distribution.

Create durable per-thread CODEX_SESSION rows

Rejected. Durable rows are not automatically cleaned up and would turn every short-lived child into permanent routing state. The ordinary locality row must use an existing bounded lifecycle; only proven bridge/object ownership is durable/hard.

Add a new sticky-session kind, database schema, or setting

Rejected as unnecessary. Existing typed source provenance, bounded sticky lifecycle, and hard-owner machinery can express the behavior. A new enum, migration, dashboard control, or operator knob would add compatibility and cleanup obligations without resolving identity more accurately.

Rebind the shared process-session row after spill/failover

Rejected. Moving that row moves every sibling, requires publication and rollback across transport admission, and recreates the distributed settlement problem rejected by #1382. A selected alternate may establish one thread's bounded row without rewriting the process default.

Rely only on #1169 request-scoped parallel fork lanes

Rejected. Those lanes correctly handle concurrent unanchored collisions but do not separate sequential root/child reuse once (session-id, prompt_cache_key) has collapsed.

Fall back to the legacy bridge canonical key during migration

Rejected for requests with thread-id. The old key can denote any sibling in the tree. Only exact hard aliases can safely recover an old bridge.

Change request-log conversation grouping or owner lookup wholesale

Rejected. Conversation grouping already uses thread-id. Request logging, response-object ownership, soft selection, bridge identity, and in-memory replay are different roles. A global helper replacement would create migration and ownership regressions unrelated to the defect.

Non-goals

• Guarantee that one thread never changes accounts after quota exhaustion, account health failure, explicit safe restart, or another documented failover condition.
• Make account-scoped previous responses, files, conversations, or turn states portable.
• Change account-health, quota, retry, circuit-breaker, or WebSocket transport policy.
• Replace fix(proxy): isolate unanchored parallel bridge requests #1169's concurrent unanchored fork/reservation design.
• Change upstream request headers or payload cache controls.
• Change dashboard conversation grouping or historical request logs.
• Add settings, migrations, dashboard UI, or operator cleanup work.

Safety invariants

1. Thread locality never overrides exact hard ownership.
2. Conflicting hard evidence fails before upstream dispatch.
3. A raw legacy owner remains hard and wins over new soft rows.
4. A normal first admission may initialize an absent process preference and its thread row atomically; later movement may rebind only that bounded thread row, never an existing process preference or sibling row.
5. Once transport, replay, or durable bridge ownership exists, account switching follows the existing hard-continuity rules.
6. prompt_cache_key is forwarded unchanged.
7. An old shared bridge key is never accepted as thread identity without an exact hard alias.
8. Forwarded trusted affinity is not re-derived on the receiver.
9. Active long-lived threads do not lose reconnect locality merely because a selection-time TTL elapsed while their socket remained active.
10. No client header can reproduce the internal source-separated key namespace.

Expected implementation footprint

The change is conceptually narrow but crosses the places that currently reuse the generic session parser:

• affinity identity extraction and selection policy;
• backend Responses and compact classification;
• direct WebSocket continuity/replay indexing;
• HTTP bridge canonical keying and owner-forward compatibility;
• thread-goal payload routing;
• focused unit/integration tests and existing OpenSpec requirements.

It should not require schema, settings, dashboard, API response-shape, or upstream payload changes. If implementation expands into those areas or needs a generalized owner-resolution rewrite, stop and reassess scope. The bridge migration is the only expected deep compatibility boundary; if it alone makes the PR exceed one reviewable object-level concern, split it as a stacked PR over the typed identity/account-affinity change rather than broadening the base PR.

Test plan

This PR includes focused product-path regressions for:

• identity extraction with both headers, thread-id only, process session only, blank values, mixed casing, and values that resemble internal keys;
• root and child sharing (session-id, prompt_cache_key) but receiving distinct internal thread keys while the upstream payload remains unchanged;
• initial sibling placement inheriting an eligible process-session preference;
• first-thread process initialization when an exact owner chooses the admitted account;
• one safely moved thread retaining its alternate locality without moving a sibling or rewriting the process row;
• Responses and compact parity;
• direct WebSocket interleaving and reconnect without cross-thread retained response/tool state;
• an unknown exact client turn alias neither borrowing nor replacing broader thread replay state;
• a long-lived active socket retaining reconnect locality beyond the nominal bounded-affinity age;
• sequential and concurrent HTTP bridge root/child isolation;
• current-key and exact-alias legacy bridge recovery, including mixed-version forwarded affinity without double derivation;
• a legacy forwarded request with thread-id never reconstructing the shared process lane after a missing generated turn alias;
• thread-goal get/set/clear selecting by payload threadId;
• turn-state, previous-response, file, bridge, replay, and raw legacy owner precedence/conflict behavior;
• process-session fallback when thread-id is absent;
• request-log conversation_id=thread-id remaining unchanged;
• prompt-cache forwarding remaining byte-for-byte unchanged.

Commands run on the final local tree:

pytest -q tests/unit/test_proxy_utils.py
pytest -q tests/unit/test_proxy_http_bridge.py
pytest -q tests/unit/test_load_balancer_concurrency.py tests/integration/test_proxy_sticky_sessions.py
pytest -q tests/integration/test_proxy_responses.py tests/integration/test_proxy_compact.py \
  tests/integration/test_proxy_websocket_responses.py \
  tests/integration/test_proxy_sticky_sessions.py \
  tests/integration/test_proxy_api_extended.py \
  tests/integration/test_http_responses_bridge.py
ruff check .
ruff format --check .
ty check <changed Python files>
python scripts/check_proxy_architecture.py
npx --yes @fission-ai/openspec@latest validate --specs --strict --no-interactive
npx --yes @fission-ai/openspec@latest validate scope-codex-affinity-by-thread \
  --type change --strict --no-interactive

Validation

• tests/unit/test_proxy_utils.py: 1007 passed.
• tests/unit/test_proxy_http_bridge.py: 548 passed on the rebased head.
• selection concurrency + sticky repository/product path: 142 passed.
• protocol integration set: 441 passed before the one-commit fix(http-bridge): keep idle retirements out of retry circuit #1677 rebase; the affected HTTP bridge integration suite then passed 126/126 on the rebased head (plus one upstream Starlette/httpx deprecation warning).
• Ruff check/format, proxy architecture, compileall, changed-file ty, all 49 main OpenSpecs, and the strict change validation passed.

Repository-wide ty check currently reports four diagnostics in untouched websocket/helpers.py and tool_call_dedupe.py; changed-file type checking is clean. GitHub CI and current-head cloud Codex review remain merge gates after publication and are not inferred from local results.

Simplicity

• The fix works with zero configuration.
• No new required setup step.
• No new CODEX_LB_* setting.
• No database migration or new sticky-session kind.
• No README section, .env.example entry, dashboard navigation, or UI.
• Existing bounded locality and hard-owner mechanisms are reused.

Screenshots / output

Not applicable: no dashboard-visible change. The product-path regression test_backend_thread_rows_route_sibling_responses_and_compact_independently demonstrates distinct root/child account locality while both upstream calls retain the same process-wide prompt_cache_key.

Checklist

• Title is in Conventional Commits format.
• Linked the exact issue with Fixes #N / Closes #N, or stated partial coverage explicitly.
• Added externally failing-path regression coverage.
• Ran the relevant local CI subset on the final head.
• openspec validate --specs and /opsx:verify are clean.
• Simplicity gates P1-P5 are satisfied.
• Current-head GitHub CI is green.
• Current-head @codex review is clean or every finding is addressed.
• GitHub reports mergeable=CLEAN.
• CHANGELOG.md is not edited manually.

[chatgpt-codex-connector]

To use Codex here, create a Codex account and connect to github.

[leventov]

@Komzpa @Soju06 I think this is important issue, it seems to cause significant quota drain in big multi-agent conversations

[leventov]

@Komzpa @Soju06 please review

[Komzpa]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. 👍

Reviewed commit: 8d274c8e1f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".