feat(proxy): congestion-aware per-API-key fair-share stream admission

[Soju06]

Summary

Adds a congestion-aware per-API-key fair-share gate on stream-lease admission, so one bursty key can no longer exhaust the pool-wide per-account stream slots and starve every other key.

Fixes #1535

• Work-conserving: with the pool below the configured congestion threshold (or the feature at its default-off setting), admission is byte-for-byte unchanged — no static per-key cap, idle capacity is never wasted.
• Under congestion (pool_inflight * 100 >= pool_capacity * threshold_pct, integer math over the selection's candidate account set): a key admits a new stream only while key_inflight + 1 <= max(2, pool_capacity // active_keys). Keys holding fewer than two streams can never be denied (light interactive keys are starvation-proof); over-share keys cannot reacquire freed slots until they drop back under, so freed capacity flows to the keys the congestion was starving.
• Denials reuse the existing local-capacity machinery via one new stable reason api_key_stream_fair_share registered in _LOCAL_ACCOUNT_CAP_ERROR_CODES and LOCAL_OVERLOAD_CODES: transport parks with waiting_for_account_capacity keepalives and re-runs selection (each retry re-evaluates the gate with fresh counters), then 429 rate_limit_error + Retry-After on budget exhaustion. Zero new transport code. Deliberately absent from the bridge soft-affinity reroute set — the gate is per-key and pool-wide, so rerouting to another account cannot help.
• Accounting: AccountLease carries api_key_id; RuntimeState.stream_key_inflight maintains per-account per-key counts under the existing runtime lock across acquire/release/stale-reclaim (entries deleted at zero, pruned with account runtime state). Candidate-set scoping means account-scoped keys are measured against the pool they can actually use.
• Gate placement: evaluated inside both selection paths' runtime-lock sections. The sticky path re-checks at commit (sticky DB I/O sits between its two lock sections; concurrent selections for one key could otherwise overshoot the share — the exact workload this targets). The unbound path evaluates and acquires in one lock section. Bypasses: keyless requests (still counted in pool inflight), reattach stage (resuming stranded in-flight work), non-stream leases, unlimited caps.
• Setting: proxy_api_key_fair_share_congestion_threshold_pct (env + nullable dashboard override with null-inherits-env, 0-100, default 0 = off), surfaced next to the per-account capacity limits in routing settings (en/ko/zh-CN).
• Observability: gauges codex_lb_stream_pool_capacity / codex_lb_stream_pool_inflight (the previously missing pool numerator/denominator), counter codex_lb_api_key_fair_share_rejections_total, and a warning log with the denial numbers. No per-key metric labels by design (cardinality); the log carries api_key_id.

Production motivation

Sanitized from the deployment in #1535: one agentic fan-out key held ~30-36 concurrent streams (91 live upstream streams, ~87% of a 13-account × 8-slot pool), one account additionally hit a genuine upstream 429, and every remaining candidate then filtered at account_stream_cap — selection degraded to no_accounts and all keys, including interactive ones holding 0-2 streams, received sustained 503 bursts (20-60% of requests per minute) while client retries amplified the load.

Simplicity gates (P1/P2)

• P1: default off (0 disables; the gate short-circuits before any counting work).
• P2 (MAX_SETTINGS_FIELDS 115 → 116): the congestion threshold cannot be a hardcoded default — the right value depends on pool size and workload mix, and 0 is the P1 off switch. The companion minimum-guarantee constant (2 streams) stayed hardcoded rather than becoming a second setting.
• Scope: per-replica / per-worker statistical enforcement, identical in kind to the existing partitioned per-account caps (see design.md D8). v1 excludes response-create leases, the opportunistic edge pre-check, and token-weighted shares (D9).

OpenSpec

openspec/changes/add-api-key-stream-fair-share/ — proposal, design (D1-D9), tasks, and deltas for proxy-admission-control, proxy-runtime-observability, frontend-architecture. openspec validate add-api-key-stream-fair-share --strict passes.

Testing

• New tests/unit/test_fair_share.py (pure math: thresholds incl. the >= boundary, capacity floors, min-guarantee dominance, divisor counting, message content).
• tests/unit/test_load_balancer_concurrency.py: per-key lease lifecycle (acquire/release/stale-reclaim/delete-at-zero), default-off characterization, congested deny-heavy/admit-light, re-admission after release, keyless bypass with pool counting, response-create non-interference, sticky-path denial.
• tests/unit/test_load_balancer_contract.py: unchanged defaults + denial shape.
• Settings: null-inherits-env service test, API round-trip + range rejection, settings-reference regen (ratchet bumped with justification above).
• Frontend: schemas/payload/routing-settings tests, three-locale keys.

Screenshots

Before	After

Local verification

• make lint / make typecheck / architecture ratchets: green (select_account and load_balancer.py stay inside their line budgets; new math lives in fair_share.py).
• pytest: unit 5,089 passed / integration 1,788 / e2e 27 / integration-bridge 185 (full local runs).
• frontend: typecheck, lint, vitest settings+i18n suites green.
• Local codex review --base origin/main: one P2 (multiprocess gauge aggregation) — fixed in the second commit; re-review clean.

🤖 Generated with Claude Code

[Soju06]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f5853e543f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".