Codex assign can delete active worker when cached status lags visible screen

[karstenjakobsen]

Summary

Codex workers created through asynchronous assign() can receive and start the task, but CAO may still classify the deferred submission as dropped and delete the active worker.

Environment

• CAO: 2.4.1, main commit d3efc643b342f262b44eae0c95021539b942d8c1
• Codex CLI: 0.148.0
• Provider: codex
• Backend: tmux

Reproduction

1. Launch a Codex supervisor through CAO.
2. Have it call assign() with a Codex worker and a small task that runs pwd and git branch --show-current, then callbacks with send_message.
3. Observe that the worker reaches a visible Working (...) state and can even deliver its callback.
4. The deferred-init confirmation can nevertheless time out against the cached status, log paste dropped, retry, and eventually tear down the worker.

Representative ordering from the server log:

Sent input to terminal: <worker>
wait_until_status [...]: waiting for {waiting_user_answer, processing, completed}, timeout=8.0s
Delivered 1 message(s) to terminal <supervisor>
Terminal <worker> status changed: completed
Terminal <worker> status changed: processing
wait_until_status [...]: timeout waiting for {waiting_user_answer, processing, completed}

Before the workaround, longer tasks showed Working (3s) or Working (7s) in the terminal scrollback, but CAO logged three Deferred assign ... not accepted (paste dropped) retries and then deleted each worker.

Cause

_confirm_worker_started_or_resubmit() already has a direct visible-screen fallback for an event-driven status-cache race, guarded by provider.supports_direct_status_probe. CodexProvider supports screen detection and its get_status() recognizes the visible Working (... • esc to interrupt) indicator, but it does not opt into the direct probe. Therefore the fallback is skipped for Codex.

Tested fix

class CodexProvider(BaseProvider):
    supports_direct_status_probe = True
    supports_screen_detection = True

After enabling that flag and restarting cao-server, the same assign() test completed successfully:

• no paste dropped retry;
• no worker teardown;
• callback reached the supervisor with the literal command output;
• worker remained available until explicitly deleted.

The cached eight-second wait still emitted a timeout, confirming that the direct probe—not a timing coincidence—prevented the false retry/teardown.

Suggested regression test

Simulate a Codex deferred assignment where cached status remains IDLE while the direct capture reports PROCESSING, and assert that confirmation succeeds without re-delivery or deletion.

[call-me-ram]

Triaged — confirmed, every claim checks out against current main (d697702). Nice writeup @karstenjakobsen.

The chain, verified with file:line:

• The deferred-submit confirm loop's direct visible-screen fallback is gated on supports_direct_status_probe (terminal_service.py:847-849), and CodexProvider never opts in — the flag defaults False at providers/base.py:180, and codex.py has zero occurrences of it despite supports_screen_detection = True (codex.py:793).
• Meanwhile Codex's detector genuinely recognizes the working indicator: TUI_PROGRESS_PATTERN (codex.py:82) matches (… esc to interrupt) → PROCESSING (codex.py:1367-1368), and it's pure stateless text analysis already run on rendered frames in production (get_status_from_screen, codex.py:1397-1409) — so a capture-pane snapshot is a proven-valid input, and the kiro-style dispatch-bookkeeping disqualifier documented at terminal_service.py:769-773 does not apply.
• The cached-IDLE lag mechanism is real: wait_until_status polls the monitor's cached _last_status (status_monitor.py:587-609), which only re-derives fresh when the cache says PROCESSING — no help when it's stuck IDLE. Under the default pyte path, detection fires on the rising edge and at 0.2s quiescence, and a repainting spinner defers quiescence — so cached IDLE can outlive the whole 8s confirm window. That is verbatim the mechanism fix(assign): prevent deferred-init retry loop from re-pasting into working OpenCode workers #496 documented for OpenCode.
• The outcome is double-barreled: up to 3 re-deliveries of the full task into the already-working pane (terminal_service.py:860-873), then delete_worker=True teardown of an active worker (terminal_service.py:970-980 → delete_terminal).

Lineage: this extends #496 (same bug class, fixed for OpenCode only — the probe was deliberately scoped to opted-in providers in review), not #407/#480 (that was the opposite direction: stale cached COMPLETED). MiniMax opted in at birth (minimax_code.py:89). This family is getting fixed one provider at a time: claude_code (#480), opencode (#496), minimax (#625), now codex.

Fix: your suggested one-liner is right — supports_direct_status_probe = True next to codex.py:793, plus your regression (cached IDLE + capture showing the working indicator → confirm succeeds, no re-delivery, no delete) slots naturally into test/services/test_deferred_submit_verification.py (see test_direct_probe_skipped_when_provider_not_opted_in at :146 for the shape).

Two follow-ups worth discussing while we're here:

1. Invert the opt-in: derive probe eligibility from supports_screen_detection plus an explicit opt-out for bookkeeping-dependent providers (kiro_cli), so the next provider doesn't re-hit this.
2. Even for non-opted providers, consider running the direct probe before the delete_worker=True arm specifically — re-submitting is recoverable, deleting an active worker is not, so the teardown decision deserves the strongest evidence available.

Related: #648 (Codex inbox delivery gate, same cached-status theme) and #558 (capture-pane as truth for the stuck-PROCESSING direction). PRs welcome — the one-liner + regression is a well-scoped first contribution.

[call-me-ram]

PR up: #714 — your one-liner plus the regression you suggested (real CodexProvider, real rendered Working frame, cached status pinned IDLE for every poll → confirm succeeds with nothing typed into the pane). Mutation-checked: reverting the flag turns the end-to-end test red at the re-delivery assertion. The two follow-ups from the triage (inverting the opt-in to an opt-out, and probing before the teardown arm even for non-opted providers) are deliberately NOT in scope — they're design changes that deserve their own discussion.