[Control Plane] Execution queue doesn't behave like a queue

[pocesar]

Describe the bug

When the max parallel executions are reached, any new exec async request will hard fail instead of returning an execution ID and queueing it to be executed. This forces to use an external queueing solution (Redis / Durable) just for making sure a burst of 300 requests don't bring everything down, which happens quite often and the AF CP get's killed and the pod is evicted then restarted.

Steps to reproduce

1. Go to '...'
2. Run '...'
3. See error

Expected behavior

Screenshots / Logs

Environment

• Control plane version: 0.1.127
• SDK version (if applicable):
• Deployment environment (local, docker, kubernetes, etc.): Kubernetes

Additional context

[AbirAbbas]

Triage

Classification: bug ✓
Suggested labels: bug

Likely affected files:

• ./reasoners/helpers.py (matched: help)

---

Triage by github-buddy

[AbirAbbas]

Shipping state, then the design question.

v0.1.137 (#1001) fixes the part that was making bursts destructive: a rejected async execute no longer writes anything — the per-agent gate and the queue check run before the payload blob and the two execution rows — 429 and 503 carry Retry-After plus a retry_after field, the async pool drains on shutdown (queued and in-flight jobs get a terminal control_plane_shutdown instead of staying running forever), and workers default to max(NumCPU, 16) with a 1024-deep queue, so ~1040 async executions are admitted before the first 503 — a much larger window than 0.1.127 had. Knobs: AGENTFIELD_EXEC_ASYNC_WORKERS, AGENTFIELD_EXEC_ASYNC_QUEUE_CAPACITY, AGENTFIELD_MAX_EXECUTE_BODY_BYTES. One thing worth setting now: AGENTFIELD_MAX_CONCURRENT_PER_AGENT (default 0 = unlimited), because with it unset the burst lands in your agent pod rather than at the control plane.

#1033 finishes the contract: the sync, restart and MCP lanes also gate before persisting (they still wrote a row first), an execution admitted right as the pool stops is terminated instead of abandoned in running, Retry-After is on every retryable rejection, and the README no longer advertises a "durable PostgreSQL queue with lease-based processing" — that doesn't exist, and I suspect that copy is what set the expectation that a burst would queue rather than 503. Sorry about that.

The title of this issue — admit at capacity and hand back a queued execution id instead of rejecting — is still open, and it's a design decision. Unbounded admission is precisely what evicts the pod, so it has to come with a durable, lease-based poller rather than a bigger in-memory queue. @santoshkumarradha the three calls:

1. Admit-and-queue with a durable queue (I'd keep bounded rejection until that exists, and build the queue as its own milestone).
2. Persist async executions as queued at admission and flip to running at dispatch, so GET agrees with the 202 body that already says "queued". Inventory is small (queued is already canonical in the SDKs/UI/predicates; needs one migration for the workflow_executions CHECK and two state-machine edges) — I'd land it with the queue work.
3. max_concurrent_per_agent releases the slot on the agent's 202 ACK, so it caps dispatch rate, not in-flight work. Holding until the terminal callback makes it a real cap but adds a wedge class (one lost callback pins the slot); I'd ship that opt-in.

@pocesar — for the burst case, would you rather get a 202 with a queued id and poll, or a 429/503 with Retry-After and retry from the client? And are these sync or async invocations? That changes which slice lands first.

[santoshkumarradha]

@pocesar Thanks !

Yes, I would prioritize bounded rejections first as immediate work and lets add a milestone for queue, actually potentially letting us even add priority and other scheduling ideas in modular way to control at constrained time which gets pulled in. @AbirAbbas can you scope a rough milestone for this?

[pocesar]

@AbirAbbas a clear rejection is fine for sync, but async I'd expect it to always return something, because I poll for the exec id from the outside until it either completes or fails/timeout