[Control Plane] Kubernetes new deployments causes every in-flight run to fail and never recover

[pocesar]

Describe the bug

When there's a reasoner that is acting like an orchestrator, and can take 5-10 minutes to complete, it has a high chance of destroying the entire run (which is composed by multiple reasoners and skills) with the "in-flight reasoner cannot be revived" error.

The only workaround is to re-run the same entrypoint reasoner with the same inputs and start again. Happily, when this is done, the run will blast through until the step where it got killed due the persisted intermediary steps and the 1h cache that is enabled.

Steps to reproduce

1. Go to '...'
2. Run '...'
3. See error

Expected behavior

When a reasoner or skill gets killed, there should be a way to restart the root run, keeping the same execution ID so the external consumers don't need to implement any sort of retries or re-submission.
It's a trade-off for the immutability and deterministic behavior that is expected when you call the same reasoner with the same input, but generates different execution / run IDS, should be an opt-in flag

Screenshots / Logs

Environment

• Control plane version: 0.1.127
• SDK version (if applicable): Python SDK
• Deployment environment (local, docker, kubernetes, etc.): Kubernetes

Additional context

[AbirAbbas]

Triage

Classification: bug ✓
Suggested labels: bug

Likely affected files:

• ./reasoners/helpers.py (matched: reasoner, reasoners, reasoner, reasoner, reasoner, reasoner, help)
• ./reasoners/models.py (matched: reasoner, reasoners, reasoner, reasoner, reasoner, reasoner)
• ./reasoners/init.py (matched: reasoner, reasoners, reasoner, reasoner, reasoner, reasoner)
• ./reasoners/conversational.py (matched: reasoner, reasoners, reasoner, reasoner, reasoner, reasoner)
• ./reasoners/repo_config.py (matched: reasoner, reasoners, reasoner, reasoner, reasoner, reasoner)

---

Triage by github-buddy

[AbirAbbas]

Where this stands after v0.1.137 (released yesterday):

The root cause is fixed. Re-registration used to synchronously fail every non-terminal execution for the agent id, so under maxSurge the new pod killed the old pod's still-running work. Now executions carry the instance_id that dispatched them, the reap is scoped to the departing instance and deferred by AGENTFIELD_AGENT_DRAIN_GRACE (default 60s) — a completion arriving inside that window wins — and dispatch to a pod that just announced shutdown is held for AGENTFIELD_AGENT_RESTART_GRACE and retried on the replacement instead of 503ing (#1004). All three SDKs drain in-flight reasoners under AGENTFIELD_SHUTDOWN_TIMEOUT (#1000, #1006), and af server itself drains on SIGTERM (#1010, #1011).

Two things you need to set for 5–10 minute orchestrators, because the defaults are sized for short reasoners:

• AGENTFIELD_AGENT_DRAIN_GRACE ≥ the longest reasoner you want to protect (e.g. 15m). The reap fires that long after the replacement registers, regardless of the departing pod's own drain budget.
• terminationGracePeriodSeconds ≥ AGENTFIELD_SHUTDOWN_TIMEOUT + 15s on the agent pods, and AGENTFIELD_SHUTDOWN_TIMEOUT ≥ that same longest reasoner.

One caveat that matters for Kubernetes: this assumes replicas: 1 per node id. A second replica registering is indistinguishable from a replacement today, so the deferred reap would fail the still-alive sibling's in-flight work 60s later — which is exactly your symptom. #1031 adds AGENTFIELD_AGENT_ORPHAN_REAP_ENABLED=false as the escape hatch if you scale horizontally (the stale sweep stays as the backstop), and exposes agent_node_id / instance_id on execution reads so a reaped execution can be attributed to the instance that owned it. #1030 ships the tuning recipe as one doc.

Could you confirm two things about your setup: replicas per agent Deployment, and whether the agent's version changes on each deploy (image tag = version)? The second one matters — see below.

Restarting under the same execution id: not planning to do same-id reuse. It breaks the immutability the execution/workflow tables, the per-execution VC/DID records and replay matching all assume. The addressable alternative is a forward restarted_as lineage pointer on the original execution, surfaced on GET /executions/{id}, so a consumer polling the original id can follow the restart without re-submitting.

@santoshkumarradha three calls for you here:

1. Same-id restart vs forward restarted_as pointer — I'd do the pointer.
2. replicas > 1 of one node id: registry rows are (node_id, version) with a last-writer-wins instance_id, and presence leases are per node id, so a sibling looks like a replacement. Supporting it properly means per-instance registration/presence and stamping executions with the instance that actually served the request. Kill switch above buys time; the real fix is a design change.
3. Version-bumping rollouts bypass the drain machinery entirely: GetAgent orders version ASC LIMIT 1, so with two versioned rows an unversioned dispatch resolves the lowest version (the retired pod), and selectVersionedAgent's weighted routing is unreachable. Proposal: order by registered_at DESC for id-scoped reads and route through ListAgentVersions when more than one version is registered. Not "treat a version change as re-registration and reap" — that would kill in-flight work on intentional canaries.

[santoshkumarradha]

Yes agree on all 3 @AbirAbbas GTG

[pocesar]

@AbirAbbas just a single replica, but due to rolling strategy it's:

[new revision]--[healthcheck green]--------------[starts serving]
[old revision]-------------[deferred prestop hook with sleep]--[got SIGTERM]--[removed]

Versions are exactly the same at all times. But if we could (or already can with the new additions) stop serving requests to the old revision and forward the same request to the new revision this would be interesting.

I still can't work a solution for the orchestrator approach, since it's purely code that is issuing the external app.call. It's not the same as snapshotting the python code execution, so it would need to start from scratch right?