fix(control-plane): stop counting agent restarts as failed executions

[santoshkumarradha]

Why

New users produce failed executions they did not cause. The largest single source is the edit-restart loop.

A long-running agent node restarts constantly during development — save a file, Ctrl-C, or a syntax error kills the process. The control plane keeps believing the node is healthy for as long as it takes the health checker to notice (check_interval × consecutive_failures, ~30s) or the heartbeat to go stale (60s). Every call inside that window created an execution record, failed to dial the dead process, and was recorded as failed.

Reproduced against a real Python SDK agent on a clean install, telemetry captured locally:

Sequence	Before	After
Call fired while the agent is down, process returns 4s later	HTTP 502, agent_restart_orphaned, execution_failed	HTTP 200 succeeded, execution_completed
Call to a node already known down	execution row created, then failed	503 node_unavailable, no execution row
Quickstart curl before python main.py	error_category: internal_error	error_category: target_not_found

What changed

Three fixes, in the order a call meets them.

1. Fail fast on a node we already know is down. prepareExecutionForTarget checked pending-approval but never health — the versioned routing path filtered healthy nodes via selectVersionedAgent, the unversioned path (what every new user has) did not. It now returns 503 node_unavailable before the execution record exists. A request we never dispatched is a rejected request, not a failed execution — which is already how an unknown node behaves, so this only makes the two consistent. Mirrors the check reasoners.go has always had on the legacy proxy route.

Only definitively-down states are rejected. unknown (no heartbeat yet) and degraded still go through, or the first call of every session would fail.

2. Absorb the restart when we do not know the node is down. A dial failure is the one transport error that is unambiguously safe to retry — no bytes reached the agent, so nothing can execute twice. net.OpError.Op == "dial" is the portable signal; a read/write failure mid-request is deliberately not retried.

The dispatch waits for the node to come back and replays against its current address, which may be a different port. Recovery is read from the node record — a fresh instance_id, or a heartbeat that advanced — rather than guessed by blind retrying. Bounded by agent_restart_grace (default 15s, well inside the 90s agent call timeout, AGENTFIELD_AGENT_RESTART_GRACE / node_health.agent_restart_grace to tune, negative to disable). Serverless targets are excluded: they have no resident process to come back, so a dial failure there is a real outage and must surface immediately.

3. Close the detection gap. When the wait expires the node is demoted to inactive via UpdateAgentHealthAtomic, conditional on the heartbeat we last observed — so if it heartbeated while we waited, the health checker keeps ownership. The next caller then takes path 1 and fails fast instead of repeating the wait.

Two things the end-to-end run turned up

The orphan reaper was failing the execution the retry was saving. Re-registration is exactly what ends a restart and what triggers MarkAgentExecutionsOrphaned. The first end-to-end run returned HTTP 502 with an agent_restart_orphaned message even though the retry had succeeded and telemetry recorded execution_completed — the sync caller was reading the reaped row. Executions marked awaiting_agent_restart are now exempt, on both the executions table and the workflow_executions row the DAG UI and dashboard counts read. Those are not orphans: the control plane still owns them and is about to re-dispatch.

agent 'x' not found reported as internal_error. That is the quickstart curl run before python main.py — a normal mistake presented as a broken control plane. It now classifies as target_not_found, a category that already existed in canonicalFailureCategory and that nothing ever assigned.

Scope

Not addressed here, and worth separate PRs:

• The three sweeps (MarkAgentExecutionsOrphaned, MarkStaleExecutions, MarkStaleWorkflowExecutions) write terminal failures with no event published, so execution_started ≠ completed + failed in telemetry.
• validation and permission_denied are still dead categories; an input-schema rejection reports as agent_error.
• Execution telemetry carries no usage_context, so the local dev loop cannot be separated from real traffic in product metrics.

Verification

• New unit tests: 16 in internal/handlers, 1 in internal/storage, 1 in internal/config. New code is at 89–100% per function.
• go build ./..., go vet, gofmt clean. golangci-lint reports nothing in the changed files.
• Full go test ./... shows the same pre-existing failures as the base commit (internal/cli, internal/packages, internal/skillkit — they need a furrow binary and interactive secret prompts). Confirmed identical by re-running against a stash of these changes.
• The documented quickstart (af init → af server → python main.py → the printed curl) runs clean end to end on the final build.

🤖 Generated with Claude Code

[AbirAbbas]

Rebased onto latest main — the branch was carrying #937's already-merged desktop commits, which is what the conflict was. The diff is now just the 10 control-plane files.

Verified the fix locally against a real Python SDK agent (isolated control plane, fresh home): a call fired while the agent was down with the process returning 4s later gives HTTP 200 succeeded with no failed row (baseline main: instant 504 + failed execution); a known-down node gets an instant 503 with no execution row; grace expiry demotes the node; the quickstart curl before python main.py reports target_not_found.

Also pushed a hardening commit for issues that came out of adversarial review of the retry path:

• serverless nodes are exempt from the fail-fast gate — they have no heartbeat loop, so the presence sweep marks them inactive as a matter of course and the gate would have 503'd every serverless invocation permanently
• replay requests bypass the gate (a replay hit never contacts the agent)
• a cancel that lands during the restart wait now wins — the loop re-reads the execution before replaying, so the agent is never handed work the caller disowned
• the wait aborts early once the node record says the node is definitively down, so queued dispatches at one dead node don't serialize the async worker pool for a full grace each
• the awaiting_agent_restart hold is released on every exit path, and the test fake now enforces the same conditional-heartbeat semantics as LocalStorage

Each fix has a test. Full handlers/storage/config/types suites green locally.

[github-actions]

📊 Coverage gate

Thresholds from .coverage-gate.toml: per-surface ≥ 84%, aggregate ≥ 85%, max per-surface regression ≤ 1.0 pp, max aggregate regression ≤ 0.50 pp.

Surface	Current	Baseline	Δ
control-plane	87.20%	87.40%	↓ -0.20 pp	🟡
sdk-go	92.90%	92.00%	↑ +0.90 pp	🟢
sdk-python	94.20%	93.73%	↑ +0.47 pp	🟢
sdk-typescript	91.39%	90.42%	↑ +0.97 pp	🟢
web-ui	84.76%	84.79%	↓ -0.03 pp	🟡
aggregate	85.66%	85.75%	↓ -0.09 pp	🟡

✅ Gate passed

No surface regressed past the allowed threshold and the aggregate stayed above the floor.

[github-actions]

📐 Patch coverage gate

Threshold: 80% on lines this PR touches vs origin/main (from .coverage-gate.toml:thresholds.min_patch).

Surface	Touched lines	Patch coverage	Status
control-plane	279	93.00%	✅
sdk-go	0	—	➖ no changes
sdk-python	0	—	➖ no changes
sdk-typescript	0	—	➖ no changes
web-ui	0	—	➖ no changes

✅ Patch gate passed

Every surface whose lines were touched by this PR has patch coverage at or above the threshold.

[AbirAbbas]

Rebased, adversarially reviewed, hardened (serverless gate exemption, cancel-during-wait, early demotion abort, replay bypass), and verified live against a real Python SDK agent — restart absorb returns 200 succeeded where main returns 504 + a failed row. Full verification details in the comment above.