bug: recurring git-sync stalls permanently after a worker dies mid-submission (leaked deployment concurrency slot)

[rvandepu]

Current Behavior

The recurring git_repositories_sync Prefect deployment (entrypoint backend/infrahub/git/tasks:sync_remote_repositories, cron * * * * *, global concurrency limit = 1) stops running permanently after a task-worker is interrupted while one of its runs is in the PENDING / "Submitting" phase.

When a worker acquires the deployment's single concurrency slot and then dies (or the container is restarted) before the run transitions to Running, the flow run is orphaned in PENDING / "Submitting" (run_count=0, start_time=None) and its concurrency slot is never released (active_slots stays at 1/1). Because the deployment concurrency limit is 1, every subsequent per-minute scheduled run is created but sits in Scheduled forever and never executes. The git fetch never runs again — CoreRepository.commit freezes and no new "Sync git repo with origin" tasks appear — even though the schedule is active (status: READY, polled recently) and the worker is otherwise healthy (on-demand / event-driven flows keep completing normally).

Restarting the stack does not recover it: the orphaned run and the leaked slot persist in the Prefect PostgreSQL backend across restarts. The clean-up-deadlocks recurring flow is subject to the exact same mechanism (it also has a concurrency limit of 1) and was itself stuck "Submitting", so it could not clean this up.

Expected Behavior

• A flow run orphaned in PENDING / "Submitting" that never reached Running should eventually be marked crashed/cancelled and its deployment concurrency slot released, so the recurring schedule resumes on its own.
• Alternatively, the deployment concurrency slot lease should carry a TTL that expires when the holding worker stops renewing it, so a dead worker cannot hold the slot indefinitely.
• clean-up-deadlocks should be resilient to this failure mode (not blockable by its own single-slot limit) so it can actually recover leaked slots.

Steps to Reproduce

1. Configure a git repository so the recurring git_repositories_sync flow runs regularly.
2. While a "Sync Git Repositories" run is in the PENDING / "Submitting" phase (slot acquired, not yet Running), kill/restart the task-worker container.
3. Observe the run stays PENDING / "Submitting" (run_count=0, start_time=None), and the deployment concurrency limit reports active_slots=1 of limit=1.
4. Observe subsequent scheduled runs accumulate in Scheduled and never start; the repository commit stops advancing.
5. Restart the whole stack — the stall persists.

Additional Information

Diagnostics (Prefect 3.7.5 API):

• git_repositories_sync deployment: schedule active, paused: false, status: READY, last_polled recent.
• GET /v2/concurrency_limits/filter -> deployment:<id> shows limit=1, active_slots=1.
• POST /flow_runs/filter with state=PENDING -> the orphaned "Sync Git Repositories" (and "Clean up deadlocks") runs, state_name="Submitting", run_count=0, start_time=None, stuck for hours.

Manual recovery that worked:

1. Cancel the orphaned run: POST /flow_runs/{id}/set_state {state:{type:CANCELLED}, force:true} — this alone did not release the slot (the concurrency lease was orphaned from the run).
2. Reset the leaked slot: PATCH /v2/concurrency_limits/deployment:<id> with body {"active_slots":0}. The next scheduled run then acquired the slot and the fetch resumed normally.

Environment: Infrahub 1.10.5 (Docker Compose), Prefect 3.7.5, task-manager backed by PostgreSQL, host OS Linux x86_64.

[rvandepu]

A redacted diagnostics bundle for this issue is available for private hand-off (I'd rather not attach it to a public issue as it contains service logs and config). It was produced with infrahub-collect create (v2.1.0) on the affected Docker Compose deployment and includes the task-manager state (flow-runs, work-pools, work-queues, automations), per-service logs covering the incident window, and RabbitMQ/Redis/Neo4j status.

A couple of notes for whoever picks this up:

• By collection time the stall was already recovered manually (cancel the two orphaned Submitting runs, then reset the deployment's v2 concurrency-limit active_slots back to 0), so task-manager/flow-runs.txt shows the git-sync flow running normally again — the stall history lives in the task-worker logs.
• On top of infrahub-collect's built-in ******** masking, the instance hostname was redacted by hand. No secrets, GitLab URLs, emails, or public IPs are present in the bundle.

Happy to send it through whichever private channel OpsMill prefers (support email / Slack / Discord).

[iddocohen]

Root cause analysis for #9925

Issue: bug: recurring git-sync stalls permanently after a worker dies mid-submission (leaked deployment concurrency slot)
Based on: 3e57b5b7b59ce2a3cb308d49aa3e7704f58c91bb
Bug clarity: CLEAR
Code identification: RESOLVED

Root cause

The recurring git_repositories_sync (and clean-up-deadlocks) deployments run with a deployment-level concurrency_limit=1 / CANCEL_NEW; when a worker dies while a run is still in PENDING/"Submitting" (never reached Running), Prefect leaks that deployment's single concurrency slot, and Infrahub has no recovery path that both (a) detects a run stuck in PENDING that never emitted a heartbeat and (b) resets the leaked active_slots on the deployment's concurrency limit — so every future scheduled run stays Scheduled forever.

Affected files

• backend/infrahub/workflows/catalogue.py — lines 177-185 (GIT_REPOSITORIES_SYNC) and 654-662 (CLEAN_UP_DEADLOCKS): both set concurrency_limit=1 + ConcurrencyLimitStrategy.CANCEL_NEW on a * * * * * recurring deployment. This single slot is the thing that leaks.
• backend/infrahub/workflows/models.py — lines 70-84 (to_deployment): passes concurrency_limit / concurrency_options straight to create_deployment, which Prefect backs with a v2 global concurrency limit named deployment:{deployment_id}. That limit's active_slots is what is left stuck at 1/1.
• backend/infrahub/trigger/system.py — lines 13-36 (TRIGGER_CRASH_ZOMBIE_FLOWS): the only existing self-healing mechanism. It is a ProactiveEventTrigger armed by after={"prefect.flow-run.heartbeat"}, i.e. it only fires for a run that has already emitted at least one heartbeat and then goes quiet. A run killed in PENDING/"Submitting" (run_count=0, start_time=None) never entered Running, never emitted a heartbeat, so this automation never arms for it.
• backend/infrahub/task_manager/task.py — lines 366-411 (delete_flow_runs): the closest existing cleanup routine, but it filters on FlowRunFilterStartTime(before_=cutoff). The orphaned run has start_time=None, so it can never match — this path cannot catch the orphan either. It also only crashes/deletes the run; it never touches the concurrency slot.

Explanation

Both recurring flows are gated by a deployment-level concurrency limit of 1. In Prefect 3.7.5 that limit is implemented as a v2 global concurrency limit named deployment:{deployment_id}; a slot is acquired when a run enters submission (PENDING) and is expected to be released when the run reaches a terminal state through the normal worker lifecycle.

When the task-worker is killed/restarted after it has acquired the slot but before the run transitions to Running, that release never happens. The run is orphaned in PENDING/"Submitting" and active_slots stays at 1/1. Because the limit is 1, every subsequent per-minute scheduled run is created but can never acquire the slot, so it sits in Scheduled forever. The git fetch stops advancing even though the schedule is READY and the worker is otherwise healthy — which is exactly what the reporter observed.

Two independent defects combine to make this permanent and non-self-healing:

1. Detection gap. The crash-zombie-flows automation is keyed on the heartbeat stream, so it structurally cannot see a run that died before it ever started running. Nothing else scans for PENDING-forever runs. (The bundle confirms this automation is present and Enabled: True with the "expecting 1 event within 0:01:30" proactive window — it is active, and still could not help.)

2. Slot leak survives a crash. The reporter's manual recovery proves that forcing the orphan to CANCELLED/CRASHED alone did not release the slot — the concurrency lease is orphaned from the run. Recovery required an explicit PATCH /v2/concurrency_limits/deployment:<id> with {"active_slots":0}. So even if the detection gap were closed and the run were crashed, the schedule would still be stuck.

3. Self-blocking recovery. clean-up-deadlocks is subject to the identical mechanism (concurrency_limit=1 + CANCEL_NEW) and was itself stuck "Submitting", so it cannot be relied on to recover the system. (It is also the wrong tool — clean_up_deadlocks in backend/infrahub/locks/tasks.py only prunes stale Infrahub distributed-lock cache keys; it has no knowledge of Prefect concurrency slots.)

The diagnostic bundle (Infrahub 1.10.5, Prefect 3.7.5, Postgres-backed task-manager) is a healthy-baseline capture: both git_repositories_sync and clean-up-deadlocks run once per minute with their Sync git repo with origin / Fetch git repository sub-flows, and both workers submit continuously. It corroborates the topology, the per-minute cadence, the concurrency config, and the presence/enabled state of the crash-zombie-flows automation described in the issue; it does not capture the stalled moment itself.

Fix strategy

Approach: Add a periodic reconciliation of stuck submissions and leaked deployment concurrency slots, and make that recovery path structurally immune to the very deadlock it repairs.

• Reclaim leaked slots (primary). For each Infrahub-managed deployment that carries a concurrency_limit, read its v2 global concurrency limit by name deployment:{deployment_id} (client.read_global_concurrency_limit_by_name), count the deployment's genuinely-active runs (RUNNING + CANCELLING + PENDING that are recent / still plausibly mid-submission), and when active_slots exceeds that real count, (1) force the orphaned PENDING runs to CRASHED (set_flow_run_state(..., force=True), already used in delete_flow_runs), and (2) reset the slot with client.update_global_concurrency_limit(name, GlobalConcurrencyLimitUpdate(active_slots=<real_count>)). This is the programmatic equivalent of the two manual steps that worked.
• Make recovery un-deadlockable. The recovery must NOT itself be a concurrency_limit=1 / CANCEL_NEW deployment. Prefer either a new recurring INTERNAL deployment with no concurrency_limit (the reconciliation is idempotent and safe to overlap), or implement it as a Prefect automation. clean_up_deadlocks (backend/infrahub/locks/tasks.py) is a natural sibling home for the logic, but only if its own concurrency_limit=1 is removed/replaced first.
• Optionally close the detection gap with a companion trigger in backend/infrahub/trigger/system.py that catches runs stuck in PENDING (e.g. proactive: expect a prefect.flow-run.Running/terminal event within N minutes of prefect.flow-run.Pending). This is secondary — detection without slot reset does not fix the stall.

Scope:

• New reconciliation logic in backend/infrahub/task_manager/ (e.g. a new reconcile.py, or extend task.py) — the read/count/reset routine. Small-to-moderate.
• backend/infrahub/workflows/catalogue.py — register the reconciliation as a recurring INTERNAL deployment without a concurrency limit, and/or revisit the concurrency strategy of the existing recurring flows. Ask First (this is a workflow/concurrency-config change).
• Reuse an interval setting in backend/infrahub/config.py (there is already clean_up_deadlocks_interval_mins) or add one.
• A changelog fragment under changelog/.
• Tests (see notes).

Do NOT:

• Do NOT give the new recovery flow concurrency_limit=1 + CANCEL_NEW — that reintroduces the exact deadlock and is precisely why clean-up-deadlocks cannot self-recover today.
• Do NOT rely solely on crashing/cancelling the stuck run — the reporter proved that does not release the deployment slot. The slot must be reset explicitly via the v2 global concurrency limit API.
• Do NOT try to fix this only by widening the crash-zombie-flows heartbeat automation — a PENDING/"Submitting" run never emits a heartbeat, and crashing it still won't free the slot.
• Do NOT reuse delete_flow_runs' start_time-based filter to find the orphans — start_time is None for them; filter by state=PENDING + created/age instead.
• Do NOT delete-and-recreate the deployment as the recovery mechanism — it destroys schedule state and races the deployment setup.

Notes for downstream steps

• Name mapping: a deployment's concurrency limit lives as a v2 global concurrency limit named deployment:{deployment_id}. Enumerate deployments, build that name, and read/patch it. GlobalConcurrencyLimitUpdate.active_slots is a settable field (prefect/client/schemas/actions.py:952-958), and PATCH /v2/concurrency_limits/{name} is update_global_concurrency_limit (prefect/client/orchestration/_concurrency_limits/client.py:842).
• Race safety: reconciliation can run while a legitimate submission is genuinely mid-flight. Only reset when active_slots exceeds the real active count, and only crash PENDING runs that are clearly orphaned (e.g. run_count=0 / start_time=None and created older than a few minutes) so an in-progress submission is never killed. The reconciliation must be idempotent.
• Two recurring flows affected: both git_repositories_sync and clean-up-deadlocks (and any other concurrency_limit-bearing recurring deployment) are vulnerable — the reconciliation should iterate all such deployments, not special-case git-sync.
• Testing: the real Prefect + Postgres testcontainer stack (tests/e2e / component tests) can reproduce deterministically: create a deployment with concurrency_limit=1, drive active_slots to 1 without a live run to simulate the leak (e.g. leave a forced-PENDING run + patch the limit), then assert the reconciliation crashes the orphan and resets active_slots to the true active count so the next scheduled run acquires the slot. A pure-unit test can cover the count-vs-slots decision logic with an in-memory/adapter Prefect client.
• Ask-First boundary (per AGENTS.md): changing the concurrency strategy of existing recurring deployments and adding a new recurring deployment are CI/workflow-adjacent — confirm the chosen shape (drop concurrency on clean-up-deadlocks vs. new no-concurrency reconciliation flow vs. automation) with maintainers before implementing.