Component
API Server / GraphQL (task manager / Prefect workflow orchestration)
Infrahub version
1.10.3 (not version-specific — originates in the Prefect deployment concurrency model, prefect==3.7.5)
Current Behavior
A flow run occupies its Prefect deployment concurrency slot from the moment it enters PENDING, not when it reaches RUNNING. If a task worker is killed/stopped after moving a run SCHEDULED → PENDING but before advancing it to RUNNING, the run is stranded in PENDING and holds the slot indefinitely.
For a deployment with concurrency_limit=1 + CANCEL_NEW — notably clean-up-deadlocks (cron * * * * *) — this wedges the deployment: every subsequent scheduled run collides with the stuck PENDING run and is immediately cancelled ("Deployment concurrency limit reached."), so the flow never runs again until the stuck run is cleared manually. Seen in production: the deadlock-cleanup flow stopped running and its scheduled runs were being cancelled; deleting the stuck PENDING run unblocked it.
Expected Behavior
A run abandoned in PENDING by a dead worker should not hold a concurrency slot forever. The system should recover on its own — the slot released and/or the abandoned run moved to a terminal state — so scheduled runs of the affected deployment resume without manual intervention.
Steps to Reproduce
- Use a deployment with
concurrency_limit=1 and collision_strategy=CANCEL_NEW (e.g. clean-up-deadlocks).
- Transition one of its runs
SCHEDULED → PENDING and stop there (e.g. kill the task worker between the PENDING and RUNNING transitions). The deployment's active_slots is now 1.
- Trigger/await another scheduled run of the same deployment.
- The new run is rejected into
CANCELLED ("Deployment concurrency limit reached."); the stuck PENDING run keeps active_slots at 1 until it is manually deleted/crashed.
Unit-level reproduction of the state-machine behavior: backend/tests/unit/workflows/test_deployment_concurrency.py.
Additional Information
Root cause. Prefect (prefect==3.7.5) acquires the deployment slot on the SCHEDULED → PENDING transition via the SecureFlowConcurrencySlots rule (prefect/server/orchestration/core_policy.py, TO_STATES = {PENDING}) — its docstring notes PENDING is the target "so a worker can secure a slot before provisioning infrastructure." A run parked in PENDING therefore legitimately owns the slot, and nothing frees it while it sits there.
Why built-in recovery doesn't save us. Securing the slot creates a lease (default TTL initial_deployment_lease_duration = 300s) the worker is meant to renew, and the Repossessor service (repossessor.py, every 15s) expires stale leases to release the slot. But active_slots is persisted in Postgres while the lease lives in PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE, which defaults to in-memory and is not overridden in the shipped docker-compose.yml task-manager service. On a task-manager restart the in-memory lease is lost but Postgres active_slots=1 persists, so the Repossessor finds no lease to expire and the slot leaks permanently — matching the production symptom.
Mitigation. Enabling redis-backed concurrency lease storage resolves this: set PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE=prefect_redis.lease_storage (and PREFECT_SERVER_DOCKET_URL to redis) on the task-manager server so leases persist across restarts and the Repossessor can expire abandoned leases and release the slot — as the testcontainers stack (python_testcontainers/infrahub_testcontainers/docker-compose.test.yml) already does. Redis is already a hard dependency (prefect-redis).
Also affected (all concurrency_limit=1): git_repositories_sync (CANCEL_NEW) and webhook-configure (ENQUEUE, which backs up behind the stuck run instead of cancelling).
References: backend/infrahub/workflows/catalogue.py, backend/infrahub/workflows/models.py (to_deployment()); Prefect SecureFlowConcurrencySlots / ReleaseFlowConcurrencySlots / Repossessor / server/concurrency/lease_storage/memory.py.
Component
API Server / GraphQL (task manager / Prefect workflow orchestration)
Infrahub version
1.10.3 (not version-specific — originates in the Prefect deployment concurrency model,
prefect==3.7.5)Current Behavior
A flow run occupies its Prefect deployment concurrency slot from the moment it enters
PENDING, not when it reachesRUNNING. If a task worker is killed/stopped after moving a runSCHEDULED → PENDINGbut before advancing it toRUNNING, the run is stranded inPENDINGand holds the slot indefinitely.For a deployment with
concurrency_limit=1+CANCEL_NEW— notablyclean-up-deadlocks(cron* * * * *) — this wedges the deployment: every subsequent scheduled run collides with the stuckPENDINGrun and is immediately cancelled ("Deployment concurrency limit reached."), so the flow never runs again until the stuck run is cleared manually. Seen in production: the deadlock-cleanup flow stopped running and its scheduled runs were being cancelled; deleting the stuckPENDINGrun unblocked it.Expected Behavior
A run abandoned in
PENDINGby a dead worker should not hold a concurrency slot forever. The system should recover on its own — the slot released and/or the abandoned run moved to a terminal state — so scheduled runs of the affected deployment resume without manual intervention.Steps to Reproduce
concurrency_limit=1andcollision_strategy=CANCEL_NEW(e.g.clean-up-deadlocks).SCHEDULED → PENDINGand stop there (e.g. kill the task worker between thePENDINGandRUNNINGtransitions). The deployment'sactive_slotsis now 1.CANCELLED("Deployment concurrency limit reached."); the stuckPENDINGrun keepsactive_slotsat 1 until it is manually deleted/crashed.Unit-level reproduction of the state-machine behavior:
backend/tests/unit/workflows/test_deployment_concurrency.py.Additional Information
Root cause. Prefect (
prefect==3.7.5) acquires the deployment slot on theSCHEDULED → PENDINGtransition via theSecureFlowConcurrencySlotsrule (prefect/server/orchestration/core_policy.py,TO_STATES = {PENDING}) — its docstring notes PENDING is the target "so a worker can secure a slot before provisioning infrastructure." A run parked inPENDINGtherefore legitimately owns the slot, and nothing frees it while it sits there.Why built-in recovery doesn't save us. Securing the slot creates a lease (default TTL
initial_deployment_lease_duration= 300s) the worker is meant to renew, and the Repossessor service (repossessor.py, every 15s) expires stale leases to release the slot. Butactive_slotsis persisted in Postgres while the lease lives inPREFECT_SERVER_CONCURRENCY_LEASE_STORAGE, which defaults to in-memory and is not overridden in the shippeddocker-compose.ymltask-manager service. On a task-manager restart the in-memory lease is lost but Postgresactive_slots=1persists, so the Repossessor finds no lease to expire and the slot leaks permanently — matching the production symptom.Mitigation. Enabling redis-backed concurrency lease storage resolves this: set
PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE=prefect_redis.lease_storage(andPREFECT_SERVER_DOCKET_URLto redis) on the task-manager server so leases persist across restarts and the Repossessor can expire abandoned leases and release the slot — as the testcontainers stack (python_testcontainers/infrahub_testcontainers/docker-compose.test.yml) already does. Redis is already a hard dependency (prefect-redis).Also affected (all
concurrency_limit=1):git_repositories_sync(CANCEL_NEW) andwebhook-configure(ENQUEUE, which backs up behind the stuck run instead of cancelling).References:
backend/infrahub/workflows/catalogue.py,backend/infrahub/workflows/models.py(to_deployment()); PrefectSecureFlowConcurrencySlots/ReleaseFlowConcurrencySlots/Repossessor/server/concurrency/lease_storage/memory.py.