Cancelled asset backfill runs occupy concurrency pools with k8s run launcher

[alexbuyn]

What's the issue?

I have a partitioned asset with concurrency pool "test" and set concurrency limit on the pool to 1. I run Dagster in kubernetes. I trigger backfill for 4 partitions of the assets, which I expect to be run 1 by 1 due to the concurrency limit. It starts as expected - first run in progress, 3 queued. When I cancel the running partition materialization, other partitions DO NOT start running and are stuck in "queued" status.

What did you expect to happen?

After I cancelled the partition materialization run, I expect the next partition materialization to start.

How to reproduce?

• Create dagster helm chart deployment on kubernetes
• Create a partitioned asset with concurrency pool
• Set up pool limit to be 1
• Start a backfill with 2+ materializations
• Cancel first that's in progress
• Verify that other materialization do not start and cancelled materialization is still occupying the concurrency slot.

Dagster version

1.10.1

Deployment type

Dagster Helm chart

Deployment details

No response

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.