Summary
concurrency.job-discriminator (added for #24037) correctly fixes the static gh-aw-conclusion-<workflow-id> group - but only if you know to set it. For a workflow_dispatch fan-out workflow the default group is essentially always wrong, and since queue: max landed, getting it wrong no longer produces an error or a cancellation. It produces a silent throughput collapse.
I'd like the compiler to say something when it can see the combination is wrong.
What we hit
We dispatch a compiled workflow once per target repository from an orchestrator, up to 50 concurrently. Every job that does real work is per-target: the top-level group and the agent group both key on inputs.target_repo. Only conclusion doesn't:
conclusion:
concurrency:
group: "gh-aw-conclusion-campaign-prompt-eval"
cancel-in-progress: false
queue: max
So all 50 runs queue for one conclusion slot. The job itself takes 22-36s, but the wait before it starts grows across the sweep:
| conclusion # |
queued before start |
| 1 |
7s |
| 2 |
55s |
| 3 |
1m12s |
| 4 |
1m43s |
| 5 |
4m42s |
| 6 |
5m07s |
| 7 |
6m16s |
| 8 |
8m28s |
That's a drain rate of roughly 2.4 runs/minute. Our orchestrator waits for the run to report completed, and completed needs conclusion - so each target held its parallelism slot on a queue shared with all the others, and effective concurrency collapsed toward 2.4 regardless of what we configured. On one run, activation, agent, detection and safe_outputs were all green at 14:50:12 while conclusion was still pending eleven minutes later.
Why this is worth a separate issue from #24037
#24037 reported the cancellation symptom: Canceling since a higher priority waiting request for gh-aw-conclusion-<id> exists. That's loud. It shows up in the run, and you go and read the docs.
With queue: max, runs queue instead of being cancelled. The symptom changed from "a job got cancelled" to "everything succeeds, just slowly" - which is strictly harder to notice. Nothing fails. There's no warning at compile time. The runs are all green. You only find it if you go and diff job start times across a fan-out and notice the queue wait growing linearly.
It took us a while to get there, and our first fix was in the wrong place - we changed our orchestrator to stop waiting on conclusion before realising the knob already existed. The docs are clear once you're reading Fan-Out Concurrency, but nothing points you at that page when you need it.
Suggestion
The compiler already knows both halves of this at compile time: the workflow is triggered by workflow_dispatch with inputs, and no concurrency.job-discriminator is set. That combination means "every dispatch of this workflow serializes on one conclusion slot", which is almost never what the author wants.
Options, roughly in order of how much I'd like them:
- Warn at compile time. Something like: "workflow_dispatch workflow has no
concurrency.job-discriminator; the generated conclusion group is shared by all dispatches of this workflow. Set a discriminator (e.g. ${{ github.run_id }}) to give each dispatch its own slot." Cheap, no behaviour change, and it lands exactly when the author can act on it.
- Default the discriminator to
${{ github.run_id }} for workflow_dispatch triggers. Per-run is the safe default for dispatch; a shared slot across unrelated dispatches doesn't seem to protect anything. Behaviour change, so a warning may be the better first step.
- Mention it in the
workflow_dispatch docs, not only under concurrency - that's where someone building fan-out is actually reading.
Happy to send a PR for (1) if you'd take it.
Environment
- gh-aw
v0.86.2 (confirmed ConcurrencyJobDiscriminator is present in pkg/workflow/concurrency.go at that tag, so this is purely about discoverability, not a missing feature)
- Compiled workflow triggered by
workflow_dispatch, one dispatch per target repository, ~50 concurrent
Summary
concurrency.job-discriminator(added for #24037) correctly fixes the staticgh-aw-conclusion-<workflow-id>group - but only if you know to set it. For aworkflow_dispatchfan-out workflow the default group is essentially always wrong, and sincequeue: maxlanded, getting it wrong no longer produces an error or a cancellation. It produces a silent throughput collapse.I'd like the compiler to say something when it can see the combination is wrong.
What we hit
We dispatch a compiled workflow once per target repository from an orchestrator, up to 50 concurrently. Every job that does real work is per-target: the top-level group and the agent group both key on
inputs.target_repo. Onlyconclusiondoesn't:So all 50 runs queue for one
conclusionslot. The job itself takes 22-36s, but the wait before it starts grows across the sweep:That's a drain rate of roughly 2.4 runs/minute. Our orchestrator waits for the run to report
completed, andcompletedneedsconclusion- so each target held its parallelism slot on a queue shared with all the others, and effective concurrency collapsed toward 2.4 regardless of what we configured. On one run,activation,agent,detectionandsafe_outputswere all green at 14:50:12 whileconclusionwas still pending eleven minutes later.Why this is worth a separate issue from #24037
#24037 reported the cancellation symptom:
Canceling since a higher priority waiting request for gh-aw-conclusion-<id> exists. That's loud. It shows up in the run, and you go and read the docs.With
queue: max, runs queue instead of being cancelled. The symptom changed from "a job got cancelled" to "everything succeeds, just slowly" - which is strictly harder to notice. Nothing fails. There's no warning at compile time. The runs are all green. You only find it if you go and diff job start times across a fan-out and notice the queue wait growing linearly.It took us a while to get there, and our first fix was in the wrong place - we changed our orchestrator to stop waiting on
conclusionbefore realising the knob already existed. The docs are clear once you're reading Fan-Out Concurrency, but nothing points you at that page when you need it.Suggestion
The compiler already knows both halves of this at compile time: the workflow is triggered by
workflow_dispatchwith inputs, and noconcurrency.job-discriminatoris set. That combination means "every dispatch of this workflow serializes on one conclusion slot", which is almost never what the author wants.Options, roughly in order of how much I'd like them:
concurrency.job-discriminator; the generatedconclusiongroup is shared by all dispatches of this workflow. Set a discriminator (e.g.${{ github.run_id }}) to give each dispatch its own slot." Cheap, no behaviour change, and it lands exactly when the author can act on it.${{ github.run_id }}forworkflow_dispatchtriggers. Per-run is the safe default for dispatch; a shared slot across unrelated dispatches doesn't seem to protect anything. Behaviour change, so a warning may be the better first step.workflow_dispatchdocs, not only under concurrency - that's where someone building fan-out is actually reading.Happy to send a PR for (1) if you'd take it.
Environment
v0.86.2(confirmedConcurrencyJobDiscriminatoris present inpkg/workflow/concurrency.goat that tag, so this is purely about discoverability, not a missing feature)workflow_dispatch, one dispatch per target repository, ~50 concurrent