Skip to content

Warn when a workflow_dispatch workflow has no job-discriminator: the shared conclusion group silently caps fan-out #55813

Description

@radiantspace

Summary

concurrency.job-discriminator (added for #24037) correctly fixes the static gh-aw-conclusion-<workflow-id> group - but only if you know to set it. For a workflow_dispatch fan-out workflow the default group is essentially always wrong, and since queue: max landed, getting it wrong no longer produces an error or a cancellation. It produces a silent throughput collapse.

I'd like the compiler to say something when it can see the combination is wrong.

What we hit

We dispatch a compiled workflow once per target repository from an orchestrator, up to 50 concurrently. Every job that does real work is per-target: the top-level group and the agent group both key on inputs.target_repo. Only conclusion doesn't:

conclusion:
  concurrency:
    group: "gh-aw-conclusion-campaign-prompt-eval"
    cancel-in-progress: false
    queue: max

So all 50 runs queue for one conclusion slot. The job itself takes 22-36s, but the wait before it starts grows across the sweep:

conclusion # queued before start
1 7s
2 55s
3 1m12s
4 1m43s
5 4m42s
6 5m07s
7 6m16s
8 8m28s

That's a drain rate of roughly 2.4 runs/minute. Our orchestrator waits for the run to report completed, and completed needs conclusion - so each target held its parallelism slot on a queue shared with all the others, and effective concurrency collapsed toward 2.4 regardless of what we configured. On one run, activation, agent, detection and safe_outputs were all green at 14:50:12 while conclusion was still pending eleven minutes later.

Why this is worth a separate issue from #24037

#24037 reported the cancellation symptom: Canceling since a higher priority waiting request for gh-aw-conclusion-<id> exists. That's loud. It shows up in the run, and you go and read the docs.

With queue: max, runs queue instead of being cancelled. The symptom changed from "a job got cancelled" to "everything succeeds, just slowly" - which is strictly harder to notice. Nothing fails. There's no warning at compile time. The runs are all green. You only find it if you go and diff job start times across a fan-out and notice the queue wait growing linearly.

It took us a while to get there, and our first fix was in the wrong place - we changed our orchestrator to stop waiting on conclusion before realising the knob already existed. The docs are clear once you're reading Fan-Out Concurrency, but nothing points you at that page when you need it.

Suggestion

The compiler already knows both halves of this at compile time: the workflow is triggered by workflow_dispatch with inputs, and no concurrency.job-discriminator is set. That combination means "every dispatch of this workflow serializes on one conclusion slot", which is almost never what the author wants.

Options, roughly in order of how much I'd like them:

  1. Warn at compile time. Something like: "workflow_dispatch workflow has no concurrency.job-discriminator; the generated conclusion group is shared by all dispatches of this workflow. Set a discriminator (e.g. ${{ github.run_id }}) to give each dispatch its own slot." Cheap, no behaviour change, and it lands exactly when the author can act on it.
  2. Default the discriminator to ${{ github.run_id }} for workflow_dispatch triggers. Per-run is the safe default for dispatch; a shared slot across unrelated dispatches doesn't seem to protect anything. Behaviour change, so a warning may be the better first step.
  3. Mention it in the workflow_dispatch docs, not only under concurrency - that's where someone building fan-out is actually reading.

Happy to send a PR for (1) if you'd take it.

Environment

  • gh-aw v0.86.2 (confirmed ConcurrencyJobDiscriminator is present in pkg/workflow/concurrency.go at that tag, so this is purely about discoverability, not a missing feature)
  • Compiled workflow triggered by workflow_dispatch, one dispatch per target repository, ~50 concurrent

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions