Nightly release missed the 2026-09-17 schedule again

[sigilmakes]

Summary

The nightly release workflow (release.yml, on.schedule) failed to start again on 2026-09-17, despite the cron being moved off the top of the hour in 54798ff6 (fix for #215).

This is the second consecutive dropped nightly:

• 2026-09-16 03:00 UTC (0 3 * * *): no run at all (Nightly release workflow missed the 2026-09-16 schedule #215)
• 2026-09-17 03:17 UTC (17 3 * * *): no run at all

Evidence (observed 2026-09-17 07:34 UTC)

• gh run list --workflow release --event schedule --limit 3 returns nothing.
• gh run list --event schedule --limit 10 (all workflows) shows only one scheduled run ever: ci run 34946581012, created 2026-09-15T08:22:16Z.
• The full recent run list shows the newest release run is the manual recovery 35067038714 (workflow_dispatch, 2026-09-16T07:08:32Z, success). No run was created after 03:17 UTC on 2026-09-17.
• docker buildx imagetools inspect ghcr.io/stfc/goldilocks-workbench:nightly-2026-09-17 → not found.
• gh release list unchanged; latest release is still v0.2.3 (as expected — nightly creates no release).

Open questions

1. Is the schedule actually registered with GitHub? The workflow file exists on the default branch and a push touched it on 2026-09-16 (commit 54798ff6), which should have re-registered the new cron. Two drops at two different minutes makes the "top-of-hour load drop" theory less convincing.
2. Could an org-level or repo-level policy be blocking scheduled workflows entirely? (The only scheduled run that ever fired was ci on Sep 15.)
3. Should the nightly be driven by an external trigger (e.g., a cron hitting the workflow_dispatch.publish_nightly=true input) instead of relying on on.schedule, which has now failed twice in a row?

No fix applied

Per direction from Willow: this issue reports the observation only. No pushes to main, no PRs, and no tag re-cuts were made; waiting for explicit direction on the approach.

Written by an agent on behalf of Willow Sparks.

[sigilmakes]

Root-cause investigation (read-only, no changes made)

What was ruled out

• Workflow file: the same release.yml runs fine on push, tag, and workflow_dispatch events; on: is valid; workflow state is active; repo Actions settings are enabled: true, allowed_actions: all.
• Platform incidents: no GitHub Actions incidents cover 2026-09-16 03:00 UTC or 2026-09-17 03:17 UTC (status page checked).
• Org-wide schedule breakage: other stfc repos deliver scheduled runs reliably in the same days, e.g. stfc/iris-bench fired its daily cron on 09-13, 09-14, 09-15, 09-16, 09-17 (all success).

What the history shows

The nightly cron has existed since Sep 11 and has fired once in six opportunities:

Window	Registered workflow	Result
Sep 12 03:00	ci.yml (0 3 * * *)	no run
Sep 13 03:00	ci.yml	no run
Sep 14 03:00	ci.yml	no run
Sep 15 03:00	ci.yml	no run at 03:00; one run delivered 5 h 22 m late (08:22 UTC), executing pre-restructure HEAD b8b12e61
Sep 16 03:00	release.yml	no run
Sep 17 03:17	release.yml (after 54798ff6)	no run

Timeline from the public events API: the restructure push (3e5820a9, which moved the schedule from ci.yml to release.yml) landed on main at 2026-09-15 09:43 UTC — after the lone 08:22 delivery, whose HEAD confirms it came from the old ci.yml registration.

Interpretation

The failure is repo-specific and upstream of workflow execution: no run object is ever created, so the event never reaches the workflow. Two different workflow files, two registration generations, six opportunities, one delivery (late). Pushes that touch the workflow files did not restore delivery. This matches the known failure class where GitHub's internal schedule registration for a single repository goes stale and silently drops its cron events; the scheduler's internal state cannot be inspected from outside, so this is inference from the controls above, not proof.

Suggested remedies (none require a push to main)

1. Force re-registration: gh workflow disable release && gh workflow enable release (or via the Actions UI). Reversible, instant, costs nothing while no tags are pending. Then verify the next 03:17 UTC fire.
2. Bypass the scheduler: a scheduled relay workflow in a different, healthy repo (or a timer on a machine that is on overnight) that runs gh workflow dispatch -R stfc/goldilocks-core release.yml -f publish_nightly=true daily — the dispatch path is proven to work (recovery run 35067038714).
3. GitHub Support ticket for the org, since scheduler-side state may need GitHub to clear it.

Recommendation: try 1 first, verify against the next fire; if it still drops, fall back to 2 and/or 3.

Written by an agent on behalf of Willow Sparks.

[sigilmakes]

Settings sweep (gh-verified, read-only)

Every repo-level setting that can affect Actions was checked and is normal:

Setting	Value
Repository flags	archived: false, disabled: false, fork: false, template: false
Actions enabled	enabled: true, allowed_actions: all, sha_pinning_required: false
Workflow permissions	default_workflow_permissions: write
All 5 workflows	state: active (none disabled_manually / disabled_inactivity)
Branch protection on main	none (classic protection returns 404)
Ruleset "Protect main/default branch"	rules: deletion, non_fast_forward only — governs branch deletion/force-push, not workflow runs
Environments	none (no protection rules to gate jobs)
Fork PR approvals / access policy	n/a for public repos (API 404/422)

Full run sweep Sep 12–17 (any status, any event): 92 runs, exactly one event=schedule (Sep 15 08:22 UTC). Zero runs in queued/waiting/pending/in_progress right now — scheduled events are not being created and are not stuck in a queue.

One surface gh cannot read: org-level Actions policies (needs org admin; gh api /orgs/stfc/actions/permissions returns 403 with this token). Note however that documented org Actions policies govern enabled repos, allowed actions, fork approvals, and retention — GitHub documents no policy that drops scheduled events while continuing to deliver push/PR/dispatch events.

Conclusion after the settings sweep: configuration-based explanations are exhausted. The delivery failure sits in GitHub's scheduler pipeline for this repository, which is not observable via the API. The discriminating experiments remain: (1) disable/re-enable the release workflow to force re-registration, (2) relay dispatch from a healthy repo or external timer, (3) GitHub Support. None require a push to main.

Written by an agent on behalf of Willow Sparks.

[sigilmakes]

Soooo this seems to have fixed itself and nightly has been working fine the past few days? Odd.
On a separate note dependabot opened a PR which broke/flaked due to a flakiness with pseudodojo rather than anything it did. Wonder if that's related.
I'll leave if this open until I know what happened here for certain