EVALS_PUBLISH=false (#66) is designed so a run still executes and still uploads artifacts while results/ is left alone — the evidence is kept, and publishing happens later. The "later" half does not work.
What happened
The 1 September matrix ran held. Clearing the variable and re-running the publish-results job of run 33484972784 did the right thing up to the last step:
Exported 114 result(s) to apps/web/src/data/eval-results.json (101 pass, 13 fail)
Published snapshot
error: failed to push some refs to 'https://github.com/hookdeck/evals'
A re-run checks out the SHA the original run used. main had moved six merges since 1 September, so pushing HEAD:main was a non-fast-forward, and the rebase-and-retry loop did not recover it — plausibly because actions/checkout fetches shallow, so git rebase origin/main has no common history to work with.
The matrix jobs were not re-executed; their timestamps stay at 1 September. So nothing was spent, and the artifacts remain valid — the data was published by hand from them in #73.
Why it matters
The hold exists so an expensive run can be kept and published once someone has read it. If publishing a held run needs a person to download artifacts, stage them into .eval-runs/ and run two scripts locally, then the hold quietly converts every held run into manual work — and hand-publishing is exactly the path that put a local sweep on the public page in the first place.
Options
- Fetch enough history to rebase —
fetch-depth: 0 on the publish job's checkout, so the retry loop can actually rebase. Smallest change.
- Re-target the checkout — check out
origin/$branch at publish time rather than the run's SHA. The results being committed do not depend on the tree, only on the artifacts.
- A
publish-snapshot workflow that takes a run id and publishes from its artifacts, so publishing a held run is a dispatch rather than a job re-run. Also covers publishing a run someone deliberately held for a week.
(1) makes the current path work; (3) is the one that matches how the hold is actually used.
EVALS_PUBLISH=false(#66) is designed so a run still executes and still uploads artifacts whileresults/is left alone — the evidence is kept, and publishing happens later. The "later" half does not work.What happened
The 1 September matrix ran held. Clearing the variable and re-running the
publish-resultsjob of run 33484972784 did the right thing up to the last step:A re-run checks out the SHA the original run used.
mainhad moved six merges since 1 September, so pushingHEAD:mainwas a non-fast-forward, and the rebase-and-retry loop did not recover it — plausibly becauseactions/checkoutfetches shallow, sogit rebase origin/mainhas no common history to work with.The matrix jobs were not re-executed; their timestamps stay at 1 September. So nothing was spent, and the artifacts remain valid — the data was published by hand from them in #73.
Why it matters
The hold exists so an expensive run can be kept and published once someone has read it. If publishing a held run needs a person to download artifacts, stage them into
.eval-runs/and run two scripts locally, then the hold quietly converts every held run into manual work — and hand-publishing is exactly the path that put a local sweep on the public page in the first place.Options
fetch-depth: 0on the publish job's checkout, so the retry loop can actually rebase. Smallest change.origin/$branchat publish time rather than the run's SHA. The results being committed do not depend on the tree, only on the artifacts.publish-snapshotworkflow that takes a run id and publishes from its artifacts, so publishing a held run is a dispatch rather than a job re-run. Also covers publishing a run someone deliberately held for a week.(1) makes the current path work; (3) is the one that matches how the hold is actually used.