fix(ci): eliminate the false positives, false negatives and warnings across all seven publish workflows - #299
Merged
Merged
Conversation
Adding .gitkeep for PR creation (default mode). This file will be removed when the task is complete. Issue: #298
The rust workflow published links-notation 0.16.0 successfully and then failed itself. `cargo publish` reported "Published links-notation v0.16.0 at registry crates-io" at 11:50:09, and the very next step spent five minutes deciding the crate was not there. The step ran `curl -fsS https://crates.io/api/v1/crates/$NAME/$VERSION`. crates.io answers 403 to curl's default `User-Agent: curl/8.x`, so every one of the twenty attempts failed for a reason that had nothing to do with the release. `>/dev/null 2>&1` then threw the status code away, which is why twenty identical "Not visible yet" lines were the only evidence. The same bare-curl pattern was in eleven places across seven workflows, so rather than patching the one that happened to fail, all of them now go through scripts/ci/registry-probe.sh: - every request carries an identifying User-Agent, as crates.io asks; - the status code of the final attempt is always in the failure message, so 404 (indexing lag) is distinguishable from 403 (broken probe); - CI_VERBOSE=true logs the status of every attempt. Default off. Two related defects in the same job: - the crates.io OIDC step wrote an `::error::` annotation on runs that published fine through CARGO_TOKEN, because trusted publishing is not registered yet. It is now gated on an opt-in repository variable. - the dependency wait polled the JSON API, but `cargo` resolves against the sparse index, and the index answers 200 for any existing crate. wait_for_crate_version matches the version inside the index document. scripts/ci/registry-probe.test.sh reproduces the 403 against a local server, so it needs no network. It fails against the old behaviour. Refs #298
Two more annotation classes on the same push, neither of which described a real problem. release-audit runs on push to main, which is the same event that starts the seven publish workflows. Run 33168552493 finished at 11:48:36, eight seconds after the push; rust was still publishing until 11:55:14. The audit compared the freshly bumped 0.16.0 against registries that could not possibly have it yet and annotated all seven languages as drifted. The registries now hold 0.16.0 for js, python, rust, csharp and go, so every one of those warnings was noise. The audit now waits for this commit's other workflow runs to complete before comparing. On a pull request no publish job runs at all, so being ahead of the registry is the expected state there and is reported as a notice rather than a warning. pypa/gh-action-pypi-publish defaults `attestations` to true, but attestations require trusted publishing, and the step passes PYPI_TOKEN. Every release therefore logged "the attestations input is ignored". Both settings now derive from one opt-in variable, so they cannot contradict each other. Refs #298
… does
`cargo publish` prints
error: crate links-notation-macro@0.1.0 already exists on crates.io index
whenever a version is re-published, and the rust workflow re-publishes the
macro crate on every run where only the main crate was bumped. The
function that calls it already handled the case and returned "skipped",
so the job was correct, but the log carried a red error line for a run
that was doing exactly the right thing.
The js, python, csharp, php and java workflows all check the registry
before pushing. Rust now does too, through the same shared helper.
crates_index_url gained a CRATES_INDEX_BASE override so the new
crate_version_published tests can run against the local stand-in server
instead of the real index. Three of them cover the case that made the
first draft of this helper wrong: the sparse index returns 200 for any
crate that exists, so the version has to be matched inside the document,
and 0.1 must not match 0.16.0.
Refs #298
Collects the raw logs, job and annotation inventories, and the seven pipeline templates under dev/log/issues/298/pulls/299, plus the analysis they support: the timeline of the failing push, the requirements from the issue, a root cause for each defect, the template comparison behind the two upstream reports, and the annotations that remain with the reason each one stays. The .gitignore change is what makes this committable at all. `[Ll]og/`, inherited from the Visual Studio section, matches the `dev/log/` directory, and `*.log` matches the CI logs collected inside it, so every file here had to be force-added. Two negation rules make the development log behave like the deliverable it is. Refs #298
Prepares the release that exercises the fixed publish path end to end. The whole premise of #298 is that the pipeline cannot be trusted to report what it actually did, so the fix is only proven once a version ships through it. js/package-lock.json carries the version too, so it is bumped alongside js/package.json; version-consistency.mjs now reports 0.16.1 for all seven.
konard
changed the title
konard
marked this pull request as ready for review
August 28, 2026 12:51
…hecks This PR's first run failed two checks, and both were correct findings about files it had just added: - links (33172748012): the verbatim copy of CI-CD-BEST-PRACTICES.md keeps the upstream document's repository-relative links, which do not resolve from dev/log. Given a provenance header and absolute hive-mind URLs, the same treatment the copy under dev/log/issues/290/pulls/291/analysis/ already has. All five rewritten URLs verified to return 200. - bom-check (33172747906): 'gh run view --log' prepends a UTF-8 BOM, so the collected publishToCratesIO log carried one. Stripped from byte 1; nothing else in the log changed. Both runs' logs are kept alongside the rest of the evidence, and analysis/ README.md records the two edits so the copies are not mistaken for untouched downloads.
Member
Author
CI verification on
|
| Level | Message |
|---|---|
| notice | 7 × <lang>: declared 0.16.1, latest on <registry> is 0.16.0 |
| notice | CODECOV_TOKEN is not configured, so coverage was not uploaded |
| notice | Summary report available at: ... (lychee's own job-summary link) |
The first row is the release-audit fix demonstrating itself: on main those seven were ::warning:: and fired 8 seconds after the push while the publishes were still running. On a pull request no publish job runs at all, so being ahead of the registry is the expected state, and they are now reported at ::notice::.
The two 0.16.1 / nothing published rows for java and php are accurate — those languages have never released. They are left as-is deliberately, with the follow-up actions in §8 of the analysis; silencing them would trade a false positive for a false negative.
Two findings this PR's own CI produced
The first run failed bom-check and links, and both were correct findings about files this PR had just added — the evidence I collected, not the fixes:
gh run view --logprepends a UTF-8 BOM, so the collectedpublishToCratesIOlog carried one. Stripped from byte 1.- The verbatim copy of
CI-CD-BEST-PRACTICES.mdkeeps the upstream document's repository-relative links, which cannot resolve fromdev/log. Given a provenance header and absolute hive-mind URLs — the same treatment the copy underdev/log/issues/290/pulls/291/analysis/already has. All five rewritten URLs verified to return 200.
Both runs' logs are committed alongside the rest of the evidence, and the two edits are recorded at the top of analysis/README.md so the copies are not mistaken for untouched downloads.
Local gates
actionlint 1.7.7 rc=0
shellcheck -x scripts/ci/*.sh rc=0
zizmor 1.29.0 --min-severity medium No findings to report.
lychee 0.24.2 --offline './**/*.md' 987 unique, 0 errors
registry-probe.test.sh passed: 21, failed: 0
version-consistency.mjs All 7 implementations declare 0.16.1.
Member
Author
🤖 Solution Draft LogThis log file contains the complete execution trace of the AI solution draft process. 💰 Cost: $11.902565📊 Context and tokens usage:Claude Opus 5: (3 sub-sessions)
Total: (4.9K new + 268.3K cache writes + 12.3M cache reads) input tokens, 120.3K output tokens, $11.877740 cost Claude Haiku 4.5:
Total: 12.0K input tokens, 574 output tokens, $0.024825 cost 🤖 Models used:
📎 Log file uploaded as Gist (4276KB)Now working session is ended, feel free to review and add any feedback on the solution draft. |
Member
Author
🎉 Auto-mergedThis pull request has been automatically merged by hive-mind.
Auto-merged by hive-mind with --auto-merge flag |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #298.
What was wrong
Run 33168552506 failed on
mainwhile the release it was verifying had already shipped. That single run contained three distinct defect classes, and the same patterns were spread across the other six language workflows.1. False negative — the release that succeeded but reported failure
cargo publishgenuinely worked; the collected log showsPublished links-notation v0.16.0 at registry crates-ioat 11:50:09. The verification step then ran:crates.io answers HTTP 403 to clients that do not send an identifying
User-Agent— curl's defaultcurl/8.xis rejected. Verified live:So all 20 attempts failed for a reason unrelated to the release, and
>/dev/null 2>&1threw away the status code that would have said so — which is why the log holds 20 identicalNot visible yetlines and nothing else. The repository already knew this rule:scripts/release-audit.mjshas always sent'user-agent': 'links-notation-release-audit'. The knowledge never reached the workflows.The bare-curl / status-discarding pattern existed in 11 call sites across 7 workflows. All of them now go through one shared helper,
scripts/ci/registry-probe.sh, so they cannot drift apart again.2. False positive — an error annotation on an unconfigured optional integration
rust-lang/crates-io-auth-action@v1fails withNo Trusted Publishing config foundand writes an::error::annotation.continue-on-error: truedoes not suppress annotations. The step is now gated behindvars.CRATES_IO_TRUSTED_PUBLISHING == 'true'.3. False positive — release-audit racing the publishes it audits
7 of the 17 annotations on that commit came from the audit finishing 8 seconds after the push (11:48:36) while rust kept publishing until 11:55:14. The audit now waits for this commit's other workflows on
push, and reports at::notice::instead of::warning::on pull requests, where being ahead of the registry is the expected state.4. Contradictory inputs — PyPI attestations
pypa/gh-action-pypi-publishdefaultsattestationsto true, but attestations require trusted publishing, and apasswordwas being passed. Both inputs are now driven by the same variable.5. Log noise — rust re-publishing an already-released crate
Every other language workflow checked the registry before pushing; rust did not, so a re-run printed a red
already exists on crates.io index. Added.What is in this PR
scripts/ci/registry-probe.sh— shared registry probing:probe_registry,fetch_registry,crates_index_url,crate_version_published,wait_for_crate_version,wait_for_registry_match,wait_for_registry. Every request sends an identifying User-Agent, and every failure message ends with(last status: HTTP <code>)so the next occurrence is diagnosable from the log alone.scripts/ci/registry-probe.test.sh— 21 assertions against a local server that reproduces the crates.io 403 behaviour. No network access, so it cannot go flaky. It fails on the pre-fix code and passes on the fix. One assertion guards a mistake I made while writing this: the sparse index returns 200 for any crate that exists, so a version must be matched inside the body —0.1must not match0.16.0..github/workflows/workflows.yml— newci-scriptsjob running shellcheck and the probe test.actionlintonly reaches shell inlined in arun:block, so a standalone helper needs its own gate.CI_VERBOSEconvention now actually reaches the polls — per-attempt status codes and index contents. Both the silent and verbose paths are covered by tests..gitignore—dev/log/was matched by the inherited Visual Studio rule[Ll]og/, so every file under it had to be force-added. Un-ignored properly.Annotations deliberately left in place
php not registered on Packagist, java Maven Central secrets missing, the go CODECOV_TOKEN notice, and two advisory notices are accurate reports of unconfigured optional integrations. Silencing them would trade a false positive for a false negative. Each is documented with a follow-up action in §5 and §8 of the analysis.
Template comparison and upstream reports
All files were compared against the seven
link-foundation/*-ai-driven-development-pipeline-templaterepos. None uses bare curl — each has a language-native wait script — so the specific #298 trigger does not exist upstream. But the same defect class does, and it is reported with a reproduction, a workaround, and a code-level fix in each:wait-for-crate.rscollapses 403/429/5xx into "not published": wait-for-crate.rs reports a successful release as failed when crates.io returns 403, 429 or 5xx rust-ai-driven-development-pipeline-template#143wait-for-npm.mjshas a barecatch { return false }: wait-for-npm.mjs reports a successful release as failed when the registry is unreachable js-ai-driven-development-pipeline-template#149The csharp template's
wait-for-nuget.mjs, which returns{available, status, url}, is the in-family reference both should follow.Evidence and analysis
Everything collected is committed under
dev/log/issues/298/pulls/299/— 42 files: the raw failing-run logs, the jobs and annotations API dumps, all seven upstream templates, the referenced CI-CD-BEST-PRACTICES.md, and a 336-line analysis with the timeline, requirements R1–R11, per-defect root causes, and the survey of existing components (including whycargo-release,release-plz, andnick-fields/retrywere rejected).Verification
Running the audit live independently confirms the diagnosis: js, python, rust, csharp and go were all in sync at 0.16.0 — the release the workflow called a failure had shipped.