Skip to content

Fall back across registries when pulling the musl build image - #40

Merged
cb1kenobi merged 2 commits into
mainfrom
chris/musl-image-mirrors
Oct 27, 2026
Merged

cb1kenobi merged 2 commits into
mainfrom
chris/musl-image-mirrors

Conversation

@cb1kenobi

Copy link
Copy Markdown
Member

Why

The scheduled 11.12.0 build (run 40440107271) failed on the linux-x64-musl leg because ECR Public refused to serve the alpine image:

Error response from daemon: toomanyrequests: Data limit exceeded

When you pull without logging in, ECR Public limits how much data each IP can download. GitHub-hosted runners share IPs with other customers, so that runner's quota was already spent before our job ran. The arm64 musl leg in the same run pulled the image fine. Retrying couldn't help because the quota resets monthly, not within minutes.

We moved from Docker Hub to ECR in 8177629 because Docker Hub's login-token service was timing out. Each registry has now failed on its own, so the fix is to stop depending on any single one.

What changed

  • Registry fallback. The pull step tries mirror.gcr.io, then ECR Public, then Docker Hub, all at the same pinned digest. It retries the full list 3 times with backoff. I confirmed that all three registries serve the same digest, so whichever one answers, we build on identical bytes.
  • Local name. The pulled image is tagged locally as rocksdb-musl-build. The build and verify steps use that name, so they don't depend on which registry served the image.
  • Bump to alpine 3.24.3 (separate commit). The pin was 3.24.2. This makes little difference in practice, because apk add installs the latest 3.24 packages either way.

Testing

actionlint reports nothing in the changed lines. The workflow has not been re-run yet.

🤖 Generated with Claude Code

cb1kenobi and others added 2 commits October 27, 2026 10:03
ECR Public returned "toomanyrequests: Data limit exceeded" to the x64 musl
runner, an anonymous per-IP quota shared with every other tenant on that runner IP.
Pull the pinned digest from mirror.gcr.io, ECR Public, then Docker Hub, and tag it
locally so later steps do not depend on which registry served it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Note

Gemini is unable to generate a review for this pull request due to the file types involved not being currently supported.

@cb1kenobi
cb1kenobi merged commit 2dcb2c8 into main Oct 27, 2026
7 checks passed
@cb1kenobi
cb1kenobi deleted the chris/musl-image-mirrors branch October 27, 2026 15:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant