fix(nitro-worker): retry registration instead of crashlooping - #973
Merged
Conversation
`--auto-register` treated on-chain registration as a startup gate: any failure propagated
out of `run()` and exited the process. Every way registration can fail — PCR set not yet
approved, registration key unfunded, L1 unreachable, certificate chain not yet verifiable —
is a condition an operator resolves while the worker is running, so exiting turns all of
them into CrashLoopBackOff.
Worse, it destroys the path needed to fix them: `just proof-get-attestation`,
`proof-certmanager-prewarm` and `proof-register-key` all exec into the container and
require it to be in `Running` state. The failure blocked its own remedy — observed on
alphanet as 34 restarts with no way to inspect the enclave.
Registration now retries on a `backon` exponential backoff (5s → 5m, jittered, unbounded),
raced against ctrl-c so a pod being rolled does not wait out a full interval. The worker
stays up, keeps serving metrics, and simply does not lease proof jobs until the key is
registered — proofs signed by an unregistered key would not verify, so an unregistered
worker must be inert, not absent.
Jitter is load-bearing: replicas share one funding key, so un-jittered retries collide on
the same nonce every interval and fail as a group.
Adds two metrics so this is visible rather than inferred:
- `enclave_key.registered` gauge, published as 0 before the first attempt so "never
registered" is a zero rather than an absent series a threshold monitor would ignore.
- `enclave_key.registration_attempts{outcome}` counter.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
0xOsiris
marked this pull request as ready for review
August 4, 2026 18:48
0xOsiris
requested review from
0xForerunner,
Dzejkop,
Kemperino,
alessandromazza98,
cichaczem,
karankurbur,
kilianglas,
murph and
piohei
as code owners
August 4, 2026 18:48
Contributor
Benchmark ResultsBase and PR measured on the same runner in the same workflow run.
Raw
|
piohei
approved these changes
Aug 5, 2026
piohei
approved these changes
Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Medium Risk
Changes nitro worker startup and L1 registration behavior when auto-register is enabled; misconfiguration could delay job leasing but avoids taking work with an unregistered enclave key.
Overview
Nitro worker
--auto-registerno longer exits on the first registration failure. Registration runs through a newregister_with_retrypath using unbounded, jittered exponential backoff (5s–300s) viabackon, so transient or operator-fixable issues (unfunded key, PCR not approved, L1 down) keep the pod Running instead ofCrashLoopBackOff.The worker still does not lease proof jobs until on-chain registration succeeds; shutdown during the retry loop (Ctrl-C) exits cleanly without starting the worker. Shared metrics add
enclave_key.registered(initialized to0before the first attempt) andenclave_key.registration_attemptsby outcome (registered,already_registered,failed). Unit tests lock in unbounded retry and delay ceiling behavior for the backoff builder.Reviewed by Cursor Bugbot for commit 0d775ab. Configure here.