chore: aggregate nitro fixes for alphanet deploy - #976
Closed
0xOsiris wants to merge 17 commits into
Closed
Conversation
`registerKey` reaches `NitroValidator.validateAttestationWithHints`, which re-walks the attestation's certificate bundle via `verifyCachedCertBundle` passing **empty** hint streams. That only succeeds on certificates already in the `CertManager` cache; an uncached cert falls through to signature verification against an empty hint stream and reverts with "inverse hint underflow" — regardless of how good `attestationSigHints` is. AWS rotates the enclave leaf certificate roughly every three hours, so the cache goes cold on its own and the pre-warm had to be re-run by hand inside that window. In practice it never was: the alphanet registry has emitted zero `SignerRegistered` events, and the worker crashlooped on this revert with a leaf cached ~19h earlier and one intermediate CA that had expired 11 minutes before the pod started. The worker now pre-warms its own chain. `prewarm::build_prewarm_plan` turns an attestation into the ordered `verifyCACertWithHints` / `verifyClientCertWithHints` calls needed, and `register_enclave_key` submits the uncached ones before `registerKey`. The CertManager address is discovered via `registry.verifier().certManager()`, so there is no second address to configure and drift. Details: - The pinned AWS root is written into the cache by the CertManager constructor, so it never needs a transaction — it only seeds the parent hash for the next cert in the chain. - Cache keys mirror `CertManager._certCacheKey`: keccak over the TBSCertificate element for every cert, and the pinned constant for the root. - A cert that is cached but expired cannot be re-verified (`_verifyCert` short-circuits on the cache and reverts "cert expired"), so that case fails with an explicit message pointing at the real fix rather than an opaque revert later. - A peer replica caching the same cert between the check and the submit is treated as success, not an error. Verified: 47 nitro tests pass (7 new); `cargo check` and `clippy -D warnings` clean for x86_64-unknown-linux-gnu with `--features enclave`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`--auto-register` treated on-chain registration as a startup gate: any failure propagated
out of `run()` and exited the process. Every way registration can fail — PCR set not yet
approved, registration key unfunded, L1 unreachable, certificate chain not yet verifiable —
is a condition an operator resolves while the worker is running, so exiting turns all of
them into CrashLoopBackOff.
Worse, it destroys the path needed to fix them: `just proof-get-attestation`,
`proof-certmanager-prewarm` and `proof-register-key` all exec into the container and
require it to be in `Running` state. The failure blocked its own remedy — observed on
alphanet as 34 restarts with no way to inspect the enclave.
Registration now retries on a `backon` exponential backoff (5s → 5m, jittered, unbounded),
raced against ctrl-c so a pod being rolled does not wait out a full interval. The worker
stays up, keeps serving metrics, and simply does not lease proof jobs until the key is
registered — proofs signed by an unregistered key would not verify, so an unregistered
worker must be inert, not absent.
Jitter is load-bearing: replicas share one funding key, so un-jittered retries collide on
the same nonce every interval and fail as a group.
Adds two metrics so this is visible rather than inferred:
- `enclave_key.registered` gauge, published as 0 before the first attempt so "never
registered" is a zero rather than an absent series a threshold monitor would ignore.
- `enclave_key.registration_attempts{outcome}` counter.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The enclave image is built from a mutable tag, so every EIF rebuild changes PCR0 (enclave image) and PCR2 (application) while PCR1 (kernel) stays put. Nothing re-approves the new measurements, so the running enclave silently drops off the on-chain allowlist and every registerKey / TEE proof reverts — with no signal until someone reads a revert reason. This is not hypothetical: on alphanet the running enclave measures PCR0=4f1adb…/PCR2=e89ada… while the verifier only approves b94235…/f0e0ed…, and `isPCRSetApproved` returns false. `just proof-verify-pcrs` measures the enclave that is actually running and asserts its PCR set is approved on-chain, printing both sets side by side and the remediation command when it is not. It deliberately re-measures rather than trusting PCR0/1/2 from the shell — trusting the shell would reproduce exactly the drift this is meant to catch. Wired into `proof-setup` as Step 3c so setup verifies rather than assumes (skipped under dry_run, which approves nothing). Also documents the mutable-tag hazard where PROOF_NITRO_IMAGE is defined — note that variable is currently declarative only, so pinning a digest there does not by itself prevent drift. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rldcoin/world-chain into fix/nitro-worker-self-prewarm
…rldcoin/world-chain into chore/pin-nitro-enclave-image
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…l' into nitro/alphanet-bundle
…nto nitro/alphanet-bundle
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
0xOsiris
requested review from
0xForerunner,
Dzejkop,
Kemperino,
alessandromazza98,
cichaczem,
karankurbur,
kilianglas,
murph and
piohei
as code owners
August 4, 2026 21:16
Contributor
Benchmark ResultsBase and PR measured on the same runner in the same workflow run.
Raw
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Deploy vehicle:
main+ #972 + #973 + #974, merged with no conflicts. This is the branchsha-6ed502bwas built from and pinned in crypto-apps#751.This overlaps the three PRs below — review those, not this one. Opened for traceability of what shipped to alphanet. Close it if you'd rather land the parts individually.
registerKeybackonbackoff instead of exiting the processjust proof-verify-pcrs— asserts the running enclave's PCR set is approvedAlso carries one commit of its own: the approved PCR set recorded into
pkg/contracts/deployments/alphanet-nitro.jsonbyjust proof-approve-pcrs.Why these three together
They're the three stacked blockers behind alphanet producing zero proofs — 33 games, all with
proofBitmap = 0. Fixing any one alone leaves it broken:inverse hint underflow. fix(nitro): pre-warm the CertManager cert chain before registerKey #972.Verification
cargo check+clippy -D warningsclean forx86_64-unknown-linux-gnu --features enclave🤖 Generated with Claude Code
Note
Medium Risk
Changes L1 registration behavior (extra cert-verify txs and long-lived retry loops) and proof deploy gates; mistakes could delay job leasing or block setup until PCRs are approved, but scope is Nitro worker/ops tooling rather than core chain consensus.
Overview
Unblocks alphanet Nitro proof workers by fixing three operational failure modes that left games with no TEE proofs.
CertManager self-prewarm before
registerKey: Adds aprewarmmodule that builds an ordered verify plan from the enclave attestation’s AWS cert chain (cache keys, P-384 hints).register_enclave_keydiscoversCertManagervia the registry’s verifier, submits missingverifyCACertWithHints/verifyClientCertWithHintstxs, and handles peer races and expired cached certs—so leaf rotation (~3h) no longer requires a manualproof-certmanager-prewarmfor registration.Resilient
--auto-register: The nitro-worker retries registration with jittered exponential backoff (unbounded, up to 5 min) instead of exiting on failure; it stays up without leasing jobs until registration succeeds or Ctrl-C. New metrics exposeenclave_key.registered(initialized to 0) and registration attempt outcomes.PCR drift detection in deploy:
just proof-verify-pcrsmeasures the running enclave viaproof-get-pcrsand assertsisPCRSetApprovedon L1;proof-setupruns it as phase 3c after approve. Alphanet deployment JSON records an additional approved PCR set from the latest approve step.Reviewed by Cursor Bugbot for commit 54d28a9. Configure here.