feat(blaze): restore sandbox checkpoints - #2475
Draft
WeissonHan wants to merge 25 commits into
Draft
Conversation
This enables explicit destroy and restart cleanup to distinguish a backend that never started from one whose process may still exist. Each spawner creates and syncs an empty PID handoff before Starting ownership is published. During spawn, the child writes and syncs its PID before exec while an inherited file lock prevents cleanup from treating an active handoff as absent. Missing metadata remains an unknown state for compatibility. An unlocked empty handoff proves that no process crossed the spawn boundary. Bubblewrap and Firecracker use the same protocol. Fixes: c33b495 ("feat(blaze): own sandbox runtime resources") Signed-off-by: Jing Zhang <renyu.zj@linux.alibaba.com> Signed-off-by: Weisson <Weisson@linux.alibaba.com>
This enables POST /v1/instances/{id}/reset to reject requests until a
backend can reset both runtime and storage state.
The endpoint now returns 501 without changing lifecycle state, runtime
ownership, or pool membership. Warm-claim tests use a test-only pool
fixture so candidate validation does not rely on a production reset.
A later runtime-reset capability can replace this guard after it resets
both backend and storage state.
Fixes: 1f0cfac
Signed-off-by: Jing Zhang <renyu.zj@linux.alibaba.com>
Signed-off-by: Weisson <Weisson@linux.alibaba.com>
This prevents POST /v1/instances/{id}/checkpoint from reporting a
checkpoint before backend and storage capture are wired.
The handler validates instance existence under the operation lock and
returns HTTP 501 without changing lifecycle state, rewriting state.json,
releasing runtime ownership, or generating a checkpoint ID. The focused
test proves the in-memory and persisted state remain unchanged.
The checkpoint and restore feature will replace this guard after it can
pause the backend, capture runtime and storage data, and compensate
partial failures.
Fixes: 1f0cfac
Signed-off-by: Jing Zhang <renyu.zj@linux.alibaba.com>
Signed-off-by: Weisson <Weisson@linux.alibaba.com>
This enables daemon startup to stop when an owned state directory cannot be loaded or its record names a different sandbox. Persisted runtime ownership is never hidden under an invalid or misleading key. UUID-named directories must contain readable state whose embedded ID matches the directory name, and ServerState::build propagates any scan failure. A failed first state publication removes only the empty directory it created; a directory containing another owned artifact remains in place and fails closed. One damaged owned record now prevents API startup until it is repaired. This avoids serving an incomplete inventory while also preventing an ordinary first-write failure from leaving an empty directory that blocks every later startup. Fixes: 1f0cfac Signed-off-by: Jing Zhang <renyu.zj@linux.alibaba.com> Signed-off-by: Weisson <Weisson@linux.alibaba.com>
This enables a long-running daemon to serialize per-instance mutations without retaining one lock entry for every UUID it has seen. The registry stores weak references and prunes entries with no holders or waiters while holding the registry mutex. Concurrent callers for a live UUID still share one async lock. Completed instances therefore leave no permanent strong reference in the registry. Fixes: c33b495 Signed-off-by: Jing Zhang <renyu.zj@linux.alibaba.com> Signed-off-by: Weisson <Weisson@linux.alibaba.com>
This keeps accepted HTTP connections under daemon ownership so shutdown no longer returns while request handlers are still using runtime resources. The accept loop tracks and reaps connection tasks. On termination it closes listeners, asks HTTP/1 connections to finish, waits for a bounded drain window, and aborts and joins any remaining tasks. This enables shutdown to establish a deterministic boundary before process exit while still preventing an idle or stuck client from blocking the daemon indefinitely. Signed-off-by: Jing Zhang <renyu.zj@linux.alibaba.com> Signed-off-by: Weisson <Weisson@linux.alibaba.com>
The daemon previously stopped after connection handling ended while tracked backend processes and provider slots could remain owned by the exiting process. After connection tasks drain, shutdown starts the existing destroy path for every non-terminal sandbox and retained backend owner. Cleanup attempts run concurrently under one shared work deadline, so a stalled owner cannot multiply stop latency or prevent independent owners from being released. Both shutdown stages cancel and join unfinished tasks before returning. The service manager provides the final stop deadline with cancellation headroom, so no cleanup task continues to access daemon state after an in-process stage finishes. This enables SIGTERM and SIGINT shutdown to release tracked runtime resources while keeping incomplete ownership recoverable and reporting observed cleanup failures. Signed-off-by: Jing Zhang <renyu.zj@linux.alibaba.com> Signed-off-by: Weisson <Weisson@linux.alibaba.com>
All daemon routes previously buffered request bodies to completion. Add a validated, configurable 1 MiB default and a shared collector that rejects oversized declared lengths before polling and stops streamed bodies at the first frame above the limit. This enables every existing daemon endpoint to reject oversized input before buffering the complete body while preserving requests at or below the configured limit. Body read failures map to HTTP 400, and size violations map to HTTP 413. Use one global boundary so new routes inherit it automatically. Per-route payload limits remain outside this change. Signed-off-by: Jing Zhang <renyu.zj@linux.alibaba.com> Signed-off-by: Weisson <Weisson@linux.alibaba.com>
This enables blazed to reject a competing daemon before runtime initialization or Unix socket replacement. Daemon startup now holds an exclusive lock derived from the configured API socket for the entire run. The lock file stays on disk so processes always contend on one inode. It is opened without following symbolic links and must be a single-link regular file owned by the effective user with mode 0600. Unexpected lock or endpoint state aborts startup without deleting the configured socket. An existing socket receives a bounded ownership probe. A reachable endpoint is preserved, including one created by an older daemon without the lock; only an endpoint that reports no listener is treated as stale and replaced. The guarantee is scoped to one configured API socket. Independent daemons using different sockets do not contend on this lock. Signed-off-by: Jing Zhang <renyu.zj@linux.alibaba.com> Signed-off-by: Weisson <Weisson@linux.alibaba.com>
Prepare a new lifecycle record in a non-owner staging directory and sync its file and directory before publishing the UUID owner directory. The previous sequence exposed the final UUID directory before state.json existed and did not sync the state root. A crash could therefore leave an owner that startup could neither load nor classify. Rename the complete staging directory into place and sync the state root. Existing owner updates keep their atomic file replacement and now sync both the file and owner directory. This lets startup observe either no owner or a complete lifecycle record. Signed-off-by: Jing Zhang <renyu.zj@linux.alibaba.com> Signed-off-by: Weisson <Weisson@linux.alibaba.com>
Reconstruct provider-owned slots only when the slot root and every required artifact are direct filesystem objects. The previous metadata checks followed links, so a restored sandbox could silently read or write files outside the provider-owned slot after local state changed. Use link-aware metadata and classify linked roots or artifacts as incomplete. This keeps restart and dirty flush bound to the provider's own directory tree. Signed-off-by: Jing Zhang <renyu.zj@linux.alibaba.com> Signed-off-by: Weisson <Weisson@linux.alibaba.com>
Sandbox metadata previously recorded only lifecycle state. A crash between provider allocation, backend start, and final publication could not distinguish completed work from an operation that might still own resources. Add a durable create or destroy journal and the RecoveryRequired state. State publication now syncs the temporary file, atomically renames it, and syncs the owner directory. A failed first publication removes only the empty directory it created, and a new operation cannot overwrite an unfinished journal. This enables later lifecycle management to discover interrupted work after a restart without mistaking it for a clean sandbox. Older records remain readable through conservative ownership defaults. This commit defines and persists the model only; it does not start or release runtime resources. Signed-off-by: Jing Zhang <renyu.zj@linux.alibaba.com> Signed-off-by: Weisson <Weisson@linux.alibaba.com>
Instance handlers previously coordinated metadata, provider slots, backend processes, and warm claims themselves. Startup and shutdown had no single owner that could enumerate incomplete work and retry every retained resource. Add SandboxManager as the owner of create, warm activation, destroy, startup reconciliation, and graceful shutdown cleanup. Every mutation uses one per-sandbox operation lock, persists intent before changing resources, and retains incomplete ownership as RecoveryRequired. This enables Blaze to create, list, inspect, and destroy sandboxes while keeping partial failures recoverable. Startup reconciliation bounds each record and continues after a failure. Graceful shutdown starts every owned sandbox concurrently under one shared deadline, cancels and joins tasks that overrun it, and reports all remaining owners. Reserved checkpoint and reset actions use the same quiescent-state gate. They still return HTTP 501 without changing runtime or persisted state until their backend and storage operations are implemented. This commit does not add sandbox-named routes, guest commands, VM networking, checkpoint capture, hibernation, or background retries. Signed-off-by: Jing Zhang <renyu.zj@linux.alibaba.com> Signed-off-by: Weisson <Weisson@linux.alibaba.com>
Lifecycle management was available only through the instance route family, and destroy required the older POST action form. Callers could not treat a sandbox as a conventional collection and item resource. Add sandbox collection and item aliases plus DELETE forms for sandbox and instance resources. Every route delegates to the same managed create, list, inspect, or destroy handler. This enables callers to create, enumerate, inspect, and delete sandboxes through `/v1/sandboxes` without introducing a second lifecycle implementation. Existing instance routes, checkpoint and reset actions, and response formats remain available. Signed-off-by: Jing Zhang <renyu.zj@linux.alibaba.com> Signed-off-by: Weisson <Weisson@linux.alibaba.com>
The API reference listed only instance-named routes and did not explain what happens when create, destroy, startup recovery, or daemon shutdown stops partway through resource cleanup. Document sandbox and compatible instance routes, the operation journal, RecoveryRequired state, per-record startup reconciliation, and ordered graceful shutdown cleanup. The English and Chinese examples now use the sandbox collection while retaining all compatible route forms. This enables operators and reviewers to distinguish implemented lifecycle behavior from reserved interfaces. Checkpoint and reset are documented as validation-only routes that return `501` without changing runtime or persisted state. The documented limits remain explicit: interrupted create is cleaned up rather than resumed, existing backend processes are not adopted after restart, and failed recovery has no background retry loop. Signed-off-by: Jing Zhang <renyu.zj@linux.alibaba.com> Signed-off-by: Weisson <Weisson@linux.alibaba.com>
Managed sandboxes had no shared request and response format for communicating with a compatible guest agent. Define ping, exec, read, and write operations with request IDs, command inputs, file data, exit status, and standard-base64 outputs. Response IDs and outcome flags are required so a missing field cannot be mistaken for a confirmed rejection. This gives later transport and API commits one bounded wire contract to validate. It does not open a guest connection or expose a daemon route. Signed-off-by: Jing Zhang <renyu.zj@linux.alibaba.com> Signed-off-by: Weisson <Weisson@linux.alibaba.com>
Managed sandboxes had no production path from the daemon API to a compatible guest endpoint, so callers could not execute commands or transfer files. Add a JSON-line client for the Firecracker vsock proxy with bounded inputs, deadlines, response frames, decoded outputs, and readiness polling. Exec and write failures become outcome-unknown only after delivery begins; pre-delivery failures and reads remain safe to retry. Let backend owners report an optional guest endpoint. SandboxManager holds the existing per-sandbox operation lock while it validates Running state and performs guest work. Cold create waits for readiness only when the selected owner reports an endpoint, and cancelled readiness runs the normal create compensation path. Expose exec, read, and write actions through both sandbox and compatible instance routes. Handlers enforce command, HTTP body, base64, decoded file, and response limits before delegating to the manager, and return stable error codes for known failures and unknown outcomes. This enables API callers to use bounded guest operations without bypassing runtime ownership. Guestless backends keep their create behavior and reject guest work. Warm activation does not repeat readiness, and the default 1 MiB HTTP limit must be raised to use the full 16 MiB decoded file limit. Signed-off-by: Jing Zhang <renyu.zj@linux.alibaba.com> Signed-off-by: Weisson <Weisson@linux.alibaba.com>
The daemon documentation did not explain how callers use the new guest operations or which runtime and size conditions apply. Document the exec, read, and write routes in English and Chinese. Explain cold readiness, lifecycle serialization, warm-pool behavior, retry boundaries, stable error codes, per-request buffering, and the relationship between the default 1 MiB HTTP limit and the 16 MiB decoded file limit. Update Firecracker policy comments to state when guest transport is enabled, and make the quick-start example use a policy that does not require a guest agent. This gives operators enough information to configure and call the feature without assuming unsupported streaming, session reuse, or automatic retries. It does not change runtime behavior. Signed-off-by: Jing Zhang <renyu.zj@linux.alibaba.com> Signed-off-by: Weisson <Weisson@linux.alibaba.com>
The daemon had no stable record for checkpoint identity, parentage, runtime provenance, or artifact integrity. Backend snapshot requests also had no shared full-snapshot vocabulary. Add canonical checkpoint IDs, manifests, API views, artifact-set validation, and full snapshot request/result DTOs. This enables capture code in this PR to reject malformed catalog data before publication or listing; it does not expose a runtime operation by itself. Signed-off-by: Jing Zhang <renyu.zj@linux.alibaba.com> Signed-off-by: Weisson <Weisson@linux.alibaba.com>
Lifecycle journals previously recorded only an operation kind and start time. After an interrupted capture, persisted state could not distinguish an unpublished stage from a committed checkpoint or a changed HEAD. Record checkpoint identity, monotonic durable phases, and the last completed capture, and allow a checkpointed sandbox to return to running. This exposes the latest boundary confirmed in persisted state while catalog listing reports what is actually visible; it does not add a provider or API entry point by itself. Signed-off-by: Jing Zhang <renyu.zj@linux.alibaba.com> Signed-off-by: Weisson <Weisson@linux.alibaba.com>
Checkpoint requests previously stopped after lifecycle validation with 501. The daemon had no provider contract, durable catalog, or list operation for captured state. Add opt-in backend and storage capture contracts, self-contained file-provider copies, a mock backend implementation, verified catalog publication with HEAD tracking, and serialized capture/list API paths. Compensation resumes the runtime and removes unpublished work, while uncertain publication or resume outcomes retain ownership for recovery. This enables callers to capture and list independent checkpoints on supporting backends. Unsupported backends or storage providers fail before journal or catalog mutation. The contracts, first implementations, catalog coordinator, and API stay together because they form one production call path and none is independently usable here. Signed-off-by: Jing Zhang <renyu.zj@linux.alibaba.com> Signed-off-by: Weisson <Weisson@linux.alibaba.com>
This enables Firecracker-backed sandboxes to use the existing checkpoint API by pausing the VM, writing full VM and memory snapshots through the Firecracker UDS API, and resuming the owned runtime. The spawner resolves the configured binary version on every start and freezes it into the runtime owner, so concurrent probes and binary replacement cannot reuse stale global provenance. Firecracker records require that version before capture begins, and the UDS client bounds response size and request duration. Only full capture is enabled; restore and incremental snapshots remain unavailable. Signed-off-by: Jing Zhang <renyu.zj@linux.alibaba.com> Signed-off-by: Weisson <Weisson@linux.alibaba.com>
This provides blaze-core with a durable restore operation and monotonic ownership phases for replacing a running sandbox from a selected checkpoint. A later sandbox-manager commit consumes these records; this commit does not expose restore through a production entry point. Previously the lifecycle model could journal capture progress but could not distinguish a staged rootfs, a stopped backend, an activated replacement, or a committed storage transaction during restore. Treating every interruption as the same state would hide whether compensation is still possible. The Restoring state is entered only after the old backend is stopped and may return to Running only after storage commit. The journal keeps the selected target separate from last_checkpoint. That field records only the last completed capture; restore never changes it, while HEAD records the selection. Storage, backend, manager, and API implementations remain out of scope. Signed-off-by: Jing Zhang <renyu.zj@linux.alibaba.com> Signed-off-by: Weisson <Weisson@linux.alibaba.com>
This enables FileStorageProvider to prepare a checkpoint rootfs through opt-in StorageProvider restore transactions and later select that staged copy. Staging copies the checkpoint without changing the live rootfs. Activation uses same-directory renames to retain the original inode. Durable states let commit, abort, or restart reconciliation finish each interrupted boundary. This trades temporary disk capacity for deterministic rollback. The provider contract is fail-closed by default. Lifecycle orchestration, backend restore, daemon routes, and user-facing documentation remain unwired. Signed-off-by: Jing Zhang <renyu.zj@linux.alibaba.com> Signed-off-by: Weisson <Weisson@linux.alibaba.com>
Blaze could capture and list full checkpoints, but the daemon had no route that selected one as the live runtime state. Callers could inspect history but could not replace a running sandbox from that history. Add opt-in storage and backend restore contracts, Firecracker and mock adapters, a verified checkpoint target, and the manager-owned rollback API. This enables callers to replace a running sandbox from a selected checkpoint. The file provider stages an independent rootfs while the old backend runs, retains the predecessor through activation, and commits only after the new owner is ready and HEAD has changed. Firecracker requires the exact captured binary version before starting a replacement process. Failures before backend shutdown restore the original running state. Later failures retain the resources that actually exist under RecoveryRequired, including partial owners and uncertain HEAD outcomes, so destroy can finish cleanup without losing ownership. The contracts, adapters, manager, API, tests, and documentation stay together because they form the first production restore path; none is independently usable without the others. Signed-off-by: Jing Zhang <renyu.zj@linux.alibaba.com> Signed-off-by: Weisson <Weisson@linux.alibaba.com>
This was referenced Aug 13, 2026
github-actions
Bot
added
component:blaze
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This draft adds durable checkpoint restore and rollback on top of the generic checkpoint catalog and backend adapter. The final workflow will validate checkpoint artifacts, transfer storage and backend ownership through journaled phases, publish only a ready restored generation, and reconcile interrupted operations on restart.
Draft status: the current branch is a historical stacked candidate with three restore-oriented commits. The latest architecture review requires five independently understandable commits and a rebuild after #2471, #2472, and #2473. The current cumulative diff and old test results are not the final review evidence.
Related Issue
closes #2463
Type of Change
Scope
blaze(blaze)Checklist
package-lock.json/Cargo.lock)Testing
Not run for the final rebased commits. Required evidence includes default and all-feature Linux checks for every commit, focused restore-state tests, artifact and host-shape validation, cancellation and restart at each handoff, removal proofs, and real Firecracker restore smoke tests.
Additional Notes
Depends on #2471, #2472, and #2473. Checkpoint-history pruning in #2474 is independent and must not become a runtime dependency of restore. This branch will be rewritten after its prerequisites merge.