Skip to content

feat(blaze): capture sandbox checkpoints - #2472

Draft
WeissonHan wants to merge 2 commits into
alibaba:mainfrom
WeissonHan:feature/blaze/checkpoint-restore-draft-v1
Draft

feat(blaze): capture sandbox checkpoints#2472
WeissonHan wants to merge 2 commits into
alibaba:mainfrom
WeissonHan:feature/blaze/checkpoint-restore-draft-v1

Conversation

@WeissonHan

@WeissonHan WeissonHan commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Description

Blaze can now create and inspect self-contained checkpoints through the
sandbox API:

  • POST /v1/sandboxes/{id}/checkpoint
  • GET /v1/sandboxes/{id}/checkpoints

A successful capture pauses the sandbox backend, records backend state and
guest memory, copies the provider-owned writable root, publishes an
integrity-checked manifest, advances the checkpoint HEAD, and resumes the
sandbox. The response retains the existing checkpoint_id and
instance_id JSON fields.

The former /v1/instances namespace is not restored. Requests below that
namespace continue to return 404 without reading or changing sandbox
state.

Behavior and failure boundary

  • Capture is accepted only for a running sandbox with no unfinished operation.
  • Both the backend and storage provider must explicitly advertise full capture
    support. Unsupported combinations return 501 before state mutation.
  • Checkpoint intent and each durable publication boundary are journaled.
  • Known pre-publication failures remove temporary data and resume the sandbox.
  • Unknown publication, persistence, HEAD-update, or resume outcomes retain
    ownership and require recovery instead of guessing or deleting data.
  • Listing validates committed manifests and reports parentage, logical size,
    HEAD status, and HEAD reachability.
  • Sandbox destruction removes checkpoint transaction data and committed
    checkpoint history. If that cleanup cannot be proven complete, destruction
    records RecoveryRequired and retains the sandbox's storage ownership.
    A retry synchronizes the checkpoint catalog even when the sandbox namespace
    was already unlinked.

Relationship to #2585

This draft is based on the exact #2585 head
e0ebb106b963ba4784c0daba7a3370b1ddf3e508, which removes the legacy instance
routes and establishes /v1/sandboxes as the only lifecycle namespace. The
checkpoint implementation itself is the single following commit
6f2234d998658727bd20b27ece2e857dfff7dcbf.

The PR does not depend on #2293, #2296, or #2471. It remains a draft until
#2585 is merged. After that merge, this one checkpoint commit will be replayed
onto the then-current main, revalidated, and presented without #2585's
changes in this PR's own diff.

Scope

Included:

  • checkpoint data models and validation;
  • durable checkpoint operation phases;
  • backend and storage capture capabilities;
  • descriptor-retained checkpoint catalog and atomic HEAD publication;
  • sandbox capture and history handlers;
  • cancellation, restart, replacement, and publication-failure tests;
  • English and Chinese user and design documentation.

Not included:

  • checkpoint restore;
  • checkpoint deletion or pruning;
  • Firecracker, Bubblewrap, or process-backend capture support;
  • any /v1/instances compatibility route.

Validation

Exact revision:

  • commit: 6f2234d998658727bd20b27ece2e857dfff7dcbf
  • tree: bea101768daadb02ddf062fd838bce47235aaa4e
  • parent: e0ebb106b963ba4784c0daba7a3370b1ddf3e508

Author-run validation used Linux x86_64 with Rust and Cargo 1.88, locked and
offline dependencies, a fresh source tree, a fresh Cargo home, and separate
empty target directories for each major stage.

Passed:

  • cargo fmt --all -- --check;
  • locked metadata, default and all features;
  • workspace all-target builds, default and all features;
  • strict Clippy with -D warnings, default and all features;
  • serial workspace tests: 354 default and 397 all-feature tests;
  • strict rustdoc, default and all features;
  • repository documentation lint and link checks;
  • 42 exact checkpoint, parent-directory durability retry, retired-route body
    precedence, routing, cancellation, recovery, catalog, storage, and backend
    regression tests;
  • source-tree identity and commit-message/trailer checks;
  • strict Clippy confirms that no unused production checkpoint entry points remain.

Evidence archive SHA-256:
1863cf0bcd2d3df01612e0a58d4b78e9f977833a679939ed672c1c307527fbe4.

The exact Linux gate passed. Hosted checks and an exact-head Codex review are
still required for this rewritten draft. All validation will be repeated after
#2585 merges and the commit is replayed onto the current main.

Related issue

Closes #2444

Copy link
Copy Markdown
Collaborator Author

@codex review Please review the generic full-checkpoint capture and listing workflow, with particular attention to durable publication boundaries, cancellation and restart recovery, and retained filesystem ownership.

@WeissonHan
WeissonHan force-pushed the feature/blaze/checkpoint-restore-draft-v1 branch from c7aa317 to 56bc391 Compare August 13, 2026 13:43

Copy link
Copy Markdown
Collaborator Author

@codex review Please review the updated exact head. The only change since the previous request is a test-only feature gate required by the default-feature Rust 1.88 Clippy job; please focus on the generic checkpoint capture/list workflow and its durable publication, cancellation, restart, and retained-ownership behavior.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 56bc391ed7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/blaze/crates/blazed/src/api.rs Outdated
Comment thread src/blaze/crates/blazed/src/checkpoint_store.rs
@WeissonHan
WeissonHan force-pushed the feature/blaze/checkpoint-restore-draft-v1 branch from 56bc391 to 995bab7 Compare August 13, 2026 14:06

Copy link
Copy Markdown
Collaborator Author

@codex review Please review exact head 995bab7e, which addresses both prior P1 findings by preserving the established checkpoint response fields and removing snapshot-payload rehashing from list requests while retaining full hashing at mutation and explicit verification boundaries.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 995bab7ebb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/blaze/crates/blazed/src/checkpoint_store.rs Outdated
@WeissonHan
WeissonHan force-pushed the feature/blaze/checkpoint-restore-draft-v1 branch from 995bab7 to 93cbdb9 Compare August 13, 2026 15:26

Copy link
Copy Markdown
Collaborator Author

Addressed the exact-head Codex permission finding in 93cbdb994d76c30dbe3eb813fc79d43f477c3e64.

Checkpoint catalog, sandbox, staging, and committed directories are now created with owner-only 0700 permissions. Daemon-created metadata and HEAD files use 0600. Before publication, all three backend-produced artifacts are opened relative to the retained stage, validated as regular files owned by the daemon user with one link, and only then restricted to 0600, synchronized, and hashed. Validation completes for every artifact before any mode change, so a linked artifact fails without modifying another file or an external alias.

Focused coverage verifies the complete tree under a permissive umask, rejects external hard links without permission side effects, and preserves cleanup of an interrupted internal rootfs link. The two storage artifact synchronization documents remain unchanged. Exact-head hosted and native Linux validation are running now.

Copy link
Copy Markdown
Collaborator Author

@codex review Verify that checkpoint directories and published artifacts remain private under the packaged service defaults, including hard-link failure behavior.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: 93cbdb994d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@WeissonHan
WeissonHan marked this pull request as ready for review August 13, 2026 15:32
@WeissonHan
WeissonHan requested a review from casparant as a code owner August 13, 2026 15:32

@qoderai qoderai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] 沙箱检查点生命周期和 journal 设计整体合理,当前审查重点是并发保护与补偿路径是否始终与持久化边界一致。下面三点是本次审查中相对更重要的细节建议,均已通过行内评论标注。

  1. 检查点操作在并发下对 operation journal 的保护需要严格约束所有调用路径使用同一锁,并在 API 层明确拒绝重复 checkpoint 请求。
  2. 失败补偿与恢复路径目前主要根据状态和 staging id 区分,建议进一步利用 OperationPhase 的 durable 边界信息,以保证 journal 与目录真实状态持续同步。
  3. 文件存储检查点目标目录在 Linux 与其他平台的 retained adapter 语义存在差异,建议在能力探测和错误处理上做显式约束,并在设计文档中标明跨平台行为差异。

🤖 Generated by QoderView workflow run

Comment thread src/blaze/crates/blaze-core/src/lifecycle.rs
Comment thread src/blaze/crates/blazed/src/sandbox/checkpoint.rs
Comment thread src/blaze/crates/blazed/src/file_provider.rs

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 93cbdb994d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/blaze/crates/blazed/src/sandbox/checkpoint.rs Outdated
Comment thread src/blaze/crates/blaze-core/src/lifecycle.rs Outdated
@WeissonHan
WeissonHan marked this pull request as draft August 13, 2026 15:57
@WeissonHan
WeissonHan force-pushed the feature/blaze/checkpoint-restore-draft-v1 branch from 93cbdb9 to 39b9446 Compare August 13, 2026 16:14
@WeissonHan
WeissonHan marked this pull request as ready for review August 13, 2026 16:17

Copy link
Copy Markdown
Collaborator Author

@codex review Add generic full-checkpoint capture and listing with durable publication boundaries and retained staging ownership.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 39b9446d7b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/blaze/crates/blazed/src/file_provider.rs Outdated
Comment thread src/blaze/crates/blazed/src/checkpoint_store.rs
@WeissonHan
WeissonHan marked this pull request as draft August 13, 2026 16:24
@WeissonHan
WeissonHan force-pushed the feature/blaze/checkpoint-restore-draft-v1 branch from 39b9446 to dcc2284 Compare August 13, 2026 17:02
@WeissonHan
WeissonHan marked this pull request as ready for review August 13, 2026 17:05

Copy link
Copy Markdown
Collaborator Author

@codex review Add generic full-checkpoint capture and listing while preserving sparse storage and bounded capture-time lineage validation.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dcc2284322

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/blaze/crates/blazed/src/checkpoint_store.rs
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: 5975b2a32a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@WeissonHan

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: 5975b2a32a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copy link
Copy Markdown
Collaborator Author

@codex review Please review exact head e07d50c34286d2b7b856cd46529d7d02dcbda1ce and the single checkpoint commit after parent e044518dcbcff8475072b269a7087962078187ef. Please verify that only the two /v1/sandboxes/{id} checkpoint routes are registered, every /v1/instances route remains unavailable without side effects, checkpoint publication and recovery preserve ownership, and no dead or test-only production code was introduced. Report any P1 or P2 issue.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e07d50c342

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/blaze/crates/blazed/src/checkpoint_store.rs Outdated
@WeissonHan
WeissonHan force-pushed the feature/blaze/checkpoint-restore-draft-v1 branch from e07d50c to 27ad45f Compare August 16, 2026 16:02

Copy link
Copy Markdown
Collaborator Author

@codex review Please review exact head 27ad45fa2650dd6e1593fa70644697269086ad40, its checkpoint commit, and the cumulative diff from current main. Focus on the two /v1/sandboxes/{id} checkpoint routes, rejection of every /v1/instances request before body collection, fail-closed capability checks, durable checkpoint ownership across cancellation and restart, atomic catalog and HEAD publication, cleanup and replacement safety, absence of dead production code, and the English/Chinese user and design documentation. Please report any P1 or P2 issue.

@WeissonHan
WeissonHan force-pushed the feature/blaze/checkpoint-restore-draft-v1 branch 2 times, most recently from e1ca439 to f53312f Compare August 16, 2026 16:14

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 27ad45fa26

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/blaze/crates/blazed/src/sandbox/manager.rs Outdated
Comment thread docs/user-guide/en/runtime/blaze.md Outdated

Copy link
Copy Markdown
Collaborator Author

@codex review Please review exact head f53312f4eec78dec81cdd410296d01d1510696f3, its checkpoint commit, and the cumulative diff from current main. Focus on the two /v1/sandboxes/{id} checkpoint routes, early rejection of every /v1/instances request, retrying the checkpoint catalog parent synchronization after an initial durability failure, fail-closed capability checks, durable ownership across cancellation and restart, atomic catalog and HEAD publication, cleanup and replacement safety, absence of dead production code, and the English/Chinese documentation. Please report any P1 or P2 issue.

Expose lifecycle and guest operations only under /v1/sandboxes. Leave the
former /v1/instances namespace unregistered so requests return 404 without
reading or changing sandbox state.

Remove the reset and placeholder checkpoint handlers together with their
now-unused server-state accessors. Keep existing JSON field names unchanged;
checkpoint capture follows separately.

Signed-off-by: Weisson <Weisson@linux.alibaba.com>
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 👍

Reviewed commit: f53312f4ee

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@WeissonHan
WeissonHan force-pushed the feature/blaze/checkpoint-restore-draft-v1 branch 5 times, most recently from c951f78 to caa05a1 Compare August 16, 2026 16:47

Copy link
Copy Markdown
Collaborator Author

@codex review Please review exact head caa05a1a07280576609e9c488488288e1b152e94 and the checkpoint implementation commit relative to parent e0ebb106b963ba4784c0daba7a3370b1ddf3e508. Focus on the two supported sandbox checkpoint routes, absence of every /v1/instances route, durable capture and publication ordering, cancellation and restart recovery, complete checkpoint-namespace removal during sandbox destruction with RecoveryRequired on uncertain cleanup, removal of unused production entry points, and consistent English/Chinese documentation. Please report any P1 or P2 issue.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: caa05a1a07

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/blaze/crates/blazed/src/checkpoint_store.rs Outdated
@WeissonHan
WeissonHan force-pushed the feature/blaze/checkpoint-restore-draft-v1 branch from caa05a1 to c115f2c Compare August 16, 2026 16:58
Add checkpoint creation and history queries under the sandbox API.
Capture backend state, guest memory, and the provider-owned writable root
into a self-contained, integrity-checked catalog.

Journal each durable boundary so cancellation and restart retain
ownership until publication, HEAD movement, state persistence, and backend
resume have converged. Unsupported providers fail before mutation.

Keep the former instance namespace unregistered. Restore, checkpoint
deletion, pruning, and Firecracker capture remain outside this change.

Signed-off-by: Jing Zhang <renyu.zj@linux.alibaba.com>
Signed-off-by: Weisson <Weisson@linux.alibaba.com>
@WeissonHan
WeissonHan force-pushed the feature/blaze/checkpoint-restore-draft-v1 branch from c115f2c to 6f2234d Compare August 16, 2026 17:00

Copy link
Copy Markdown
Collaborator Author

@codex review Please review exact head 6f2234d998658727bd20b27ece2e857dfff7dcbf and the checkpoint implementation commit relative to parent e0ebb106b963ba4784c0daba7a3370b1ddf3e508. Focus on the two supported sandbox checkpoint routes, absence of every /v1/instances route, durable capture and publication ordering, cancellation and restart recovery, complete checkpoint-namespace removal during sandbox destruction, retry of the catalog durability boundary when the namespace is already absent, RecoveryRequired on uncertain cleanup, removal of unused production entry points, and consistent English/Chinese documentation. Please report any P1 or P2 issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[blaze] feat: capture and list sandbox checkpoints

1 participant