Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 45 additions & 3 deletions docs/user-guide/en/runtime/blaze.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,8 @@ to execute commands, read files, and write files inside them. Sandbox
destruction uses `DELETE /v1/sandboxes/{id}`. Checkpoint capture and history
use
`POST /v1/sandboxes/{id}/checkpoint` and
`GET /v1/sandboxes/{id}/checkpoints`.
`GET /v1/sandboxes/{id}/checkpoints`, and
`POST /v1/sandboxes/{id}/checkpoints/prune` removes unreachable history.

## Host Integration Boundary

Expand Down Expand Up @@ -283,8 +284,49 @@ resumes the backend, and leaves the sandbox running. If Blaze cannot prove the
publication, HEAD update, persistence, or backend-resume outcome, it retains
the durable record and reports `RecoveryRequired`; do not retry capture until
the sandbox has been reconciled or destroyed. A committed checkpoint that did
not become HEAD can still appear in history with `is_head: false`. This release
does not provide checkpoint restore, deletion, or pruning APIs.
not become HEAD can still appear in history with `is_head: false`.

### Pruning unreachable history

```http
POST /v1/sandboxes/{id}/checkpoints/prune
```

Prune removes every committed checkpoint that is neither reachable from the
current HEAD nor listed in the request's `protected` array. The request body
is optional; when present it must be a JSON object with only the `protected`
field:

```json
{ "protected": ["ckpt-11111111-1111-4111-8111-111111111111"] }
```

An empty or absent body is equivalent to `{"protected": []}`. Each protected
identifier must name a currently committed checkpoint; an unknown identifier
is rejected with HTTP 400 before any deletion. The response reports the number
of removed checkpoints and their identifiers:

```json
{ "count": 1, "removed": ["ckpt-22222222-2222-4222-8222-222222222222"] }
```

Prune protects the current HEAD, its full parent chain, and the ancestor chain
of every explicitly protected identifier. It only publishes each removal after
a named tombstone directory has replaced the candidate; the tombstone and its
paired owner file let subsequent restart cleanup finish a partially completed
prune or preserve a concurrently reappearing candidate rather than silently
discarding it. The endpoint acquires the same per-sandbox operation lock as
capture and list, so prune serializes with any other checkpoint operation.

If a failure occurs after a candidate has already left the committed catalog,
Blaze reports the confirmed removals, marks the sandbox `RecoveryRequired`, and
returns an error instead of a success body. Prune requests for a
`RecoveryRequired` sandbox are rejected with HTTP 409: destroy the sandbox or
restart the daemon so the retained `.prune.*` scratch is reconciled before
pruning again.

This release still does not provide checkpoint restore or per-checkpoint
deletion outside prune.

## Storage Artifact Synchronization

Expand Down
40 changes: 37 additions & 3 deletions docs/user-guide/zh/runtime/blaze.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,8 @@ Blaze 通过 `/v1/sandboxes` 提供沙箱生命周期和客户机操作。客户
命名空间列出、创建、查看和删除沙箱,以及在沙箱内执行命令、读取文件和写入
文件。销毁沙箱使用 `DELETE /v1/sandboxes/{id}`。检查点捕获与历史查询分别使用
`POST /v1/sandboxes/{id}/checkpoint` 和
`GET /v1/sandboxes/{id}/checkpoints`。
`GET /v1/sandboxes/{id}/checkpoints`;
`POST /v1/sandboxes/{id}/checkpoints/prune` 用于删除不可达的历史。

## 主机集成边界

Expand Down Expand Up @@ -240,8 +241,41 @@ sandbox 或修改其生命周期记录前返回 HTTP 501。
能够确认发生在发布前的失败会删除临时数据、恢复后端,并让 sandbox 保持运行。
如果 Blaze 无法确认发布、HEAD 更新、持久化或后端恢复的结果,则会保留持久记录并
报告 `RecoveryRequired`;在 sandbox 完成恢复处理或销毁前,不应重试捕获。已经提交但
未成为 HEAD 的检查点仍可能出现在历史列表中,其 `is_head` 为 `false`。当前版本
不提供检查点恢复、删除或清理接口。
未成为 HEAD 的检查点仍可能出现在历史列表中,其 `is_head` 为 `false`。

### 裁剪不可达的检查点历史

```http
POST /v1/sandboxes/{id}/checkpoints/prune
```

Prune 会删除既不属于当前 HEAD 父链、也不在请求 `protected` 数组中的所有已提交
检查点。请求体可选;若存在则必须是仅包含 `protected` 字段的 JSON 对象:

```json
{ "protected": ["ckpt-11111111-1111-4111-8111-111111111111"] }
```

空请求体等价于 `{"protected": []}`。`protected` 中每个标识符都必须对应当前已
提交的检查点;未知标识符会在任何删除前以 HTTP 400 拒绝。响应给出被删除检查点
的数量与列表:

```json
{ "count": 1, "removed": ["ckpt-22222222-2222-4222-8222-222222222222"] }
```

Prune 会保护当前 HEAD、其完整父链,以及每个显式保护标识符的祖先链。删除只在
候选目录被一个命名的墓碑目录替换之后发布;墓碑及其配对的 owner 文件让重启清理
能够完成半途终止的 prune,或在候选目录被并发重新出现时保留它,而不是被静默
丢弃。该接口占用与捕获、查询相同的 sandbox 操作锁,因此 prune 与其他任何检查点
操作串行化。

如果失败发生在某个候选已经离开已提交目录之后,Blaze 会报告已确认删除的列表、把
sandbox 标记为 `RecoveryRequired`,并返回错误而不是成功响应。对处于
`RecoveryRequired` 的 sandbox 发起 prune 会以 HTTP 409 拒绝:需要先销毁该
sandbox 或重启 daemon,让残留的 `.prune.*` 得到清理,然后才能再次 prune。

当前版本仍不提供除 prune 之外的检查点恢复或按标识单独删除。

## 存储制品同步

Expand Down
14 changes: 13 additions & 1 deletion src/blaze/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,7 @@ Blaze exposes sandbox lifecycle and guest operations through `/v1/sandboxes`.
| POST | `/v1/sandboxes/{id}/write` | Replace a guest file |
| POST | `/v1/sandboxes/{id}/checkpoint` | Capture a full checkpoint |
| GET | `/v1/sandboxes/{id}/checkpoints` | List committed checkpoint history |
| POST | `/v1/sandboxes/{id}/checkpoints/prune` | Remove unreachable checkpoint history |
| GET | `/v1/pools` | Reserved; returns `501` |
| GET | `/v1/pools/{backend}/{class}` | Reserved; returns `501` |
| POST | `/v1/pools/{backend}/{class}/drain` | Reserved; returns `501` |
Expand Down Expand Up @@ -258,7 +259,18 @@ sandbox state.

`GET /v1/sandboxes/{id}/checkpoints` returns committed history summaries,
including parentage, logical size, current-HEAD status, and HEAD reachability.
This release does not provide checkpoint restore or deletion.

`POST /v1/sandboxes/{id}/checkpoints/prune` deletes checkpoints that are not
reachable from the current HEAD and are not listed in the optional
`protected` array of the request body. The body may be empty (equivalent to
`{"protected": []}`) or must be a JSON object with only `protected`. The
response reports how many checkpoints were removed and their identifiers. The
current HEAD and its full parent chain, together with the ancestor chains of
every protected identifier, are never removed.

This release does not provide checkpoint restore or deletion of individual
checkpoints; use prune to reclaim unreachable branches.

See the [checkpoint capture user guide](../../docs/user-guide/en/runtime/blaze.md#checkpoint-capture-and-history)
for response fields, current backend support, and failure handling.

Expand Down
14 changes: 12 additions & 2 deletions src/blaze/README_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,7 @@ Blaze 通过 `/v1/sandboxes` 提供沙箱生命周期和客户机操作。
| POST | `/v1/sandboxes/{id}/write` | 替换 guest 文件 |
| POST | `/v1/sandboxes/{id}/checkpoint` | 捕获完整检查点 |
| GET | `/v1/sandboxes/{id}/checkpoints` | 列出已提交的检查点历史 |
| POST | `/v1/sandboxes/{id}/checkpoints/prune` | 删除不可达的检查点历史 |
| GET | `/v1/pools` | 预留接口;返回 `501` |
| GET | `/v1/pools/{backend}/{class}` | 预留接口;返回 `501` |
| POST | `/v1/pools/{backend}/{class}/drain` | 预留接口;返回 `501` |
Expand Down Expand Up @@ -228,8 +229,17 @@ daemon 才会逐个处理未结束的 sandbox。后续逐项恢复期间,如
改变 sandbox 状态前返回 HTTP 501。

`GET /v1/sandboxes/{id}/checkpoints` 返回已提交检查点的历史摘要,包括父检查点、
逻辑大小、是否为当前 HEAD,以及能否从 HEAD 到达。当前版本不提供检查点恢复或
删除接口。响应字段、当前后端支持情况和失败处理方式参见
逻辑大小、是否为当前 HEAD,以及能否从 HEAD 到达。

`POST /v1/sandboxes/{id}/checkpoints/prune` 会删除既不属于当前 HEAD 的父链、也
不在请求体 `protected` 数组中列出的检查点。请求体可以为空(等价于
`{"protected": []}`),或必须是仅包含 `protected` 字段的 JSON 对象。响应给出
被删除检查点的数量和标识列表。当前 HEAD 及其完整父链,加上 `protected` 中每个
标识符的完整父链,永远不会被删除。

本版本不提供检查点恢复或按标识单独删除;使用 prune 可回收不可达的分支。

响应字段、当前后端支持情况和失败处理方式参见
[检查点捕获用户指南](../../docs/user-guide/zh/runtime/blaze.md#检查点捕获与历史)。

### Guest 操作
Expand Down
199 changes: 199 additions & 0 deletions src/blaze/crates/blazed/src/api.rs
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,9 @@ async fn dispatch(
("POST", ["v1", "sandboxes", id, "write"]) => write_sandbox_file(state, id, &body).await,
("POST", ["v1", "sandboxes", id, "checkpoint"]) => checkpoint(state, id).await,
("GET", ["v1", "sandboxes", id, "checkpoints"]) => list_checkpoints(state, id).await,
("POST", ["v1", "sandboxes", id, "checkpoints", "prune"]) => {
prune_checkpoints(state, id, &body).await
}
("DELETE", ["v1", "sandboxes", id]) => destroy_sandbox(state, id).await,
("GET", ["v1", "pools"])
| ("GET", ["v1", "pools", _, _])
Expand Down Expand Up @@ -340,6 +343,35 @@ async fn list_checkpoints(state: &Arc<ServerState>, id: &str) -> Result<Response
json_ok(&state.manager.list_checkpoints(parse_uuid(id)?).await?)
}

#[derive(Debug, Default, Deserialize)]
#[serde(deny_unknown_fields)]
struct PruneRequest {
#[serde(default)]
protected: Vec<String>,
}

async fn prune_checkpoints(
state: &Arc<ServerState>,
id: &str,
body: &[u8],
) -> Result<Response<Full<Bytes>>> {
let uuid = parse_uuid(id)?;
let request: PruneRequest = if body.is_empty() {
PruneRequest::default()
} else {
serde_json::from_slice(body)
.map_err(|error| BlazeDaemonError::BadRequest(format!("invalid prune body: {error}")))?
};
let removed = state
.manager
.prune_checkpoints(uuid, request.protected)
.await?;
json_ok(&json!({
"count": removed.len(),
"removed": removed,
}))
}

async fn destroy_sandbox(state: &Arc<ServerState>, id: &str) -> Result<Response<Full<Bytes>>> {
let uuid = parse_uuid(id)?;
state.manager.destroy(uuid).await?;
Expand Down Expand Up @@ -1838,6 +1870,128 @@ mod tests {
);
}

#[tokio::test]
async fn prune_route_reports_no_removals_when_history_is_all_reachable() {
let temp = tempfile::tempdir().expect("temp");
let config = test_config(&temp);
let storage: Arc<dyn StorageProvider> = Arc::new(FileStorageProvider::with_images(
config.storage.images_dir.clone(),
config.storage.instances_dir.clone(),
));
let state = build_test_state(
config,
test_policy(BackendKind::Mock),
spawners(BackendKind::Mock, Arc::new(MockSpawner)),
BackendKind::Mock,
storage,
);
let created = created_json(&state, &test_request()).await;
let id = created["instance"]["id"].as_str().expect("id");
let uuid = Uuid::parse_str(id).expect("uuid");
let slot = write_checkpoint_fixture(&state, id).await;

let root = state
.manager
.checkpoint(uuid)
.await
.expect("root checkpoint")
.id;
tokio::fs::write(&slot.rootfs_path, b"second-rootfs")
.await
.expect("second rootfs");
let head = state
.manager
.checkpoint(uuid)
.await
.expect("head checkpoint")
.id;

let (status, response) = dispatched_json(
&state,
Method::POST,
&format!("/v1/sandboxes/{id}/checkpoints/prune"),
Vec::new(),
)
.await;
assert_eq!(status, StatusCode::OK);
assert_eq!(response["count"], 0);
assert!(response["removed"].as_array().expect("removed").is_empty());

let listed = state
.manager
.list_checkpoints(uuid)
.await
.expect("list after prune");
let remaining: std::collections::HashSet<String> =
listed.into_iter().map(|info| info.id).collect();
assert!(remaining.contains(&root));
assert!(remaining.contains(&head));
}

#[tokio::test]
async fn prune_route_rejects_unknown_body_fields() {
let temp = tempfile::tempdir().expect("temp");
let config = test_config(&temp);
let storage: Arc<dyn StorageProvider> = Arc::new(FileStorageProvider::with_images(
config.storage.images_dir.clone(),
config.storage.instances_dir.clone(),
));
let state = build_test_state(
config,
test_policy(BackendKind::Mock),
spawners(BackendKind::Mock, Arc::new(MockSpawner)),
BackendKind::Mock,
storage,
);
let created = created_json(&state, &test_request()).await;
let id = created["instance"]["id"].as_str().expect("id");

let (status, body) = handled_json(
&state,
Method::POST,
&format!("/v1/sandboxes/{id}/checkpoints/prune"),
br#"{"unknown_field":true}"#.to_vec(),
)
.await;
assert_eq!(status, StatusCode::BAD_REQUEST);
assert_eq!(body["status"], 400);
}

#[tokio::test]
async fn prune_route_rejects_unknown_protected_checkpoint_as_bad_request() {
let temp = tempfile::tempdir().expect("temp");
let config = test_config(&temp);
let storage: Arc<dyn StorageProvider> = Arc::new(FileStorageProvider::with_images(
config.storage.images_dir.clone(),
config.storage.instances_dir.clone(),
));
let state = build_test_state(
config,
test_policy(BackendKind::Mock),
spawners(BackendKind::Mock, Arc::new(MockSpawner)),
BackendKind::Mock,
storage,
);
let created = created_json(&state, &test_request()).await;
let id = created["instance"]["id"].as_str().expect("id");

let missing = format!("ckpt-{}", Uuid::new_v4());
let body_bytes = format!(r#"{{"protected":["{missing}"]}}"#).into_bytes();
let (status, body) = handled_json(
&state,
Method::POST,
&format!("/v1/sandboxes/{id}/checkpoints/prune"),
body_bytes,
)
.await;
assert_eq!(
status,
StatusCode::BAD_REQUEST,
"canonical but uncommitted protected id must surface as 400"
);
assert_eq!(body["status"], 400);
}

#[cfg(feature = "test-failpoints")]
#[tokio::test]
async fn checkpoint_cleanup_failure_keeps_destroy_recoverable() {
Expand Down Expand Up @@ -1884,6 +2038,51 @@ mod tests {
assert!(!checkpoint_namespace.exists());
}

#[cfg(feature = "test-failpoints")]
#[tokio::test]
async fn prune_route_rejects_a_recovery_required_sandbox() {
let temp = tempfile::tempdir().expect("temp");
let state = mock_state(&temp);
let created = created_json(&state, &test_request()).await;
let id = created["instance"]["id"].as_str().expect("id");
let uuid = Uuid::parse_str(id).expect("uuid");
write_checkpoint_fixture(&state, id).await;
state
.manager
.checkpoint(uuid)
.await
.expect("seed checkpoint");

// Drive the sandbox into RecoveryRequired through an existing failure
// path so the prune guard sees the durable state, not a synthetic one.
let hook = crate::failpoint::TestFailpoint::new(&[
"checkpoint-store-sandbox-remove-before-unlink",
]);
hook.run(state.manager.destroy(uuid))
.await
.expect_err("destroy must fail and mark recovery-required");
assert_eq!(
state.manager.get(uuid).expect("lifecycle").state,
SandboxState::RecoveryRequired
);

// Prune keeps no operation journal, so this must be rejected by the
// explicit recovery-required guard rather than silently succeeding.
let (status, body) = handled_json(
&state,
Method::POST,
&format!("/v1/sandboxes/{id}/checkpoints/prune"),
Vec::new(),
)
.await;
assert_eq!(
status,
StatusCode::CONFLICT,
"a recovery-required sandbox must not accept prune retries"
);
assert_eq!(body["status"], 409);
}

#[cfg(feature = "test-failpoints")]
#[tokio::test]
async fn checkpoint_snapshot_failure_resumes_and_clears_the_journal() {
Expand Down
Loading
Loading