diff --git a/docs/user-guide/en/runtime/blaze.md b/docs/user-guide/en/runtime/blaze.md index 960b4f6db..8cb98f682 100644 --- a/docs/user-guide/en/runtime/blaze.md +++ b/docs/user-guide/en/runtime/blaze.md @@ -93,7 +93,10 @@ responds. Blaze exposes sandbox lifecycle and guest operations under `/v1/sandboxes`. Clients use this namespace to list, create, inspect, and delete sandboxes and to execute commands, read files, and write files inside them. Sandbox -destruction uses `DELETE /v1/sandboxes/{id}`. +destruction uses `DELETE /v1/sandboxes/{id}`. Checkpoint capture and history +use +`POST /v1/sandboxes/{id}/checkpoint` and +`GET /v1/sandboxes/{id}/checkpoints`. ## Host Integration Boundary @@ -211,6 +214,78 @@ The lifecycle invariants behind these compatibility responses are recorded in the [lifecycle state consistency and compatibility design](../../../../src/blaze/docs/design/lifecycle-state-consistency.md). +## Checkpoint Capture and History + +Blaze captures a running sandbox through +`POST /v1/sandboxes/{id}/checkpoint`. + +Capture requires both the selected backend and the storage provider to +advertise full-checkpoint support. The built-in file provider captures the +writable root filesystem, and the built-in mock backend supplies a complete +development implementation. Firecracker, Bubblewrap, and the other process +backends do not advertise capture support in this release. An unsupported +combination returns HTTP 501 before the sandbox is paused or its lifecycle +record is changed. + +For a supported running sandbox, Blaze holds the sandbox operation lock, +validates its current checkpoint parent, pauses the backend, and captures three +self-contained files: `vmstate.snap`, `memory.snap`, and `rootfs.snap`. It +synchronizes and hashes those files, publishes the manifest, atomically updates +the sandbox checkpoint HEAD, and resumes the backend. Guest operations and +other lifecycle changes wait for the same operation lock while capture is in +progress. + +A successful response contains the complete published manifest. The existing +`checkpoint_id` and `instance_id` fields identify the same checkpoint and +sandbox as `id` and `sandbox_id`: + +```json +{ + "checkpoint_id": "ckpt-11111111-1111-4111-8111-111111111111", + "instance_id": "22222222-2222-4222-8222-222222222222", + "format_version": 1, + "id": "ckpt-11111111-1111-4111-8111-111111111111", + "parent": null, + "sandbox_id": "22222222-2222-4222-8222-222222222222", + "policy_name": "agent-tool", + "image_digest": "sha256:0000000000000000000000000000000000000000000000000000000000000000", + "backend": "mock", + "backend_version": "mock-v1", + "created_at": "2026-08-14T00:00:00Z", + "snapshot_kind": "full", + "artifacts": [ + { + "name": "vmstate.snap", + "size_bytes": 4096, + "sha256": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + }, + { + "name": "memory.snap", + "size_bytes": 8192, + "sha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" + }, + { + "name": "rootfs.snap", + "size_bytes": 8589934592, + "sha256": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc" + } + ] +} +``` + +Use `GET /v1/sandboxes/{id}/checkpoints` to list committed history. Each list +entry contains `id`, `parent`, `created_at`, total logical `size_bytes`, +`is_head`, and `on_head_chain`. The list is a summary and does not repeat the +complete artifact manifest returned by capture. + +A failure known to occur before publication removes its temporary data, +resumes the backend, and leaves the sandbox running. If Blaze cannot prove the +publication, HEAD update, persistence, or backend-resume outcome, it retains +the durable record and reports `RecoveryRequired`; do not retry capture until +the sandbox has been reconciled or destroyed. A committed checkpoint that did +not become HEAD can still appear in history with `is_head: false`. This release +does not provide checkpoint restore, deletion, or pruning APIs. + ## Storage Artifact Synchronization Blaze can periodically persist the already-written host artifacts and directory diff --git a/docs/user-guide/zh/runtime/blaze.md b/docs/user-guide/zh/runtime/blaze.md index ceef6eb89..4001c8cfd 100644 --- a/docs/user-guide/zh/runtime/blaze.md +++ b/docs/user-guide/zh/runtime/blaze.md @@ -78,7 +78,9 @@ ID。Blaze 会保留被拒绝的记录。修复或恢复该记录后，重新启 Blaze 通过 `/v1/sandboxes` 提供沙箱生命周期和客户机操作。客户端使用该 命名空间列出、创建、查看和删除沙箱，以及在沙箱内执行命令、读取文件和写入 -文件。销毁沙箱使用 `DELETE /v1/sandboxes/{id}`。 +文件。销毁沙箱使用 `DELETE /v1/sandboxes/{id}`。检查点捕获与历史查询分别使用 +`POST /v1/sandboxes/{id}/checkpoint` 和 +`GET /v1/sandboxes/{id}/checkpoints`。 ## 主机集成边界 @@ -179,6 +181,68 @@ Blaze 仍可读取旧版本写入的 `Reset`、`Warm` 和 `start_path = "warm"` [生命周期状态一致性与兼容性设计](../../../../src/blaze/docs/design/lifecycle-state-consistency_zh.md) 中。 +## 检查点捕获与历史 + +Blaze 通过 `POST /v1/sandboxes/{id}/checkpoint` 捕获运行中的 sandbox。 + +所选后端和存储提供程序必须同时声明支持完整检查点捕获。内置文件存储提供程序 +负责捕获可写根文件系统，内置 `mock` 后端提供完整的开发环境实现。当前版本的 +Firecracker、Bubblewrap 和其他进程后端尚未声明支持捕获。不支持的组合会在暂停 +sandbox 或修改其生命周期记录前返回 HTTP 501。 + +对于受支持且正在运行的 sandbox，Blaze 会持有该 sandbox 的操作锁，验证当前 +检查点的父项，暂停后端，并捕获三个自包含文件：`vmstate.snap`、`memory.snap` +和 `rootfs.snap`。随后会同步文件、计算摘要、发布清单、原子更新该 sandbox 的 +检查点 HEAD，再恢复后端。捕获期间，对虚拟机内部执行的命令和文件操作，以及 +其他生命周期变更都会等待同一把操作锁。 + +成功响应包含已发布的完整清单。现有 `checkpoint_id` 和 `instance_id` 字段与 +`id` 和 `sandbox_id` 分别指向同一个检查点和 sandbox： + +```json +{ + "checkpoint_id": "ckpt-11111111-1111-4111-8111-111111111111", + "instance_id": "22222222-2222-4222-8222-222222222222", + "format_version": 1, + "id": "ckpt-11111111-1111-4111-8111-111111111111", + "parent": null, + "sandbox_id": "22222222-2222-4222-8222-222222222222", + "policy_name": "agent-tool", + "image_digest": "sha256:0000000000000000000000000000000000000000000000000000000000000000", + "backend": "mock", + "backend_version": "mock-v1", + "created_at": "2026-08-14T00:00:00Z", + "snapshot_kind": "full", + "artifacts": [ + { + "name": "vmstate.snap", + "size_bytes": 4096, + "sha256": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + }, + { + "name": "memory.snap", + "size_bytes": 8192, + "sha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" + }, + { + "name": "rootfs.snap", + "size_bytes": 8589934592, + "sha256": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc" + } + ] +} +``` + +可以通过 `GET /v1/sandboxes/{id}/checkpoints` 查询已提交的历史。每个列表项 +包含 `id`、`parent`、`created_at`、总逻辑大小 `size_bytes`、`is_head` 和 +`on_head_chain`。该列表只提供摘要，不会重复捕获响应中的完整制品清单。 + +能够确认发生在发布前的失败会删除临时数据、恢复后端，并让 sandbox 保持运行。 +如果 Blaze 无法确认发布、HEAD 更新、持久化或后端恢复的结果，则会保留持久记录并 +报告 `RecoveryRequired`；在 sandbox 完成恢复处理或销毁前，不应重试捕获。已经提交但 +未成为 HEAD 的检查点仍可能出现在历史列表中，其 `is_head` 为 `false`。当前版本 +不提供检查点恢复、删除或清理接口。 + ## 存储制品同步 Blaze 可以定期持久化 running sandbox 中已经写入的宿主机制品和目录元数据。 diff --git a/src/blaze/Cargo.lock b/src/blaze/Cargo.lock index f1e8cc7fe..ab69a0bf3 100644 --- a/src/blaze/Cargo.lock +++ b/src/blaze/Cargo.lock @@ -144,6 +144,7 @@ dependencies = [ "rustix", "serde", "serde_json", + "sha2", "tempfile", "thiserror", "tokio", @@ -154,6 +155,15 @@ dependencies = [ "uuid", ] +[[package]] +name = "block-buffer" +version = "0.10.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71" +dependencies = [ + "generic-array", +] + [[package]] name = "bumpalo" version = "3.20.3" @@ -256,6 +266,35 @@ version = "0.8.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" +[[package]] +name = "cpufeatures" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280" +dependencies = [ + "libc", +] + +[[package]] +name = "crypto-common" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" +dependencies = [ + "generic-array", + "typenum", +] + +[[package]] +name = "digest" +version = "0.10.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" +dependencies = [ + "block-buffer", + "crypto-common", +] + [[package]] name = "equivalent" version = "1.0.2" @@ -347,6 +386,16 @@ dependencies = [ "slab", ] +[[package]] +name = "generic-array" +version = "0.14.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" +dependencies = [ + "typenum", + "version_check", +] + [[package]] name = "getrandom" version = "0.4.2" @@ -837,6 +886,17 @@ dependencies = [ "serde", ] +[[package]] +name = "sha2" +version = "0.10.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283" +dependencies = [ + "cfg-if", + "cpufeatures", + "digest", +] + [[package]] name = "sharded-slab" version = "0.1.7" @@ -1139,6 +1199,12 @@ version = "0.2.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b" +[[package]] +name = "typenum" +version = "1.20.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20" + [[package]] name = "unicode-ident" version = "1.0.24" @@ -1175,6 +1241,12 @@ version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ba73ea9cf16a25df0c8caa16c51acb937d5712a8429db78a3ee29d5dcacd3a65" +[[package]] +name = "version_check" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" + [[package]] name = "want" version = "0.3.1" diff --git a/src/blaze/Cargo.toml b/src/blaze/Cargo.toml index e22afb5a4..5fe6cabf6 100644 --- a/src/blaze/Cargo.toml +++ b/src/blaze/Cargo.toml @@ -45,6 +45,7 @@ chrono = { version = "0.4", default-features = false, features = ["std", "clock" semver = "1.0" base64 = "0.22" rustix = { version = "1.1", features = ["fs"] } +sha2 = "0.10" # Internal crates blaze-core = { path = "crates/blaze-core" } diff --git a/src/blaze/README.md b/src/blaze/README.md index fc0435cce..8dd556beb 100644 --- a/src/blaze/README.md +++ b/src/blaze/README.md @@ -14,6 +14,9 @@ Designed as the per-host agent for E2B-style orchestrator platforms. - **HTTP API** — Unix domain socket (`/run/blaze/api.sock`) + TCP (`:14159`) - **Policy-driven backend selection** — workload class → backend priority list - **Lifecycle state machine** — durable state with restart recovery +- **Checkpoint capture** — full VM state, guest memory, and writable root + filesystem capture with queryable history for supported backends and storage + providers - **Guest operations** — bounded command execution and file transfer for running backends that expose a guest endpoint - **Template catalog** — bounded import and atomic publication of reusable artifacts @@ -148,6 +151,8 @@ Blaze exposes sandbox lifecycle and guest operations through `/v1/sandboxes`. | POST | `/v1/sandboxes/{id}/exec` | Execute a guest command | | POST | `/v1/sandboxes/{id}/read` | Read a guest file | | POST | `/v1/sandboxes/{id}/write` | Replace a guest file | +| POST | `/v1/sandboxes/{id}/checkpoint` | Capture a full checkpoint | +| GET | `/v1/sandboxes/{id}/checkpoints` | List committed checkpoint history | | GET | `/v1/pools` | Reserved; returns `501` | | GET | `/v1/pools/{backend}/{class}` | Reserved; returns `501` | | POST | `/v1/pools/{backend}/{class}/drain` | Reserved; returns `501` | @@ -233,13 +238,29 @@ See the for writer coordination, inventory publication, reset rejection, legacy-state cleanup, and failure boundaries. -The operation journal records the operation and start time, not completion of -each resource step. An interrupted create is cleaned up rather than resumed, -and an existing backend process is not adopted after restart. Failed recovery -does not run in a background retry loop. Checkpoint capture and restore are not -available in this change. Reset remains unavailable until runtime and storage -can be reset together; this recovery flow does not add backend snapshot, -capture, or restore operations. +The operation journal records create and destroy operations and the durable +phase reached by checkpoint capture. An interrupted create is cleaned up rather +than resumed, and an existing backend process is not adopted after restart. +Startup recovery destroys an interrupted sandbox instead of restoring its +checkpoint. Failed recovery does not run in a background retry loop. Reset +remains unavailable and does not restore a checkpoint. + +### Checkpoint capture and history + +`POST /v1/sandboxes/{id}/checkpoint` captures a running sandbox when both its +backend and storage provider advertise full-capture support. A successful +request pauses the backend, captures VM state, guest memory, and the writable +root filesystem, publishes a self-contained integrity manifest, moves the +sandbox checkpoint HEAD, and resumes the backend. The response includes the +complete manifest plus the `checkpoint_id` and `instance_id` fields. +Unsupported backend or storage combinations return HTTP 501 before changing +sandbox state. + +`GET /v1/sandboxes/{id}/checkpoints` returns committed history summaries, +including parentage, logical size, current-HEAD status, and HEAD reachability. +This release does not provide checkpoint restore or deletion. +See the [checkpoint capture user guide](../../docs/user-guide/en/runtime/blaze.md#checkpoint-capture-and-history) +for response fields, current backend support, and failure handling. ### Guest operations diff --git a/src/blaze/README_zh.md b/src/blaze/README_zh.md index 4c1cc5145..fe7fa67ef 100644 --- a/src/blaze/README_zh.md +++ b/src/blaze/README_zh.md @@ -13,6 +13,8 @@ Blaze 通过 HTTP API 管理 sandbox 实例的完整生命周期，支持策略 - **HTTP API** — Unix domain socket (`/run/blaze/api.sock`) + TCP (`:14159`) - **策略驱动后端选择** — workload class → 后端优先级列表 - **生命周期状态机** — 持久化状态，并支持重启恢复 +- **检查点捕获** — 对支持该能力的后端和存储提供程序捕获完整 VM 状态、客体 + 内存和可写根文件系统，并提供历史查询 - **Guest 操作** — 对提供 guest endpoint 的运行中后端执行有界命令和文件传输 - **Template catalog** — 有界导入并原子发布可复用 artifact - **内核 hook 注册** — 前/后置 hook 状态追踪 @@ -139,6 +141,8 @@ Blaze 通过 `/v1/sandboxes` 提供沙箱生命周期和客户机操作。 | POST | `/v1/sandboxes/{id}/exec` | 执行 guest 命令 | | POST | `/v1/sandboxes/{id}/read` | 读取 guest 文件 | | POST | `/v1/sandboxes/{id}/write` | 替换 guest 文件 | +| POST | `/v1/sandboxes/{id}/checkpoint` | 捕获完整检查点 | +| GET | `/v1/sandboxes/{id}/checkpoints` | 列出已提交的检查点历史 | | GET | `/v1/pools` | 预留接口；返回 `501` | | GET | `/v1/pools/{backend}/{class}` | 预留接口；返回 `501` | | POST | `/v1/pools/{backend}/{class}/drain` | 预留接口；返回 `501` | @@ -209,11 +213,24 @@ daemon 才会逐个处理未结束的 sandbox。后续逐项恢复期间，如 写入协调、清单发布、重置拒绝、旧状态清理和失败边界参见 [生命周期状态一致性与兼容性设计](docs/design/lifecycle-state-consistency_zh.md)。 -操作记录只保存操作类型和开始时间，不记录每个资源步骤是否已经完成。中断的 -创建会被清理而不是从原位置继续，重启后也不会接管先前的后端进程。恢复失败 -后目前没有后台循环自动重试。本次变更不提供检查点捕获或恢复。重置接口在 -运行环境和存储能够一起重置前不可用；这里的恢复流程没有增加后端快照、捕获 -或恢复操作。 +操作记录会保存创建和销毁操作，以及检查点捕获已经完成的持久化阶段。中断的 +创建会被清理而不是从原位置继续，重启后也不会接管先前的后端进程。启动恢复会 +销毁捕获中断的 sandbox，而不是从其检查点恢复。恢复失败后目前没有后台循环自动 +重试。重置接口仍不可用，也不会恢复检查点。 + +### 检查点捕获与历史 + +当运行中的 sandbox 所使用的后端和存储提供程序都声明支持完整捕获时， +`POST /v1/sandboxes/{id}/checkpoint` 会创建检查点。请求成功时，Blaze 会暂停后端， +捕获 VM 状态、客体内存和可写根文件系统，发布包含完整性信息的自包含清单，更新 +该 sandbox 的检查点 HEAD，然后恢复后端。响应包含完整清单，以及 +`checkpoint_id` 和 `instance_id` 字段。后端或存储组合不支持该能力时，接口会在 +改变 sandbox 状态前返回 HTTP 501。 + +`GET /v1/sandboxes/{id}/checkpoints` 返回已提交检查点的历史摘要，包括父检查点、 +逻辑大小、是否为当前 HEAD，以及能否从 HEAD 到达。当前版本不提供检查点恢复或 +删除接口。响应字段、当前后端支持情况和失败处理方式参见 +[检查点捕获用户指南](../../docs/user-guide/zh/runtime/blaze.md#检查点捕获与历史)。 ### Guest 操作 diff --git a/src/blaze/crates/blaze-core/src/backend.rs b/src/blaze/crates/blaze-core/src/backend.rs index 028cac0b2..98f057436 100644 --- a/src/blaze/crates/blaze-core/src/backend.rs +++ b/src/blaze/crates/blaze-core/src/backend.rs @@ -99,6 +99,28 @@ pub struct SpawnRequest { pub vm: Option, } +/// Snapshot flavor requested from a backend. +/// +/// The file provider currently requires self-contained artifacts, so only +/// full snapshots are exposed until a restore-independent delta format exists. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "lowercase")] +pub enum SnapshotKind { + /// Self-contained VM and memory snapshot. + Full, +} + +/// Paths and semantics for one snapshot operation. +#[derive(Debug, Clone)] +pub struct SnapshotRequest { + /// Destination for VM state. + pub snapshot_path: PathBuf, + /// Destination for guest memory. + pub mem_path: PathBuf, + /// Snapshot flavor. + pub kind: SnapshotKind, +} + /// Probed availability of a single backend on this host. #[derive(Debug, Clone, Serialize, Deserialize)] pub struct BackendStatus { @@ -203,4 +225,12 @@ mod tests { let err = select_backend(&priority, &available).expect_err("must fail"); assert!(matches!(err, BlazeError::BackendUnavailable { .. })); } + + #[test] + fn snapshot_kind_serializes_as_a_stable_lowercase_value() { + assert_eq!( + serde_json::to_value(SnapshotKind::Full).expect("snapshot kind"), + serde_json::json!("full") + ); + } } diff --git a/src/blaze/crates/blaze-core/src/checkpoint.rs b/src/blaze/crates/blaze-core/src/checkpoint.rs new file mode 100644 index 000000000..e7ccc4d01 --- /dev/null +++ b/src/blaze/crates/blaze-core/src/checkpoint.rs @@ -0,0 +1,443 @@ +// SPDX-License-Identifier: Apache-2.0 +//! Pure checkpoint records and manifest validation. +//! +//! This module deliberately contains no filesystem or path handling. The +//! daemon owns checkpoint persistence, hashing, publication, and cleanup. + +use std::collections::HashSet; + +use chrono::{DateTime, Utc}; +use serde::{Deserialize, Serialize}; +use thiserror::Error; +use uuid::Uuid; + +use crate::backend::{BackendKind, SnapshotKind}; + +/// Current on-disk checkpoint metadata format. +pub const CHECKPOINT_FORMAT_VERSION: u32 = 1; + +/// Self-contained artifacts required for every committed checkpoint. +pub const REQUIRED_ARTIFACTS: [&str; 3] = ["vmstate.snap", "memory.snap", "rootfs.snap"]; + +/// One content digest recorded in a checkpoint manifest. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct CheckpointArtifact { + /// File name relative to the checkpoint directory. + pub name: String, + /// Logical file size in bytes. + pub size_bytes: u64, + /// Lowercase SHA-256 digest. + pub sha256: String, +} + +/// Durable checkpoint identity and integrity manifest. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct CheckpointMetadata { + /// Metadata schema version. + pub format_version: u32, + /// Stable `ckpt-` identifier. + pub id: String, + /// Previous checkpoint on this branch. + #[serde(default)] + pub parent: Option, + /// Sandbox that owns the checkpoint. + pub sandbox_id: Uuid, + /// Policy that selected the captured runtime. + pub policy_name: String, + /// Image identity selected by the policy. + pub image_digest: String, + /// Backend that produced the runtime artifacts. + pub backend: BackendKind, + /// Backend version captured by the daemon, when available. + #[serde(default)] + pub backend_version: Option, + /// UTC publication time. + pub created_at: DateTime, + /// Backend snapshot semantics. + pub snapshot_kind: SnapshotKind, + /// Integrity records for all required artifacts. + pub artifacts: Vec, +} + +/// Read-only API view of a checkpoint. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct CheckpointInfo { + /// Checkpoint identifier. + pub id: String, + /// Parent checkpoint. + pub parent: Option, + /// Publication time. + pub created_at: DateTime, + /// Sum of logical artifact sizes. + pub size_bytes: u64, + /// Whether this checkpoint is the current HEAD. + pub is_head: bool, + /// Whether this checkpoint is reachable from HEAD. + pub on_head_chain: bool, +} + +/// Values supplied by the daemon when publishing a populated stage. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct CommitCheckpoint { + /// Parent checkpoint, if the sandbox already has a HEAD. + pub parent: Option, + /// Policy that selected the captured runtime. + pub policy_name: String, + /// Image identity selected by the policy. + pub image_digest: String, + /// Backend that produced the artifacts. + pub backend: BackendKind, + /// Backend version captured by the caller, when available. + pub backend_version: Option, + /// Backend snapshot semantics. + pub snapshot_kind: SnapshotKind, +} + +/// Pure validation failure for a checkpoint identifier or manifest. +#[derive(Debug, Clone, PartialEq, Eq, Error)] +pub enum CheckpointValidationError { + /// A checkpoint identifier is not canonical `ckpt-`. + #[error("invalid checkpoint identifier {checkpoint_id:?}: {reason}")] + InvalidIdentifier { + checkpoint_id: String, + reason: String, + }, + + /// A manifest uses a schema version this daemon cannot interpret. + #[error("checkpoint {checkpoint_id} uses unsupported format {actual}; expected {expected}")] + UnsupportedFormat { + checkpoint_id: String, + actual: u32, + expected: u32, + }, + + /// The durable identity does not agree with the catalog location. + #[error("checkpoint manifest identity mismatch: {reason}")] + IdentityMismatch { reason: String }, + + /// A manifest field needed to identify or reproduce a capture is empty. + #[error("checkpoint {checkpoint_id} has invalid {field}: {reason}")] + InvalidField { + checkpoint_id: String, + field: &'static str, + reason: String, + }, + + /// The artifact manifest is incomplete, duplicated, or malformed. + #[error("checkpoint {checkpoint_id} has an invalid artifact manifest: {reason}")] + InvalidArtifacts { + checkpoint_id: String, + reason: String, + }, + + /// A requested artifact name is outside the frozen format. + #[error("artifact name {name:?} is not part of the checkpoint format")] + InvalidArtifactName { name: String }, +} + +/// Validate a canonical `ckpt-` identifier. +pub fn validate_checkpoint_id(checkpoint_id: &str) -> Result { + let raw = checkpoint_id + .strip_prefix("ckpt-") + .ok_or_else(|| invalid_identifier(checkpoint_id, "missing ckpt- prefix"))?; + let uuid = Uuid::parse_str(raw) + .map_err(|error| invalid_identifier(checkpoint_id, error.to_string()))?; + if checkpoint_id != format!("ckpt-{uuid}") { + return Err(invalid_identifier( + checkpoint_id, + "identifier is not in canonical hyphenated lowercase form", + )); + } + Ok(uuid) +} + +/// Validate a name before the daemon resolves it inside a checkpoint stage. +pub fn validate_artifact_name(name: &str) -> Result<(), CheckpointValidationError> { + if REQUIRED_ARTIFACTS.contains(&name) { + Ok(()) + } else { + Err(CheckpointValidationError::InvalidArtifactName { + name: name.to_string(), + }) + } +} + +/// Validate daemon-supplied values before constructing a durable manifest. +pub fn validate_commit_checkpoint( + checkpoint_id: &str, + input: &CommitCheckpoint, +) -> Result<(), CheckpointValidationError> { + validate_checkpoint_id(checkpoint_id)?; + if let Some(parent) = &input.parent { + validate_checkpoint_id(parent)?; + if parent == checkpoint_id { + return Err(CheckpointValidationError::InvalidField { + checkpoint_id: checkpoint_id.to_string(), + field: "parent", + reason: "a checkpoint cannot be its own parent".to_string(), + }); + } + } + validate_runtime_identity( + checkpoint_id, + &input.policy_name, + &input.image_digest, + input.backend_version.as_deref(), + ) +} + +/// Validate a parsed manifest against the catalog location that contained it. +/// +/// Artifact content hashes are intentionally not checked here: reading files +/// belongs to the daemon. This function only validates the pure record. +pub fn validate_checkpoint_manifest( + metadata: &CheckpointMetadata, + expected_sandbox_id: Uuid, + expected_checkpoint_id: &str, +) -> Result<(), CheckpointValidationError> { + validate_checkpoint_id(expected_checkpoint_id)?; + validate_checkpoint_id(&metadata.id)?; + if metadata.format_version != CHECKPOINT_FORMAT_VERSION { + return Err(CheckpointValidationError::UnsupportedFormat { + checkpoint_id: metadata.id.clone(), + actual: metadata.format_version, + expected: CHECKPOINT_FORMAT_VERSION, + }); + } + if metadata.id != expected_checkpoint_id { + return Err(CheckpointValidationError::IdentityMismatch { + reason: format!( + "manifest id {:?} does not match catalog id {expected_checkpoint_id:?}", + metadata.id + ), + }); + } + if metadata.sandbox_id != expected_sandbox_id { + return Err(CheckpointValidationError::IdentityMismatch { + reason: format!( + "manifest sandbox {} does not match catalog sandbox {expected_sandbox_id}", + metadata.sandbox_id + ), + }); + } + if let Some(parent) = &metadata.parent { + validate_checkpoint_id(parent)?; + if parent == &metadata.id { + return Err(CheckpointValidationError::InvalidField { + checkpoint_id: metadata.id.clone(), + field: "parent", + reason: "a checkpoint cannot be its own parent".to_string(), + }); + } + } + validate_runtime_identity( + &metadata.id, + &metadata.policy_name, + &metadata.image_digest, + metadata.backend_version.as_deref(), + )?; + validate_artifact_manifest(&metadata.id, &metadata.artifacts) +} + +fn validate_runtime_identity( + checkpoint_id: &str, + policy_name: &str, + image_digest: &str, + backend_version: Option<&str>, +) -> Result<(), CheckpointValidationError> { + if policy_name.trim().is_empty() { + return Err(CheckpointValidationError::InvalidField { + checkpoint_id: checkpoint_id.to_string(), + field: "policy_name", + reason: "value is empty".to_string(), + }); + } + if image_digest.trim().is_empty() { + return Err(CheckpointValidationError::InvalidField { + checkpoint_id: checkpoint_id.to_string(), + field: "image_digest", + reason: "value is empty".to_string(), + }); + } + if backend_version.is_some_and(|version| version.trim().is_empty()) { + return Err(CheckpointValidationError::InvalidField { + checkpoint_id: checkpoint_id.to_string(), + field: "backend_version", + reason: "present version is empty".to_string(), + }); + } + Ok(()) +} + +fn validate_artifact_manifest( + checkpoint_id: &str, + artifacts: &[CheckpointArtifact], +) -> Result<(), CheckpointValidationError> { + if artifacts.len() != REQUIRED_ARTIFACTS.len() { + return Err(invalid_artifacts( + checkpoint_id, + format!( + "expected {} artifacts, found {}", + REQUIRED_ARTIFACTS.len(), + artifacts.len() + ), + )); + } + + let mut names = HashSet::with_capacity(artifacts.len()); + for artifact in artifacts { + validate_artifact_name(&artifact.name).map_err(|_| { + invalid_artifacts( + checkpoint_id, + format!("unexpected artifact {:?}", artifact.name), + ) + })?; + if !names.insert(artifact.name.as_str()) { + return Err(invalid_artifacts( + checkpoint_id, + format!("duplicate artifact {:?}", artifact.name), + )); + } + if artifact.sha256.len() != 64 + || !artifact + .sha256 + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) + { + return Err(invalid_artifacts( + checkpoint_id, + format!("artifact {:?} has an invalid SHA-256 digest", artifact.name), + )); + } + } + if REQUIRED_ARTIFACTS + .iter() + .any(|required| !names.contains(required)) + { + return Err(invalid_artifacts( + checkpoint_id, + "one or more required artifacts are missing", + )); + } + Ok(()) +} + +fn invalid_identifier(checkpoint_id: &str, reason: impl Into) -> CheckpointValidationError { + CheckpointValidationError::InvalidIdentifier { + checkpoint_id: checkpoint_id.to_string(), + reason: reason.into(), + } +} + +fn invalid_artifacts(checkpoint_id: &str, reason: impl Into) -> CheckpointValidationError { + CheckpointValidationError::InvalidArtifacts { + checkpoint_id: checkpoint_id.to_string(), + reason: reason.into(), + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn artifact(name: &str, fill: char) -> CheckpointArtifact { + CheckpointArtifact { + name: name.to_string(), + size_bytes: 10, + sha256: std::iter::repeat_n(fill, 64).collect(), + } + } + + fn metadata() -> CheckpointMetadata { + let sandbox_id = Uuid::new_v4(); + CheckpointMetadata { + format_version: CHECKPOINT_FORMAT_VERSION, + id: format!("ckpt-{}", Uuid::new_v4()), + parent: None, + sandbox_id, + policy_name: "default".to_string(), + image_digest: "sha256:image".to_string(), + backend: BackendKind::Mock, + backend_version: Some("mock-v1".to_string()), + created_at: Utc::now(), + snapshot_kind: SnapshotKind::Full, + artifacts: vec![ + artifact("vmstate.snap", 'a'), + artifact("memory.snap", 'b'), + artifact("rootfs.snap", 'c'), + ], + } + } + + #[test] + fn canonical_identifier_round_trips() { + let uuid = Uuid::new_v4(); + assert_eq!( + validate_checkpoint_id(&format!("ckpt-{uuid}")).expect("valid identifier"), + uuid + ); + } + + #[test] + fn noncanonical_identifier_is_rejected() { + let uuid = Uuid::new_v4(); + assert!( + validate_checkpoint_id(&format!("ckpt-{}", uuid.to_string().to_uppercase())).is_err() + ); + assert!(validate_checkpoint_id(&format!("ckpt-{}", uuid.simple())).is_err()); + assert!(validate_checkpoint_id("../checkpoint").is_err()); + } + + #[test] + fn valid_manifest_passes_pure_validation() { + let metadata = metadata(); + validate_checkpoint_manifest(&metadata, metadata.sandbox_id, &metadata.id) + .expect("valid manifest"); + } + + #[test] + fn manifest_identity_must_match_catalog_location() { + let metadata = metadata(); + let error = validate_checkpoint_manifest(&metadata, Uuid::new_v4(), &metadata.id) + .expect_err("sandbox mismatch must fail"); + assert!(matches!( + error, + CheckpointValidationError::IdentityMismatch { .. } + )); + } + + #[test] + fn manifest_requires_the_exact_artifact_set() { + let mut metadata = metadata(); + metadata.artifacts[2].name = "memory.snap".to_string(); + let error = validate_checkpoint_manifest(&metadata, metadata.sandbox_id, &metadata.id) + .expect_err("duplicate artifact must fail"); + assert!(matches!( + error, + CheckpointValidationError::InvalidArtifacts { .. } + )); + } + + #[test] + fn manifest_rejects_noncanonical_digest() { + let mut metadata = metadata(); + metadata.artifacts[0].sha256 = "A".repeat(64); + assert!( + validate_checkpoint_manifest(&metadata, metadata.sandbox_id, &metadata.id).is_err() + ); + } + + #[test] + fn commit_input_rejects_self_parent_and_empty_identity() { + let id = format!("ckpt-{}", Uuid::new_v4()); + let input = CommitCheckpoint { + parent: Some(id.clone()), + policy_name: String::new(), + image_digest: String::new(), + backend: BackendKind::Mock, + backend_version: None, + snapshot_kind: SnapshotKind::Full, + }; + assert!(validate_commit_checkpoint(&id, &input).is_err()); + } +} diff --git a/src/blaze/crates/blaze-core/src/error.rs b/src/blaze/crates/blaze-core/src/error.rs index 5f08bcd2d..aa22cf197 100644 --- a/src/blaze/crates/blaze-core/src/error.rs +++ b/src/blaze/crates/blaze-core/src/error.rs @@ -29,6 +29,10 @@ pub enum BlazeError { #[error("invalid sandbox state transition: {from} -> {to}")] InvalidStateTransition { from: String, to: String }, + /// A lifecycle caller tried to replace an unfinished durable operation. + #[error("sandbox operation already in progress: active={active}, requested={requested}")] + OperationInProgress { active: String, requested: String }, + #[error("hook '{hook_name}' error: {msg}")] HookError { hook_name: String, msg: String }, diff --git a/src/blaze/crates/blaze-core/src/lib.rs b/src/blaze/crates/blaze-core/src/lib.rs index cc6707d7d..0a7b9e622 100644 --- a/src/blaze/crates/blaze-core/src/lib.rs +++ b/src/blaze/crates/blaze-core/src/lib.rs @@ -9,12 +9,14 @@ //! - [`config`]: daemon TOML configuration //! - [`policy`]: workload class + policy file schema //! - [`backend`]: backend kinds + selection / fallback +//! - [`checkpoint`]: pure checkpoint records and manifest validation //! - [`guest_protocol`]: guest-agent wire DTOs //! - [`lifecycle`]: sandbox state machine + JSON persistence //! - [`kernel`]: kernel hook registry, per-hook mutex //! - [`error`]: unified [`BlazeError`] error enum pub mod backend; +pub mod checkpoint; pub mod config; pub mod error; pub mod guest_protocol; diff --git a/src/blaze/crates/blaze-core/src/lifecycle.rs b/src/blaze/crates/blaze-core/src/lifecycle.rs index ec0b81c05..eb2111883 100644 --- a/src/blaze/crates/blaze-core/src/lifecycle.rs +++ b/src/blaze/crates/blaze-core/src/lifecycle.rs @@ -50,10 +50,66 @@ impl SandboxState { pub enum OperationKind { /// Sandbox creation is acquiring resources or starting a backend. Create, + /// A point-in-time checkpoint is being captured and published. + Checkpoint, /// Runtime resources are being destroyed. Destroy, } +impl OperationKind { + const fn as_str(&self) -> &'static str { + match self { + OperationKind::Create => "create", + OperationKind::Checkpoint => "checkpoint", + OperationKind::Destroy => "destroy", + } + } +} + +impl std::fmt::Display for OperationKind { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str(self.as_str()) + } +} + +/// Durable boundary reached by a multi-step lifecycle operation. +/// +/// The journal keeps this separate from [`SandboxState`]: state describes +/// externally visible runtime availability, while the phase identifies which +/// checkpoint resources may already have been published after interruption. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "kebab-case")] +pub enum OperationPhase { + /// A staging directory exists, but the backend has not been paused. + CheckpointPreparing, + /// The backend is paused while snapshot artifacts are being written. + CheckpointPaused, + /// A complete checkpoint directory is visible, but HEAD is unchanged. + CheckpointPublished, + /// HEAD references the checkpoint; runtime resume is not yet committed. + CheckpointHeadUpdated, +} + +impl OperationPhase { + const fn as_str(self) -> &'static str { + match self { + OperationPhase::CheckpointPreparing => "checkpoint-preparing", + OperationPhase::CheckpointPaused => "checkpoint-paused", + OperationPhase::CheckpointPublished => "checkpoint-published", + OperationPhase::CheckpointHeadUpdated => "checkpoint-head-updated", + } + } + + const fn rank(self) -> u8 { + match self { + OperationPhase::CheckpointPreparing => 0, + OperationPhase::CheckpointPaused => 1, + OperationPhase::CheckpointPublished => 2, + OperationPhase::CheckpointHeadUpdated => 3, + } + } +} + /// Durable journal entry for one active lifecycle operation. #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] pub struct OperationJournal { @@ -61,6 +117,12 @@ pub struct OperationJournal { pub kind: OperationKind, /// UTC time at which the operation became externally visible. pub started_at: DateTime, + /// Checkpoint selected by this operation, when applicable. + #[serde(default)] + pub checkpoint_id: Option, + /// Last durably committed operation boundary. + #[serde(default)] + pub phase: Option, } impl std::fmt::Display for SandboxState { @@ -114,6 +176,9 @@ pub struct SandboxInstance { /// Active multi-step operation, if any. #[serde(default)] pub operation: Option, + /// Last checkpoint whose capture completed and returned the sandbox to running. + #[serde(default)] + pub last_checkpoint: Option, } impl SandboxInstance { @@ -137,6 +202,7 @@ impl SandboxInstance { policy_name, backend_ownership: BackendOwnership::NotStarted, operation: None, + last_checkpoint: None, } } @@ -145,8 +211,57 @@ impl SandboxInstance { self.operation = Some(OperationJournal { kind, started_at: Utc::now(), + checkpoint_id: None, + phase: None, + }); + self.updated_at = Utc::now(); + } + + /// Record checkpoint intent before pausing the backend. + pub fn begin_checkpoint_operation(&mut self, checkpoint_id: String) -> Result<()> { + if let Some(active) = &self.operation { + return Err(BlazeError::OperationInProgress { + active: active.kind.to_string(), + requested: OperationKind::Checkpoint.to_string(), + }); + } + let now = Utc::now(); + self.operation = Some(OperationJournal { + kind: OperationKind::Checkpoint, + started_at: now, + checkpoint_id: Some(checkpoint_id), + phase: Some(OperationPhase::CheckpointPreparing), }); + self.updated_at = now; + Ok(()) + } + + /// Advance the active checkpoint journal without replacing its identity. + pub fn advance_checkpoint_phase(&mut self, phase: OperationPhase) -> Result<()> { + let operation = self + .operation + .as_mut() + .ok_or_else(|| BlazeError::OperationInProgress { + active: "none".to_string(), + requested: OperationKind::Checkpoint.to_string(), + })?; + if operation.kind != OperationKind::Checkpoint { + return Err(BlazeError::OperationInProgress { + active: operation.kind.to_string(), + requested: OperationKind::Checkpoint.to_string(), + }); + } + if let Some(current) = operation.phase + && phase.rank() < current.rank() + { + return Err(BlazeError::InvalidStateTransition { + from: current.as_str().to_string(), + to: phase.as_str().to_string(), + }); + } + operation.phase = Some(phase); self.updated_at = Utc::now(); + Ok(()) } /// Clear the marker before atomically persisting the final state. @@ -224,6 +339,7 @@ fn is_valid_transition(from: SandboxState, to: SandboxState) -> bool { (Running, Paused) => true, (Paused, Checkpointed) => true, (Paused, Running) => true, // resume + (Checkpointed, Running) => true, _ => false, } } @@ -249,6 +365,7 @@ mod tests { SandboxState::Running, SandboxState::Paused, SandboxState::Checkpointed, + SandboxState::Running, SandboxState::Destroyed, ] { inst.transition(target).expect("legal transition"); @@ -337,6 +454,7 @@ mod tests { }); let loaded: SandboxInstance = serde_json::from_value(value).expect("legacy state"); assert!(loaded.operation.is_none()); + assert!(loaded.last_checkpoint.is_none()); assert_eq!(loaded.backend_ownership, BackendOwnership::Unknown); } @@ -376,4 +494,68 @@ mod tests { loaded.finish_operation(); assert!(loaded.operation.is_none()); } + + #[test] + fn checkpoint_journal_preserves_identity_and_phase() { + let tmp = tempfile::tempdir().expect("tmp"); + let mut instance = fresh(); + instance + .begin_checkpoint_operation("ckpt-00000000-0000-0000-0000-000000000001".into()) + .expect("begin checkpoint"); + instance + .advance_checkpoint_phase(OperationPhase::CheckpointPublished) + .expect("advance checkpoint"); + instance.persist(tmp.path()).expect("persist"); + + let loaded = SandboxInstance::load(tmp.path(), instance.id).expect("load"); + let journal = loaded.operation.expect("checkpoint journal"); + assert_eq!(journal.kind, OperationKind::Checkpoint); + assert_eq!( + journal.checkpoint_id.as_deref(), + Some("ckpt-00000000-0000-0000-0000-000000000001") + ); + assert_eq!(journal.phase, Some(OperationPhase::CheckpointPublished)); + } + + #[test] + fn checkpoint_journal_rejects_phase_regression() { + let mut instance = fresh(); + instance + .begin_checkpoint_operation("ckpt-00000000-0000-0000-0000-000000000001".into()) + .expect("begin checkpoint"); + instance + .advance_checkpoint_phase(OperationPhase::CheckpointPublished) + .expect("advance checkpoint"); + + let error = instance + .advance_checkpoint_phase(OperationPhase::CheckpointPaused) + .expect_err("checkpoint phase must remain a durable lower bound"); + + assert!(matches!(error, BlazeError::InvalidStateTransition { .. })); + assert_eq!( + instance + .operation + .as_ref() + .and_then(|journal| journal.phase), + Some(OperationPhase::CheckpointPublished) + ); + } + + #[test] + fn checkpoint_journal_cannot_replace_an_active_operation() { + let mut instance = fresh(); + instance.begin_operation(OperationKind::Create); + let journal = instance.operation.clone(); + + let error = instance + .begin_checkpoint_operation("ckpt-00000000-0000-0000-0000-000000000001".into()) + .expect_err("checkpoint must not replace create"); + + assert!(matches!( + error, + BlazeError::OperationInProgress { active, requested } + if active == "create" && requested == "checkpoint" + )); + assert_eq!(instance.operation, journal); + } } diff --git a/src/blaze/crates/blaze-core/src/storage.rs b/src/blaze/crates/blaze-core/src/storage.rs index 1e2d96f44..526b0a0b2 100644 --- a/src/blaze/crates/blaze-core/src/storage.rs +++ b/src/blaze/crates/blaze-core/src/storage.rs @@ -5,7 +5,7 @@ //! (copy-on-write, content-addressable dedup) but present //! a uniform interface to the daemon layer. -use std::path::PathBuf; +use std::path::{Path, PathBuf}; use async_trait::async_trait; use thiserror::Error; @@ -136,6 +136,22 @@ pub trait StorageProvider: Send + Sync { /// synchronization or cleanup must remain safe after completion. async fn sync_artifacts(&self, slot: &StorageSlot) -> Result<()>; + /// Report whether this provider can capture a self-contained checkpoint. + /// + /// The default is conservative so existing providers do not advertise a + /// data path they have not implemented. + fn supports_checkpoint_capture(&self) -> bool { + false + } + + /// Capture the slot's writable root filesystem at `target`. + async fn capture_checkpoint(&self, slot: &StorageSlot, target: &Path) -> Result<()> { + let _ = (slot, target); + Err(BlazeError::StorageError { + msg: "storage provider does not support checkpoint capture".to_string(), + }) + } + /// Return the provider's current storage capacity. fn pool_status(&self) -> PoolStatus; } diff --git a/src/blaze/crates/blazed/Cargo.toml b/src/blaze/crates/blazed/Cargo.toml index 21e32bd70..beb57b723 100644 --- a/src/blaze/crates/blazed/Cargo.toml +++ b/src/blaze/crates/blazed/Cargo.toml @@ -37,6 +37,7 @@ tracing-subscriber = { workspace = true } uuid = { workspace = true } chrono = { workspace = true } base64 = { workspace = true } +sha2 = { workspace = true } [dev-dependencies] tempfile = "3" diff --git a/src/blaze/crates/blazed/src/api.rs b/src/blaze/crates/blazed/src/api.rs index a928add4d..564873d02 100644 --- a/src/blaze/crates/blazed/src/api.rs +++ b/src/blaze/crates/blazed/src/api.rs @@ -13,6 +13,7 @@ use std::sync::Arc; use base64::Engine; use base64::engine::general_purpose::STANDARD as BASE64; use blaze_core::backend::{BackendKind, BackendStatus, select_backend}; +use blaze_core::checkpoint::CheckpointMetadata; use blaze_core::lifecycle::{SandboxInstance, StartPath}; use blaze_core::policy::{ImageMetadata, RuntimeDecision, WorkloadClass}; use http_body_util::{BodyExt, Full}; @@ -139,6 +140,8 @@ async fn dispatch( ("POST", ["v1", "sandboxes", id, "exec"]) => exec_sandbox(state, id, &body).await, ("POST", ["v1", "sandboxes", id, "read"]) => read_sandbox_file(state, id, &body).await, ("POST", ["v1", "sandboxes", id, "write"]) => write_sandbox_file(state, id, &body).await, + ("POST", ["v1", "sandboxes", id, "checkpoint"]) => checkpoint(state, id).await, + ("GET", ["v1", "sandboxes", id, "checkpoints"]) => list_checkpoints(state, id).await, ("DELETE", ["v1", "sandboxes", id]) => destroy_sandbox(state, id).await, ("GET", ["v1", "pools"]) | ("GET", ["v1", "pools", _, _]) @@ -219,6 +222,14 @@ struct CreateInstanceResp { selected_backend: BackendKind, } +#[derive(Debug, Serialize)] +struct CheckpointResp { + checkpoint_id: String, + instance_id: Uuid, + #[serde(flatten)] + checkpoint: CheckpointMetadata, +} + fn list_sandboxes(state: &Arc) -> Result>> { json_ok(&state.manager.list()?) } @@ -315,6 +326,20 @@ async fn create_sandbox(state: &Arc, body: &[u8]) -> Result, id: &str) -> Result>> { + let uuid = parse_uuid(id)?; + let checkpoint = state.manager.checkpoint(uuid).await?; + json_ok(&CheckpointResp { + checkpoint_id: checkpoint.id.clone(), + instance_id: checkpoint.sandbox_id, + checkpoint, + }) +} + +async fn list_checkpoints(state: &Arc, id: &str) -> Result>> { + json_ok(&state.manager.list_checkpoints(parse_uuid(id)?).await?) +} + async fn destroy_sandbox(state: &Arc, id: &str) -> Result>> { let uuid = parse_uuid(id)?; state.manager.destroy(uuid).await?; @@ -559,12 +584,20 @@ mod tests { use std::path::{Path, PathBuf}; use std::sync::Arc; use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; + #[cfg(feature = "test-failpoints")] + use std::time::Duration; use async_trait::async_trait; use blaze_core::BlazeError; use blaze_core::backend::BackendKind; + #[cfg(feature = "test-failpoints")] + use blaze_core::backend::SnapshotKind; + #[cfg(feature = "test-failpoints")] + use blaze_core::checkpoint::CommitCheckpoint; use blaze_core::config::DaemonConfig; use blaze_core::kernel::HookRegistry; + #[cfg(feature = "test-failpoints")] + use blaze_core::lifecycle::OperationPhase; use blaze_core::lifecycle::{BackendOwnership, OperationKind, SandboxState}; use blaze_core::policy::{ BackendConfigs, FallbackOnMissingHook, PolicyEngine, PolicyFile, PolicyHooks, PolicyMatch, @@ -574,6 +607,8 @@ mod tests { AcquireOpts, PoolStatus, StorageAcquireError, StorageProvider, StorageSlot, }; + #[cfg(feature = "test-failpoints")] + use crate::checkpoint_store::CheckpointStore; use crate::file_provider::FileStorageProvider; #[cfg(target_os = "linux")] use crate::spawner::BubblewrapSpawner; @@ -638,6 +673,16 @@ mod tests { .expect("request") } + fn configured_state_dir(state: &ServerState) -> PathBuf { + state + .config + .lock() + .expect("config") + .daemon + .state_dir + .clone() + } + fn build_test_state( config: DaemonConfig, policy: PolicyFile, @@ -660,7 +705,11 @@ mod tests { #[cfg(feature = "test-failpoints")] fn mock_state(temp: &tempfile::TempDir) -> Arc { - let config = test_config(temp); + mock_state_from_config(test_config(temp)) + } + + #[cfg(feature = "test-failpoints")] + fn mock_state_from_config(config: DaemonConfig) -> Arc { let storage: Arc = Arc::new(FileStorageProvider::with_images( config.storage.images_dir.clone(), config.storage.instances_dir.clone(), @@ -703,6 +752,174 @@ mod tests { .expect("created json") } + async fn write_checkpoint_fixture(state: &Arc, id: &str) -> StorageSlot { + let slot = state.storage.reconstruct(id).await.expect("storage slot"); + tokio::fs::write(&slot.rootfs_path, b"checkpoint-rootfs") + .await + .expect("rootfs"); + slot + } + + #[cfg(feature = "test-failpoints")] + async fn cancel_checkpoint_request_at( + state: &Arc, + id: Uuid, + failpoint: &'static str, + expected_state: SandboxState, + expected_phase: OperationPhase, + ) -> String { + let hook = crate::failpoint::TestFailpoint::new(&[failpoint]); + let capture_state = state.clone(); + let capture_hook = hook.clone(); + let capture = + tokio::spawn( + async move { capture_hook.run(capture_state.manager.checkpoint(id)).await }, + ); + hook.wait_until_paused().await; + let interrupted = state.manager.get(id).expect("interrupted lifecycle"); + let lock_was_retained = state.manager.operation_lock(id).try_lock().is_err(); + capture.abort(); + let cancelled = capture + .await + .expect_err("checkpoint task must be cancelled"); + hook.release(); + + assert!(cancelled.is_cancelled()); + assert_eq!(interrupted.state, expected_state); + assert_eq!( + interrupted.operation.and_then(|journal| journal.phase), + Some(expected_phase) + ); + assert!( + lock_was_retained, + "the detached supervisor must retain checkpoint ownership" + ); + + tokio::time::timeout(Duration::from_secs(2), async { + loop { + let lifecycle = state.manager.get(id).expect("checkpoint lifecycle"); + if lifecycle.state == SandboxState::Running + && lifecycle.operation.is_none() + && lifecycle.last_checkpoint.is_some() + && state.manager.operation_lock(id).try_lock().is_ok() + { + return lifecycle.last_checkpoint.expect("completed checkpoint"); + } + tokio::task::yield_now().await; + } + }) + .await + .expect("detached checkpoint supervisor must converge") + } + + #[cfg(feature = "test-failpoints")] + async fn persist_crashed_checkpoint_phase( + state: &Arc, + id: Uuid, + phase: OperationPhase, + ) -> String { + let store = CheckpointStore::new(state.state_store.clone()); + let stage = store.begin(id).expect("checkpoint stage"); + let checkpoint_id = stage.id().to_string(); + let mut instance = state.manager.get(id).expect("running lifecycle"); + instance + .begin_checkpoint_operation(checkpoint_id.clone()) + .expect("checkpoint journal"); + if !matches!(phase, OperationPhase::CheckpointPreparing) { + instance + .transition(SandboxState::Paused) + .expect("paused lifecycle"); + instance + .advance_checkpoint_phase(OperationPhase::CheckpointPaused) + .expect("paused journal"); + } + if matches!( + phase, + OperationPhase::CheckpointPublished | OperationPhase::CheckpointHeadUpdated + ) { + for (name, contents) in [ + ("vmstate.snap", b"crashed-vmstate".as_slice()), + ("memory.snap", b"crashed-memory".as_slice()), + ("rootfs.snap", b"crashed-rootfs".as_slice()), + ] { + std::fs::write(stage.artifact_path(name).expect("artifact path"), contents) + .expect("checkpoint artifact"); + } + store + .publish( + &stage, + CommitCheckpoint { + parent: None, + policy_name: instance.policy_name.clone(), + image_digest: instance.image_digest.clone(), + backend: instance.backend, + backend_version: Some("mock-v1".to_string()), + snapshot_kind: SnapshotKind::Full, + }, + ) + .expect("published checkpoint"); + instance + .advance_checkpoint_phase(OperationPhase::CheckpointPublished) + .expect("published journal"); + } + if phase == OperationPhase::CheckpointHeadUpdated { + store.set_head(id, &checkpoint_id).expect("checkpoint HEAD"); + instance + .advance_checkpoint_phase(OperationPhase::CheckpointHeadUpdated) + .expect("HEAD-updated journal"); + } + state + .state_store + .persist(&instance) + .expect("persist crashed checkpoint phase"); + state + .manager + .backend_owner(id) + .expect("backend owner") + .kill() + .await + .expect("stop process owned by crashed daemon"); + checkpoint_id + } + + struct NoCheckpointStorage { + inner: FileStorageProvider, + } + + #[async_trait] + impl StorageProvider for NoCheckpointStorage { + async fn probe(&self) -> blaze_core::Result { + self.inner.probe().await + } + + async fn acquire( + &self, + opts: &AcquireOpts, + ) -> std::result::Result { + self.inner.acquire(opts).await + } + + async fn release(&self, slot: StorageSlot) -> blaze_core::Result<()> { + self.inner.release(slot).await + } + + async fn release_by_id(&self, instance_id: &str) -> blaze_core::Result<()> { + self.inner.release_by_id(instance_id).await + } + + async fn reconstruct(&self, instance_id: &str) -> blaze_core::Result { + self.inner.reconstruct(instance_id).await + } + + async fn sync_artifacts(&self, slot: &StorageSlot) -> blaze_core::Result<()> { + self.inner.sync_artifacts(slot).await + } + + fn pool_status(&self) -> PoolStatus { + self.inner.pool_status() + } + } + async fn dispatched_json( state: &Arc, method: Method, @@ -1296,7 +1513,6 @@ mod tests { let routes = [ (Method::POST, format!("/v1/sandboxes/{id}/reset")), - (Method::POST, format!("/v1/sandboxes/{id}/checkpoint")), (Method::POST, format!("/v1/sandboxes/{id}/destroy")), ]; @@ -1428,6 +1644,1906 @@ mod tests { let _ = std::fs::remove_dir_all(&tmp); } + #[tokio::test] + async fn checkpoint_rejects_unsupported_storage_without_mutation() { + let temp = tempfile::tempdir().expect("temp"); + let config = test_config(&temp); + let storage: Arc = Arc::new(NoCheckpointStorage { + inner: FileStorageProvider::with_images( + config.storage.images_dir.clone(), + config.storage.instances_dir.clone(), + ), + }); + let state = build_test_state( + config, + test_policy(BackendKind::Mock), + spawners(BackendKind::Mock, Arc::new(MockSpawner)), + BackendKind::Mock, + storage, + ); + let request = test_request(); + let created = created_json(&state, &request).await; + let id = created["instance"]["id"].as_str().expect("id"); + let uuid = Uuid::parse_str(id).expect("uuid"); + let state_path = configured_state_dir(&state).join(id).join("state.json"); + let persisted_before = std::fs::read(&state_path).expect("persisted state"); + + let error = checkpoint(&state, id) + .await + .expect_err("checkpoint without backend and storage capture must fail closed"); + + assert!(matches!(error, BlazeDaemonError::UnsupportedOperation(_))); + assert_eq!(error.status_code(), 501); + assert_eq!( + state.instances.lock().expect("instances")[&uuid].state, + SandboxState::Running + ); + assert!( + state.instances.lock().expect("instances")[&uuid] + .operation + .is_none() + ); + assert_eq!( + std::fs::read(state_path).expect("persisted state"), + persisted_before + ); + assert!( + !configured_state_dir(&state) + .join("checkpoints") + .join(id) + .exists() + ); + assert!(state.manager.backend_owner(uuid).is_some()); + } + + #[tokio::test] + async fn checkpoint_rejects_unsupported_backend_without_mutation() { + let temp = tempfile::tempdir().expect("temp"); + let config = test_config(&temp); + let storage: Arc = Arc::new(FileStorageProvider::with_images( + config.storage.images_dir.clone(), + config.storage.instances_dir.clone(), + )); + let kill_count = Arc::new(AtomicUsize::new(0)); + let state = build_test_state( + config, + test_policy(BackendKind::Mock), + spawners( + BackendKind::Mock, + Arc::new(CountingSpawner { + kill_count: kill_count.clone(), + orphan_cleanup_count: Arc::new(AtomicUsize::new(0)), + }), + ), + BackendKind::Mock, + storage, + ); + let created = created_json(&state, &test_request()).await; + let id = created["instance"]["id"].as_str().expect("id"); + let uuid = Uuid::parse_str(id).expect("uuid"); + let state_path = configured_state_dir(&state).join(id).join("state.json"); + let persisted_before = std::fs::read(&state_path).expect("persisted state"); + + let error = checkpoint(&state, id) + .await + .expect_err("checkpoint without backend capture must fail closed"); + + assert!(matches!(error, BlazeDaemonError::UnsupportedOperation(_))); + assert_eq!(error.status_code(), 501); + let lifecycle = state.manager.get(uuid).expect("lifecycle"); + assert_eq!(lifecycle.state, SandboxState::Running); + assert!(lifecycle.operation.is_none()); + assert_eq!( + std::fs::read(state_path).expect("persisted state"), + persisted_before + ); + assert!( + !configured_state_dir(&state) + .join("checkpoints") + .join(id) + .exists() + ); + assert_eq!(kill_count.load(Ordering::Acquire), 0); + assert!(state.manager.backend_owner(uuid).is_some()); + } + + #[tokio::test] + async fn checkpoint_routes_capture_and_list_live_state() { + let temp = tempfile::tempdir().expect("temp"); + let config = test_config(&temp); + let storage: Arc = Arc::new(FileStorageProvider::with_images( + config.storage.images_dir.clone(), + config.storage.instances_dir.clone(), + )); + let state = build_test_state( + config, + test_policy(BackendKind::Mock), + spawners(BackendKind::Mock, Arc::new(MockSpawner)), + BackendKind::Mock, + storage, + ); + let created = created_json(&state, &test_request()).await; + let id = created["instance"]["id"].as_str().expect("id"); + let uuid = Uuid::parse_str(id).expect("uuid"); + let slot = write_checkpoint_fixture(&state, id).await; + + let (status, checkpoint) = dispatched_json( + &state, + Method::POST, + &format!("/v1/sandboxes/{id}/checkpoint"), + Vec::new(), + ) + .await; + assert_eq!(status, StatusCode::OK); + let checkpoint_id = checkpoint["id"].as_str().expect("checkpoint id"); + assert_eq!(checkpoint["checkpoint_id"], checkpoint["id"]); + assert_eq!(checkpoint["instance_id"], id); + assert_eq!(checkpoint["snapshot_kind"], "full"); + assert_eq!(checkpoint["sandbox_id"], id); + let captured_rootfs = configured_state_dir(&state) + .join("checkpoints") + .join(id) + .join(checkpoint_id) + .join("rootfs.snap"); + assert_eq!( + tokio::fs::read(&captured_rootfs) + .await + .expect("captured rootfs"), + b"checkpoint-rootfs" + ); + + tokio::fs::write(&slot.rootfs_path, b"changed-after-checkpoint") + .await + .expect("mutate live rootfs"); + assert_eq!( + tokio::fs::read(&captured_rootfs) + .await + .expect("independent captured rootfs"), + b"checkpoint-rootfs" + ); + let (status, checkpoints) = dispatched_json( + &state, + Method::GET, + &format!("/v1/sandboxes/{id}/checkpoints"), + Vec::new(), + ) + .await; + assert_eq!(status, StatusCode::OK); + assert_eq!(checkpoints.as_array().expect("checkpoint list").len(), 1); + assert_eq!(checkpoints[0]["id"], checkpoint_id); + assert_eq!(checkpoints[0]["is_head"], true); + assert_eq!(checkpoints[0]["on_head_chain"], true); + + let lifecycle = state.manager.get(uuid).expect("lifecycle"); + assert_eq!(lifecycle.state, SandboxState::Running); + assert!(lifecycle.operation.is_none()); + assert_eq!(lifecycle.last_checkpoint.as_deref(), Some(checkpoint_id)); + assert!(state.manager.backend_owner(uuid).is_some()); + + state.manager.destroy(uuid).await.expect("destroy sandbox"); + assert!( + state + .manager + .list_checkpoints(uuid) + .await + .expect("removed checkpoint history") + .is_empty() + ); + assert!( + !configured_state_dir(&state) + .join("checkpoints") + .join(id) + .exists(), + "destroy must remove the complete checkpoint namespace" + ); + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn checkpoint_cleanup_failure_keeps_destroy_recoverable() { + let temp = tempfile::tempdir().expect("temp"); + let state = mock_state(&temp); + let created = created_json(&state, &test_request()).await; + let id = created["instance"]["id"].as_str().expect("id"); + let uuid = Uuid::parse_str(id).expect("uuid"); + write_checkpoint_fixture(&state, id).await; + state + .manager + .checkpoint(uuid) + .await + .expect("seed checkpoint"); + let checkpoint_namespace = configured_state_dir(&state).join("checkpoints").join(id); + let hook = crate::failpoint::TestFailpoint::new(&[ + "checkpoint-store-sandbox-remove-before-unlink", + ]); + + let error = hook + .run(state.manager.destroy(uuid)) + .await + .expect_err("checkpoint namespace cleanup must fail"); + + assert!(matches!(error, BlazeDaemonError::RecoveryRequired(_))); + assert_eq!( + state.manager.get(uuid).expect("recovery lifecycle").state, + SandboxState::RecoveryRequired + ); + assert!(checkpoint_namespace.is_dir()); + assert_eq!( + std::fs::read_dir(&checkpoint_namespace) + .expect("retained checkpoint namespace") + .count(), + 0, + "partial cleanup must leave no committed checkpoint payload" + ); + + state.manager.destroy(uuid).await.expect("destroy retry"); + assert_eq!( + state.manager.get(uuid).expect("destroyed lifecycle").state, + SandboxState::Destroyed + ); + assert!(!checkpoint_namespace.exists()); + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn checkpoint_snapshot_failure_resumes_and_clears_the_journal() { + let temp = tempfile::tempdir().expect("temp"); + let state = mock_state(&temp); + let created = created_json(&state, &test_request()).await; + let id = created["instance"]["id"].as_str().expect("id"); + let uuid = Uuid::parse_str(id).expect("uuid"); + let hook = crate::failpoint::TestFailpoint::new(&["checkpoint-snapshot"]); + + let error = hook + .run(state.manager.checkpoint(uuid)) + .await + .expect_err("snapshot failure"); + + assert!(matches!( + error, + BlazeDaemonError::Core(BlazeError::BackendError { .. }) + )); + let lifecycle = state.manager.get(uuid).expect("lifecycle"); + assert_eq!(lifecycle.state, SandboxState::Running); + assert!(lifecycle.operation.is_none()); + assert_eq!( + state + .state_store + .load(uuid) + .expect("persisted lifecycle") + .operation, + None + ); + let checkpoint_dir = configured_state_dir(&state).join("checkpoints").join(id); + let staging = std::fs::read_dir(checkpoint_dir) + .expect("checkpoint directory") + .filter_map(std::result::Result::ok) + .filter(|entry| entry.file_name().to_string_lossy().starts_with(".ckpt-")) + .count(); + assert_eq!(staging, 0); + assert!(state.manager.backend_owner(uuid).is_some()); + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test(flavor = "current_thread")] + async fn checkpoint_compensation_cleanup_uses_the_blocking_pool() { + let temp = tempfile::tempdir().expect("temp"); + let state = mock_state(&temp); + let created = created_json(&state, &test_request()).await; + let id = created["instance"]["id"].as_str().expect("id"); + let uuid = Uuid::parse_str(id).expect("uuid"); + write_checkpoint_fixture(&state, id).await; + let hook = crate::failpoint::TestFailpoint::new(&[ + "checkpoint-rootfs-capture", + "checkpoint-before-stage-abort", + ]); + let guard_hook = hook.clone(); + let (guard_cancel, guard_cancelled) = std::sync::mpsc::channel(); + let release_guard = std::thread::spawn(move || { + if guard_cancelled + .recv_timeout(Duration::from_secs(1)) + .is_err() + { + guard_hook.release(); + } + }); + let started = std::time::Instant::now(); + let checkpoint_state = state.clone(); + let checkpoint_hook = hook.clone(); + let checkpoint = tokio::spawn(async move { + checkpoint_hook + .run(checkpoint_state.manager.checkpoint(uuid)) + .await + }); + + hook.wait_until_paused().await; + assert!( + started.elapsed() < Duration::from_millis(500), + "staging cleanup must not occupy the async runtime worker" + ); + tokio::time::timeout( + Duration::from_millis(250), + tokio::time::sleep(Duration::from_millis(1)), + ) + .await + .expect("the async runtime must remain responsive during staging cleanup"); + assert!( + state.manager.operation_lock(uuid).try_lock().is_err(), + "the sandbox operation lock must remain held during staging cleanup" + ); + + hook.release(); + guard_cancel.send(()).expect("cancel release guard"); + release_guard.join().expect("release guard"); + let error = checkpoint + .await + .expect("checkpoint task") + .expect_err("rootfs capture failure"); + assert!(matches!( + error, + BlazeDaemonError::Core(BlazeError::StorageError { .. }) + )); + let lifecycle = state.manager.get(uuid).expect("lifecycle"); + assert_eq!(lifecycle.state, SandboxState::Running); + assert!(lifecycle.operation.is_none()); + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn checkpoint_prepublication_failure_discards_the_stage() { + for failpoint in [ + "checkpoint-publish", + "checkpoint-store-publish-before-rename", + ] { + let temp = tempfile::tempdir().expect("temp"); + let state = mock_state(&temp); + let created = created_json(&state, &test_request()).await; + let id = created["instance"]["id"].as_str().expect("id"); + let uuid = Uuid::parse_str(id).expect("uuid"); + write_checkpoint_fixture(&state, id).await; + let hook = crate::failpoint::TestFailpoint::new(&[failpoint]); + + let error = hook + .run(state.manager.checkpoint(uuid)) + .await + .expect_err("publication must fail before the rename boundary"); + + assert!( + !matches!(error, BlazeDaemonError::RecoveryRequired(_)), + "{failpoint} must remain a compensated failure: {error}" + ); + let lifecycle = state.manager.get(uuid).expect("lifecycle"); + assert_eq!(lifecycle.state, SandboxState::Running); + assert!(lifecycle.operation.is_none()); + assert_eq!( + state + .state_store + .load(uuid) + .expect("persisted lifecycle") + .operation, + None + ); + assert!( + state + .manager + .list_checkpoints(uuid) + .await + .expect("checkpoint catalog") + .is_empty() + ); + let checkpoint_dir = configured_state_dir(&state).join("checkpoints").join(id); + let staging = std::fs::read_dir(checkpoint_dir) + .expect("checkpoint directory") + .filter_map(std::result::Result::ok) + .filter(|entry| entry.file_name().to_string_lossy().starts_with(".ckpt-")) + .count(); + assert_eq!(staging, 0, "{failpoint} must remove the staging owner"); + assert!(state.manager.backend_owner(uuid).is_some()); + } + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn checkpoint_head_pre_rename_failure_resumes_without_moving_head() { + let temp = tempfile::tempdir().expect("temp"); + let state = mock_state(&temp); + let created = created_json(&state, &test_request()).await; + let id = created["instance"]["id"].as_str().expect("id"); + let uuid = Uuid::parse_str(id).expect("uuid"); + write_checkpoint_fixture(&state, id).await; + let existing_head = state + .manager + .checkpoint(uuid) + .await + .expect("establish existing HEAD") + .id; + let hook = crate::failpoint::TestFailpoint::new(&["checkpoint-store-head-before-rename"]); + + let error = hook + .run(state.manager.checkpoint(uuid)) + .await + .expect_err("HEAD update must fail before rename"); + + assert!( + !matches!(error, BlazeDaemonError::RecoveryRequired(_)), + "known-unchanged HEAD failure must be compensated: {error}" + ); + let lifecycle = state.manager.get(uuid).expect("lifecycle"); + assert_eq!(lifecycle.state, SandboxState::Running); + assert!(lifecycle.operation.is_none()); + assert_eq!( + lifecycle.last_checkpoint.as_deref(), + Some(existing_head.as_str()) + ); + let persisted = state.state_store.load(uuid).expect("persisted lifecycle"); + assert_eq!(persisted.state, SandboxState::Running); + assert!(persisted.operation.is_none()); + assert_eq!( + persisted.last_checkpoint.as_deref(), + Some(existing_head.as_str()) + ); + assert!(state.manager.backend_owner(uuid).is_some()); + + let checkpoints = state + .manager + .list_checkpoints(uuid) + .await + .expect("published checkpoint"); + assert_eq!(checkpoints.len(), 2); + assert!( + checkpoints + .iter() + .any(|checkpoint| checkpoint.id == existing_head && checkpoint.is_head) + ); + assert!( + checkpoints + .iter() + .any(|checkpoint| checkpoint.id != existing_head && !checkpoint.is_head) + ); + let checkpoint_dir = configured_state_dir(&state).join("checkpoints").join(id); + assert_eq!( + std::fs::read_to_string(checkpoint_dir.join("HEAD")) + .expect("existing checkpoint HEAD") + .trim(), + existing_head + ); + assert!( + std::fs::read_dir(checkpoint_dir) + .expect("checkpoint directory") + .filter_map(std::result::Result::ok) + .all(|entry| !entry.file_name().to_string_lossy().starts_with(".HEAD.")), + "compensated HEAD failure must not retain temporary scratch" + ); + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn checkpoint_head_cleanup_failure_requires_recovery() { + let temp = tempfile::tempdir().expect("temp"); + let state = mock_state(&temp); + let created = created_json(&state, &test_request()).await; + let id = created["instance"]["id"].as_str().expect("id"); + let uuid = Uuid::parse_str(id).expect("uuid"); + write_checkpoint_fixture(&state, id).await; + let hook = crate::failpoint::TestFailpoint::new(&[ + "checkpoint-store-head-before-rename", + "checkpoint-store-head-cleanup", + ]); + + let error = hook + .run(state.manager.checkpoint(uuid)) + .await + .expect_err("failed temporary HEAD cleanup must require recovery"); + + assert!(matches!(error, BlazeDaemonError::RecoveryRequired(_))); + let lifecycle = state.manager.get(uuid).expect("lifecycle"); + assert_eq!(lifecycle.state, SandboxState::RecoveryRequired); + assert_eq!( + lifecycle.operation.and_then(|journal| journal.phase), + Some(OperationPhase::CheckpointPublished) + ); + assert!(state.manager.backend_owner(uuid).is_some()); + let checkpoints = state + .manager + .list_checkpoints(uuid) + .await + .expect("published checkpoint"); + assert_eq!(checkpoints.len(), 1); + assert!(!checkpoints[0].is_head); + let checkpoint_dir = configured_state_dir(&state).join("checkpoints").join(id); + assert_eq!( + std::fs::read_dir(checkpoint_dir) + .expect("checkpoint directory") + .filter_map(std::result::Result::ok) + .filter(|entry| entry.file_name().to_string_lossy().starts_with(".HEAD.")) + .count(), + 1 + ); + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn checkpoint_state_failures_retain_the_reached_durable_phase() { + for (failpoint, expected_phase, expected_head) in [ + ( + "checkpoint-published-state", + OperationPhase::CheckpointPublished, + false, + ), + ( + "checkpoint-head-state", + OperationPhase::CheckpointHeadUpdated, + true, + ), + ] { + let temp = tempfile::tempdir().expect("temp"); + let state = mock_state(&temp); + let created = created_json(&state, &test_request()).await; + let id = created["instance"]["id"].as_str().expect("id"); + let uuid = Uuid::parse_str(id).expect("uuid"); + write_checkpoint_fixture(&state, id).await; + let hook = crate::failpoint::TestFailpoint::new(&[failpoint]); + + let error = hook + .run(state.manager.checkpoint(uuid)) + .await + .expect_err("state commit must fail"); + + assert!(matches!(error, BlazeDaemonError::RecoveryRequired(_))); + let lifecycle = state.manager.get(uuid).expect("lifecycle"); + assert_eq!(lifecycle.state, SandboxState::RecoveryRequired); + assert_eq!( + lifecycle + .operation + .as_ref() + .and_then(|journal| journal.phase), + Some(expected_phase) + ); + let checkpoints = state + .manager + .list_checkpoints(uuid) + .await + .expect("published checkpoint"); + assert_eq!(checkpoints.len(), 1); + assert_eq!(checkpoints[0].is_head, expected_head); + assert!(state.manager.backend_owner(uuid).is_some()); + } + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn checkpoint_intent_and_stage_cleanup_failure_retain_recovery_ownership() { + let temp = tempfile::tempdir().expect("temp"); + let state = mock_state(&temp); + let created = created_json(&state, &test_request()).await; + let id = created["instance"]["id"].as_str().expect("id"); + let uuid = Uuid::parse_str(id).expect("uuid"); + write_checkpoint_fixture(&state, id).await; + let hook = crate::failpoint::TestFailpoint::new(&[ + "checkpoint-begin-state-commit", + "checkpoint-store-abort-before-rename", + ]); + + let error = hook + .run(state.manager.checkpoint(uuid)) + .await + .expect_err("intent commit and staging cleanup must fail"); + + assert!(matches!(error, BlazeDaemonError::RecoveryRequired(_))); + assert!( + error + .to_string() + .contains("checkpoint intent state commit failed") + ); + assert!( + error + .to_string() + .contains("checkpoint staging cleanup failed") + ); + + let lifecycle = state.manager.get(uuid).expect("lifecycle"); + assert_eq!(lifecycle.state, SandboxState::RecoveryRequired); + let journal = lifecycle.operation.as_ref().expect("checkpoint journal"); + assert_eq!(journal.kind, OperationKind::Checkpoint); + assert_eq!(journal.phase, Some(OperationPhase::CheckpointPreparing)); + + let persisted = state.state_store.load(uuid).expect("persisted lifecycle"); + assert_eq!(persisted.state, SandboxState::RecoveryRequired); + assert_eq!(persisted.operation, lifecycle.operation); + + let checkpoint_dir = configured_state_dir(&state).join("checkpoints").join(id); + let stages = std::fs::read_dir(&checkpoint_dir) + .expect("checkpoint directory") + .filter_map(std::result::Result::ok) + .map(|entry| entry.file_name().to_string_lossy().into_owned()) + .filter(|name| name.starts_with(".ckpt-") && name.ends_with(".tmp")) + .collect::>(); + assert_eq!(stages.len(), 1); + assert_eq!( + journal.checkpoint_id.as_deref(), + stages[0] + .strip_prefix('.') + .and_then(|name| name.strip_suffix(".tmp")) + ); + + let retry = state + .manager + .checkpoint(uuid) + .await + .expect_err("recovery-owned staging must block another checkpoint"); + assert!(matches!(retry, BlazeDaemonError::RecoveryRequired(_))); + assert_eq!( + std::fs::read_dir(checkpoint_dir) + .expect("checkpoint directory after retry") + .filter_map(std::result::Result::ok) + .filter(|entry| { + let name = entry.file_name(); + let name = name.to_string_lossy(); + name.starts_with(".ckpt-") && name.ends_with(".tmp") + }) + .count(), + 1 + ); + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn checkpoint_begin_cleanup_failure_retains_recovery_ownership() { + let temp = tempfile::tempdir().expect("temp"); + let state = mock_state(&temp); + let created = created_json(&state, &test_request()).await; + let id = created["instance"]["id"].as_str().expect("id"); + let uuid = Uuid::parse_str(id).expect("uuid"); + write_checkpoint_fixture(&state, id).await; + let hook = crate::failpoint::TestFailpoint::new(&[ + "checkpoint-store-stage-parent-sync", + "checkpoint-store-abort-before-rename", + ]); + + let error = hook + .run(state.manager.checkpoint(uuid)) + .await + .expect_err("stage synchronization and cleanup must fail"); + + assert!(matches!(error, BlazeDaemonError::RecoveryRequired(_))); + assert!( + error + .to_string() + .contains("checkpoint stage creation failed and cleanup could not be confirmed") + ); + + let lifecycle = state.manager.get(uuid).expect("lifecycle"); + assert_eq!(lifecycle.state, SandboxState::RecoveryRequired); + let journal = lifecycle.operation.as_ref().expect("checkpoint journal"); + assert_eq!(journal.kind, OperationKind::Checkpoint); + assert_eq!(journal.phase, Some(OperationPhase::CheckpointPreparing)); + + let persisted = state.state_store.load(uuid).expect("persisted lifecycle"); + assert_eq!(persisted.state, SandboxState::RecoveryRequired); + assert_eq!(persisted.operation, lifecycle.operation); + + let checkpoint_dir = configured_state_dir(&state).join("checkpoints").join(id); + let stages = std::fs::read_dir(&checkpoint_dir) + .expect("checkpoint directory") + .filter_map(std::result::Result::ok) + .map(|entry| entry.file_name().to_string_lossy().into_owned()) + .filter(|name| name.starts_with(".ckpt-") && name.ends_with(".tmp")) + .collect::>(); + assert_eq!(stages.len(), 1); + assert_eq!( + journal.checkpoint_id.as_deref(), + stages[0] + .strip_prefix('.') + .and_then(|name| name.strip_suffix(".tmp")) + ); + + let retry = state + .manager + .checkpoint(uuid) + .await + .expect_err("recovery-owned staging must block another checkpoint"); + assert!(matches!(retry, BlazeDaemonError::RecoveryRequired(_))); + assert_eq!( + std::fs::read_dir(checkpoint_dir) + .expect("checkpoint directory after retry") + .filter_map(std::result::Result::ok) + .filter(|entry| { + let name = entry.file_name(); + let name = name.to_string_lossy(); + name.starts_with(".ckpt-") && name.ends_with(".tmp") + }) + .count(), + 1 + ); + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn checkpoint_stage_open_cleanup_failure_retains_recovery_ownership() { + let temp = tempfile::tempdir().expect("temp"); + let state = mock_state(&temp); + let created = created_json(&state, &test_request()).await; + let id = created["instance"]["id"].as_str().expect("id"); + let uuid = Uuid::parse_str(id).expect("uuid"); + write_checkpoint_fixture(&state, id).await; + let hook = crate::failpoint::TestFailpoint::new(&[ + "checkpoint-store-stage-open", + "checkpoint-store-stage-open-cleanup-before-unlink", + ]); + + let error = hook + .run(state.manager.checkpoint(uuid)) + .await + .expect_err("stage opening and cleanup must fail"); + + assert!(matches!(error, BlazeDaemonError::RecoveryRequired(_))); + assert!( + error + .to_string() + .contains("checkpoint stage creation failed and cleanup could not be confirmed") + ); + + let lifecycle = state.manager.get(uuid).expect("lifecycle"); + assert_eq!(lifecycle.state, SandboxState::RecoveryRequired); + let journal = lifecycle.operation.as_ref().expect("checkpoint journal"); + assert_eq!(journal.kind, OperationKind::Checkpoint); + assert_eq!(journal.phase, Some(OperationPhase::CheckpointPreparing)); + + let persisted = state.state_store.load(uuid).expect("persisted lifecycle"); + assert_eq!(persisted.state, SandboxState::RecoveryRequired); + assert_eq!(persisted.operation, lifecycle.operation); + + let checkpoint_dir = configured_state_dir(&state).join("checkpoints").join(id); + let stages = std::fs::read_dir(&checkpoint_dir) + .expect("checkpoint directory") + .filter_map(std::result::Result::ok) + .map(|entry| entry.file_name().to_string_lossy().into_owned()) + .filter(|name| name.starts_with(".ckpt-") && name.ends_with(".tmp")) + .collect::>(); + assert_eq!(stages.len(), 1); + assert_eq!( + journal.checkpoint_id.as_deref(), + stages[0] + .strip_prefix('.') + .and_then(|name| name.strip_suffix(".tmp")) + ); + + let retry = state + .manager + .checkpoint(uuid) + .await + .expect_err("recovery-owned staging must block another checkpoint"); + assert!(matches!(retry, BlazeDaemonError::RecoveryRequired(_))); + assert_eq!( + std::fs::read_dir(checkpoint_dir) + .expect("checkpoint directory after retry") + .filter_map(std::result::Result::ok) + .filter(|entry| { + let name = entry.file_name(); + let name = name.to_string_lossy(); + name.starts_with(".ckpt-") && name.ends_with(".tmp") + }) + .count(), + 1 + ); + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn checkpoint_stage_open_cleanup_sync_failure_retains_recovery_ownership() { + let temp = tempfile::tempdir().expect("temp"); + let state = mock_state(&temp); + let created = created_json(&state, &test_request()).await; + let id = created["instance"]["id"].as_str().expect("id"); + let uuid = Uuid::parse_str(id).expect("uuid"); + write_checkpoint_fixture(&state, id).await; + let hook = crate::failpoint::TestFailpoint::new(&[ + "checkpoint-store-stage-open", + "checkpoint-store-stage-open-cleanup-parent-sync", + ]); + + let error = hook + .run(state.manager.checkpoint(uuid)) + .await + .expect_err("stage opening and cleanup synchronization must fail"); + + assert!(matches!(error, BlazeDaemonError::RecoveryRequired(_))); + assert!( + error + .to_string() + .contains("checkpoint stage creation failed and cleanup could not be confirmed") + ); + + let lifecycle = state.manager.get(uuid).expect("lifecycle"); + assert_eq!(lifecycle.state, SandboxState::RecoveryRequired); + let journal = lifecycle.operation.as_ref().expect("checkpoint journal"); + assert_eq!(journal.kind, OperationKind::Checkpoint); + assert_eq!(journal.phase, Some(OperationPhase::CheckpointPreparing)); + assert!( + journal + .checkpoint_id + .as_deref() + .is_some_and(|id| id.starts_with("ckpt-")) + ); + + let persisted = state.state_store.load(uuid).expect("persisted lifecycle"); + assert_eq!(persisted.state, SandboxState::RecoveryRequired); + assert_eq!(persisted.operation, lifecycle.operation); + + let checkpoint_dir = configured_state_dir(&state).join("checkpoints").join(id); + let stage_count = || { + std::fs::read_dir(&checkpoint_dir) + .expect("checkpoint directory") + .filter_map(std::result::Result::ok) + .filter(|entry| { + let name = entry.file_name(); + let name = name.to_string_lossy(); + name.starts_with(".ckpt-") && name.ends_with(".tmp") + }) + .count() + }; + assert_eq!(stage_count(), 0); + + let retry = state + .manager + .checkpoint(uuid) + .await + .expect_err("uncertain cleanup durability must block another checkpoint"); + assert!(matches!(retry, BlazeDaemonError::RecoveryRequired(_))); + assert_eq!(stage_count(), 0); + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn checkpoint_store_boundary_failures_preserve_observable_catalog_truth() { + for (failpoint, expected_phase, expected_head) in [ + ( + "checkpoint-store-publish-after-rename", + OperationPhase::CheckpointPaused, + false, + ), + ( + "checkpoint-store-head-after-rename", + OperationPhase::CheckpointPublished, + true, + ), + ] { + let temp = tempfile::tempdir().expect("temp"); + let state = mock_state(&temp); + let created = created_json(&state, &test_request()).await; + let id = created["instance"]["id"].as_str().expect("id"); + let uuid = Uuid::parse_str(id).expect("uuid"); + write_checkpoint_fixture(&state, id).await; + let hook = crate::failpoint::TestFailpoint::new(&[failpoint]); + + let error = hook + .run(state.manager.checkpoint(uuid)) + .await + .expect_err("durability boundary must report an uncertain result"); + + assert!(matches!(error, BlazeDaemonError::RecoveryRequired(_))); + let lifecycle = state.manager.get(uuid).expect("lifecycle"); + assert_eq!(lifecycle.state, SandboxState::RecoveryRequired); + assert_eq!( + lifecycle + .operation + .as_ref() + .and_then(|journal| journal.phase), + Some(expected_phase) + ); + let checkpoints = state + .manager + .list_checkpoints(uuid) + .await + .expect("observable checkpoint catalog"); + assert_eq!(checkpoints.len(), 1); + assert_eq!(checkpoints[0].is_head, expected_head); + assert!(state.manager.backend_owner(uuid).is_some()); + } + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn checkpoint_resume_failure_keeps_head_and_runtime_ownership() { + let temp = tempfile::tempdir().expect("temp"); + let state = mock_state(&temp); + let created = created_json(&state, &test_request()).await; + let id = created["instance"]["id"].as_str().expect("id"); + let uuid = Uuid::parse_str(id).expect("uuid"); + write_checkpoint_fixture(&state, id).await; + let hook = crate::failpoint::TestFailpoint::new(&["checkpoint-resume"]); + + let error = hook + .run(state.manager.checkpoint(uuid)) + .await + .expect_err("resume failure"); + + assert!(matches!(error, BlazeDaemonError::RecoveryRequired(_))); + let lifecycle = state.manager.get(uuid).expect("lifecycle"); + assert_eq!(lifecycle.state, SandboxState::RecoveryRequired); + assert_eq!( + lifecycle + .operation + .as_ref() + .and_then(|journal| journal.phase), + Some(OperationPhase::CheckpointHeadUpdated) + ); + assert!(state.manager.backend_owner(uuid).is_some()); + let checkpoints = state + .manager + .list_checkpoints(uuid) + .await + .expect("committed checkpoint"); + assert_eq!(checkpoints.len(), 1); + assert!(checkpoints[0].is_head); + + state.manager.destroy(uuid).await.expect("destroy retry"); + assert_eq!( + state.manager.get(uuid).expect("destroyed").state, + SandboxState::Destroyed + ); + assert_eq!( + state + .manager + .list_checkpoints(uuid) + .await + .expect("removed checkpoint history") + .len(), + 0 + ); + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn cancelled_parent_validation_precedes_mutation_and_supervisor_converges() { + let temp = tempfile::tempdir().expect("temp"); + let state = mock_state(&temp); + let created = created_json(&state, &test_request()).await; + let id = created["instance"]["id"].as_str().expect("id").to_string(); + let uuid = Uuid::parse_str(&id).expect("uuid"); + write_checkpoint_fixture(&state, &id).await; + let existing_head = state + .manager + .checkpoint(uuid) + .await + .expect("seed checkpoint") + .id; + let before = state.manager.get(uuid).expect("running lifecycle"); + let persisted_before = state + .state_store + .load(uuid) + .expect("persisted running lifecycle"); + let state_path = configured_state_dir(&state).join(&id).join("state.json"); + let state_bytes_before = std::fs::read(&state_path).expect("persisted state bytes"); + let checkpoint_root = configured_state_dir(&state).join("checkpoints").join(&id); + + let hook = crate::failpoint::TestFailpoint::new(&["checkpoint-before-read-head"]); + let capture_state = state.clone(); + let capture_hook = hook.clone(); + let capture = tokio::spawn(async move { + capture_hook + .run(capture_state.manager.checkpoint(uuid)) + .await + }); + tokio::time::timeout(Duration::from_secs(2), hook.wait_until_paused()) + .await + .expect("parent validation pause"); + + tokio::time::timeout( + Duration::from_millis(250), + tokio::time::sleep(Duration::from_millis(1)), + ) + .await + .expect("parent validation must not occupy the async runtime worker"); + assert!(state.manager.operation_lock(uuid).try_lock().is_err()); + assert_eq!( + serde_json::to_value(state.manager.get(uuid).expect("unchanged lifecycle")) + .expect("serialize current lifecycle"), + serde_json::to_value(&before).expect("serialize prior lifecycle") + ); + assert_eq!( + serde_json::to_value( + state + .state_store + .load(uuid) + .expect("unchanged persisted lifecycle") + ) + .expect("serialize current persisted lifecycle"), + serde_json::to_value(&persisted_before).expect("serialize prior persisted lifecycle") + ); + assert_eq!( + std::fs::read(&state_path).expect("state bytes during parent validation"), + state_bytes_before + ); + assert!( + std::fs::read_dir(&checkpoint_root) + .expect("checkpoint catalog") + .filter_map(std::result::Result::ok) + .all(|entry| !entry.file_name().to_string_lossy().starts_with(".ckpt-")), + "parent validation must precede staging and checkpoint journaling" + ); + + capture.abort(); + assert!( + capture + .await + .expect_err("outer checkpoint request must be cancelled") + .is_cancelled() + ); + assert!(state.manager.operation_lock(uuid).try_lock().is_err()); + + hook.release(); + let operation = tokio::time::timeout( + Duration::from_secs(2), + state.manager.operation_lock(uuid).lock_owned(), + ) + .await + .expect("parent validation must finish and release the operation lock"); + drop(operation); + + let after = state + .manager + .get(uuid) + .expect("running lifecycle after cancellation"); + assert_eq!(after.state, SandboxState::Running); + assert!(after.operation.is_none()); + let completed_head = after + .last_checkpoint + .expect("detached supervisor checkpoint"); + assert_ne!(completed_head, existing_head); + let checkpoints = state + .manager + .list_checkpoints(uuid) + .await + .expect("completed checkpoint catalog"); + assert_eq!(checkpoints.len(), 2); + assert!( + checkpoints + .iter() + .any(|checkpoint| checkpoint.id == existing_head && !checkpoint.is_head) + ); + assert!( + checkpoints + .iter() + .any(|checkpoint| checkpoint.id == completed_head && checkpoint.is_head) + ); + assert!(state.manager.backend_owner(uuid).is_some()); + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn published_checkpoint_holds_the_operation_lock_until_head_commit() { + let temp = tempfile::tempdir().expect("temp"); + let state = mock_state(&temp); + let created = created_json(&state, &test_request()).await; + let id = created["instance"]["id"].as_str().expect("id"); + let uuid = Uuid::parse_str(id).expect("uuid"); + write_checkpoint_fixture(&state, id).await; + let hook = crate::failpoint::TestFailpoint::new(&["checkpoint-after-publish-before-head"]); + let capture_state = state.clone(); + let capture_hook = hook.clone(); + let capture = tokio::spawn(async move { + capture_hook + .run(capture_state.manager.checkpoint(uuid)) + .await + }); + hook.wait_until_paused().await; + + let persisted = state + .state_store + .load(uuid) + .expect("persisted checkpoint journal"); + assert_eq!(persisted.state, SandboxState::Paused); + assert_eq!( + persisted.operation.and_then(|journal| journal.phase), + Some(OperationPhase::CheckpointPublished) + ); + let list_state = state.clone(); + let mut list = tokio::spawn(async move { list_state.manager.list_checkpoints(uuid).await }); + assert!( + tokio::time::timeout(Duration::from_millis(20), &mut list) + .await + .is_err(), + "checkpoint listing must wait for a consistent catalog boundary" + ); + let destroy_state = state.clone(); + let mut destroy = tokio::spawn(async move { destroy_state.manager.destroy(uuid).await }); + assert!( + tokio::time::timeout(Duration::from_millis(20), &mut destroy) + .await + .is_err(), + "destroy must wait for checkpoint ownership" + ); + + hook.release(); + capture + .await + .expect("capture task") + .expect("checkpoint capture"); + let checkpoints = list.await.expect("list task").expect("checkpoint list"); + assert_eq!(checkpoints.len(), 1); + assert!(checkpoints[0].is_head); + assert!(destroy.await.expect("destroy task").expect("destroy")); + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn cancelled_storage_capture_retains_ownership_until_publication() { + struct FailpointReleaseGuard<'a>(&'a crate::failpoint::TestFailpoint); + + impl Drop for FailpointReleaseGuard<'_> { + fn drop(&mut self) { + self.0.release(); + } + } + + let temp = tempfile::tempdir().expect("temp"); + let state = mock_state(&temp); + let created = created_json(&state, &test_request()).await; + let id = created["instance"]["id"].as_str().expect("id").to_string(); + let uuid = Uuid::parse_str(&id).expect("uuid"); + let slot = write_checkpoint_fixture(&state, &id).await; + let hook = crate::failpoint::TestFailpoint::new(&["storage-capture-before-publish"]); + let release_guard = FailpointReleaseGuard(&hook); + let capture_state = state.clone(); + let capture_hook = hook.clone(); + let capture = tokio::spawn(async move { + capture_hook + .run(capture_state.manager.checkpoint(uuid)) + .await + }); + hook.wait_until_paused().await; + + let interrupted = state.manager.get(uuid).expect("checkpoint lifecycle"); + assert_eq!(interrupted.state, SandboxState::Paused); + let checkpoint_id = interrupted + .operation + .as_ref() + .and_then(|journal| journal.checkpoint_id.clone()) + .expect("checkpoint id"); + assert_eq!( + interrupted.operation.and_then(|journal| journal.phase), + Some(OperationPhase::CheckpointPaused) + ); + assert!(state.manager.operation_lock(uuid).try_lock().is_err()); + + let staging = configured_state_dir(&state) + .join("checkpoints") + .join(&id) + .join(format!(".{checkpoint_id}.tmp")); + let stage_entries = || { + let mut entries = std::fs::read_dir(&staging) + .expect("checkpoint staging directory") + .map(|entry| { + entry + .expect("checkpoint staging entry") + .file_name() + .to_string_lossy() + .into_owned() + }) + .collect::>(); + entries.sort(); + entries + }; + let entries_before_cancel = stage_entries(); + assert!( + entries_before_cancel + .iter() + .any(|name| name == "vmstate.snap") + ); + assert!( + entries_before_cancel + .iter() + .any(|name| name == "memory.snap") + ); + assert!( + entries_before_cancel + .iter() + .any(|name| name.starts_with(".rootfs.snap.capture-") && name.ends_with(".tmp")) + ); + assert!( + !entries_before_cancel + .iter() + .any(|name| name == "rootfs.snap") + ); + assert!(slot.rootfs_path.exists()); + + capture.abort(); + assert!( + capture + .await + .expect_err("outer checkpoint request must be cancelled") + .is_cancelled() + ); + assert!(state.manager.operation_lock(uuid).try_lock().is_err()); + + let list_state = state.clone(); + let mut list = tokio::spawn(async move { list_state.manager.list_checkpoints(uuid).await }); + let destroy_state = state.clone(); + let mut destroy = tokio::spawn(async move { destroy_state.manager.destroy(uuid).await }); + assert!( + tokio::time::timeout(Duration::from_millis(20), &mut list) + .await + .is_err(), + "checkpoint listing must wait for blocking storage capture" + ); + assert!( + tokio::time::timeout(Duration::from_millis(20), &mut destroy) + .await + .is_err(), + "destroy must wait for blocking storage capture" + ); + assert_eq!(stage_entries(), entries_before_cancel); + assert!(slot.rootfs_path.exists()); + + hook.release(); + drop(release_guard); + let checkpoints = tokio::time::timeout(Duration::from_secs(2), &mut list) + .await + .expect("detached supervisor must release checkpoint listing") + .expect("checkpoint list task") + .expect("checkpoint list"); + assert_eq!(checkpoints.len(), 1); + assert_eq!(checkpoints[0].id, checkpoint_id); + assert!(checkpoints[0].is_head); + assert!( + tokio::time::timeout(Duration::from_secs(2), &mut destroy) + .await + .expect("detached supervisor must release destroy") + .expect("destroy task") + .expect("destroy completed checkpoint") + ); + let destroyed = state.manager.get(uuid).expect("destroyed lifecycle"); + assert_eq!(destroyed.state, SandboxState::Destroyed); + assert!(destroyed.operation.is_none()); + assert!(!staging.exists()); + assert!(!slot.rootfs_path.exists()); + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn cancelled_blocking_publish_finishes_before_unlocking() { + let temp = tempfile::tempdir().expect("temp"); + let state = mock_state(&temp); + let created = created_json(&state, &test_request()).await; + let id = created["instance"]["id"].as_str().expect("id").to_string(); + let uuid = Uuid::parse_str(&id).expect("uuid"); + write_checkpoint_fixture(&state, &id).await; + let hook = + crate::failpoint::TestFailpoint::new(&["checkpoint-after-store-publish-before-state"]); + let capture_state = state.clone(); + let capture_hook = hook.clone(); + let capture = tokio::spawn(async move { + capture_hook + .run(capture_state.manager.checkpoint(uuid)) + .await + }); + hook.wait_until_paused().await; + + tokio::time::timeout( + Duration::from_millis(250), + tokio::time::sleep(Duration::from_millis(1)), + ) + .await + .expect("blocking publication must not occupy the async runtime worker"); + let persisted = state + .state_store + .load(uuid) + .expect("persisted paused checkpoint journal"); + assert_eq!(persisted.state, SandboxState::Paused); + assert_eq!( + persisted + .operation + .as_ref() + .and_then(|journal| journal.phase), + Some(OperationPhase::CheckpointPaused) + ); + let checkpoint_id = persisted + .operation + .as_ref() + .and_then(|journal| journal.checkpoint_id.clone()) + .expect("checkpoint id"); + let checkpoint_root = configured_state_dir(&state).join("checkpoints").join(&id); + assert!(checkpoint_root.join(&checkpoint_id).is_dir()); + assert!(!checkpoint_root.join("HEAD").exists()); + + capture.abort(); + assert!( + capture + .await + .expect_err("outer checkpoint request must be cancelled") + .is_cancelled() + ); + assert!(state.manager.operation_lock(uuid).try_lock().is_err()); + + hook.release(); + let operation = tokio::time::timeout( + Duration::from_secs(2), + state.manager.operation_lock(uuid).lock_owned(), + ) + .await + .expect("publication must finish and release the operation lock"); + let completed = state + .state_store + .load(uuid) + .expect("persisted completed checkpoint"); + assert_eq!(completed.state, SandboxState::Running); + assert!(completed.operation.is_none()); + assert_eq!( + completed.last_checkpoint.as_deref(), + Some(checkpoint_id.as_str()) + ); + drop(operation); + + let checkpoints = state + .manager + .list_checkpoints(uuid) + .await + .expect("published checkpoint catalog"); + assert_eq!(checkpoints.len(), 1); + assert_eq!(checkpoints[0].id, checkpoint_id); + assert!(checkpoints[0].is_head); + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn cancelled_blocking_head_update_finishes_before_unlocking() { + let temp = tempfile::tempdir().expect("temp"); + let state = mock_state(&temp); + let created = created_json(&state, &test_request()).await; + let id = created["instance"]["id"].as_str().expect("id").to_string(); + let uuid = Uuid::parse_str(&id).expect("uuid"); + write_checkpoint_fixture(&state, &id).await; + let hook = + crate::failpoint::TestFailpoint::new(&["checkpoint-after-store-head-before-state"]); + let capture_state = state.clone(); + let capture_hook = hook.clone(); + let capture = tokio::spawn(async move { + capture_hook + .run(capture_state.manager.checkpoint(uuid)) + .await + }); + hook.wait_until_paused().await; + + tokio::time::timeout( + Duration::from_millis(250), + tokio::time::sleep(Duration::from_millis(1)), + ) + .await + .expect("blocking HEAD update must not occupy the async runtime worker"); + let persisted = state + .state_store + .load(uuid) + .expect("persisted published checkpoint journal"); + assert_eq!(persisted.state, SandboxState::Paused); + assert_eq!( + persisted + .operation + .as_ref() + .and_then(|journal| journal.phase), + Some(OperationPhase::CheckpointPublished) + ); + let checkpoint_id = persisted + .operation + .as_ref() + .and_then(|journal| journal.checkpoint_id.clone()) + .expect("checkpoint id"); + let head_path = configured_state_dir(&state) + .join("checkpoints") + .join(&id) + .join("HEAD"); + assert_eq!( + std::fs::read_to_string(&head_path) + .expect("published checkpoint HEAD") + .trim(), + checkpoint_id + ); + + capture.abort(); + assert!( + capture + .await + .expect_err("outer checkpoint request must be cancelled") + .is_cancelled() + ); + assert!(state.manager.operation_lock(uuid).try_lock().is_err()); + + hook.release(); + let operation = tokio::time::timeout( + Duration::from_secs(2), + state.manager.operation_lock(uuid).lock_owned(), + ) + .await + .expect("HEAD update must finish and release the operation lock"); + let completed = state + .state_store + .load(uuid) + .expect("persisted completed checkpoint"); + assert_eq!(completed.state, SandboxState::Running); + assert!(completed.operation.is_none()); + assert_eq!( + completed.last_checkpoint.as_deref(), + Some(checkpoint_id.as_str()) + ); + drop(operation); + + let checkpoints = state + .manager + .list_checkpoints(uuid) + .await + .expect("checkpoint catalog with HEAD"); + assert_eq!(checkpoints.len(), 1); + assert_eq!(checkpoints[0].id, checkpoint_id); + assert!(checkpoints[0].is_head); + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn cancelled_blocking_list_holds_the_operation_lock_until_scan_completion() { + let temp = tempfile::tempdir().expect("temp"); + let state = mock_state(&temp); + let created = created_json(&state, &test_request()).await; + let id = created["instance"]["id"].as_str().expect("id").to_string(); + let uuid = Uuid::parse_str(&id).expect("uuid"); + write_checkpoint_fixture(&state, &id).await; + state + .manager + .checkpoint(uuid) + .await + .expect("seed checkpoint"); + + let hook = crate::failpoint::TestFailpoint::new(&["checkpoint-before-store-list"]); + let list_state = state.clone(); + let list_hook = hook.clone(); + let list = tokio::spawn(async move { + list_hook + .run(list_state.manager.list_checkpoints(uuid)) + .await + }); + hook.wait_until_paused().await; + list.abort(); + assert!( + list.await + .expect_err("outer checkpoint list request must be cancelled") + .is_cancelled() + ); + assert!(state.manager.operation_lock(uuid).try_lock().is_err()); + + let destroy_state = state.clone(); + let mut destroy = tokio::spawn(async move { destroy_state.manager.destroy(uuid).await }); + assert!( + tokio::time::timeout(Duration::from_millis(20), &mut destroy) + .await + .is_err(), + "destroy must wait for the detached catalog scan" + ); + + hook.release(); + destroy + .await + .expect("destroy task") + .expect("destroy after checkpoint scan"); + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test(flavor = "current_thread")] + async fn checkpoint_cleanup_does_not_block_the_async_runtime_worker() { + struct FailpointReleaseGuard<'a>(&'a crate::failpoint::TestFailpoint); + + impl Drop for FailpointReleaseGuard<'_> { + fn drop(&mut self) { + self.0.release(); + } + } + + let temp = tempfile::tempdir().expect("temp"); + let state = mock_state(&temp); + let created = created_json(&state, &test_request()).await; + let id = created["instance"]["id"].as_str().expect("id").to_string(); + let uuid = Uuid::parse_str(&id).expect("uuid"); + write_checkpoint_fixture(&state, &id).await; + state + .manager + .checkpoint(uuid) + .await + .expect("seed checkpoint"); + + let hook = crate::failpoint::TestFailpoint::new(&["checkpoint-before-store-remove"]); + let release_guard = FailpointReleaseGuard(&hook); + let destroy_state = state.clone(); + let destroy_hook = hook.clone(); + let destroy = + tokio::spawn( + async move { destroy_hook.run(destroy_state.manager.destroy(uuid)).await }, + ); + hook.wait_until_paused().await; + + tokio::time::timeout( + Duration::from_millis(250), + tokio::time::sleep(Duration::from_millis(1)), + ) + .await + .expect("checkpoint cleanup must not occupy the async runtime worker"); + assert!(state.manager.operation_lock(uuid).try_lock().is_err()); + + destroy.abort(); + assert!( + destroy + .await + .expect_err("cancel the outer destroy request") + .is_cancelled() + ); + assert!(state.manager.operation_lock(uuid).try_lock().is_err()); + + let list_state = state.clone(); + let mut list = tokio::spawn(async move { list_state.manager.list_checkpoints(uuid).await }); + let retry_state = state.clone(); + let mut retry = tokio::spawn(async move { retry_state.manager.destroy(uuid).await }); + assert!( + tokio::time::timeout(Duration::from_millis(20), &mut list) + .await + .is_err(), + "checkpoint listing must wait for detached destruction" + ); + assert!( + tokio::time::timeout(Duration::from_millis(20), &mut retry) + .await + .is_err(), + "a destroy retry must wait for detached destruction" + ); + + hook.release(); + drop(release_guard); + assert!( + !retry + .await + .expect("retry task") + .expect("retry after detached destruction") + ); + assert!( + list.await + .expect("list task") + .expect("list after detached destruction") + .is_empty() + ); + let destroyed = state.manager.get(uuid).expect("destroyed lifecycle"); + assert_eq!(destroyed.state, SandboxState::Destroyed); + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn cancelled_published_checkpoint_finishes_before_destroy() { + let temp = tempfile::tempdir().expect("temp"); + let state = mock_state(&temp); + let created = created_json(&state, &test_request()).await; + let id = created["instance"]["id"].as_str().expect("id").to_string(); + let uuid = Uuid::parse_str(&id).expect("uuid"); + write_checkpoint_fixture(&state, &id).await; + let hook = crate::failpoint::TestFailpoint::new(&["checkpoint-after-publish-before-head"]); + let capture_state = state.clone(); + let capture_hook = hook.clone(); + let capture = tokio::spawn(async move { + capture_hook + .run(capture_state.manager.checkpoint(uuid)) + .await + }); + hook.wait_until_paused().await; + capture.abort(); + let cancelled = capture + .await + .expect_err("client checkpoint task must be cancelled"); + + let interrupted = state.manager.get(uuid).expect("interrupted lifecycle"); + assert_eq!(interrupted.state, SandboxState::Paused); + assert_eq!( + interrupted.operation.and_then(|journal| journal.phase), + Some(OperationPhase::CheckpointPublished) + ); + assert!( + !configured_state_dir(&state) + .join("checkpoints") + .join(&id) + .join("HEAD") + .exists() + ); + + let destroy_state = state.clone(); + let mut destroy = tokio::spawn(async move { destroy_state.manager.destroy(uuid).await }); + assert!( + tokio::time::timeout(Duration::from_millis(20), &mut destroy) + .await + .is_err(), + "destroy must wait for the detached checkpoint supervisor" + ); + + hook.release(); + assert!(cancelled.is_cancelled()); + tokio::time::timeout(Duration::from_secs(2), &mut destroy) + .await + .expect("detached supervisor and queued destroy must converge") + .expect("destroy task") + .expect("destroy completed checkpoint"); + let checkpoints = state + .manager + .list_checkpoints(uuid) + .await + .expect("removed checkpoint history"); + assert!(checkpoints.is_empty()); + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn cancelled_checkpoint_requests_finish_in_detached_supervisors() { + for (failpoint, expected_state, expected_phase) in [ + ( + "checkpoint-after-begin", + SandboxState::Running, + OperationPhase::CheckpointPreparing, + ), + ( + "checkpoint-after-pause", + SandboxState::Paused, + OperationPhase::CheckpointPaused, + ), + ( + "checkpoint-after-head", + SandboxState::Paused, + OperationPhase::CheckpointHeadUpdated, + ), + ] { + let temp = tempfile::tempdir().expect("temp"); + let state = mock_state(&temp); + let created = created_json(&state, &test_request()).await; + let id = created["instance"]["id"].as_str().expect("id").to_string(); + let uuid = Uuid::parse_str(&id).expect("uuid"); + write_checkpoint_fixture(&state, &id).await; + let checkpoint_id = cancel_checkpoint_request_at( + &state, + uuid, + failpoint, + expected_state, + expected_phase, + ) + .await; + + let completed = state.manager.get(uuid).expect("completed lifecycle"); + assert_eq!(completed.state, SandboxState::Running); + assert!(completed.operation.is_none()); + assert_eq!( + completed.last_checkpoint.as_deref(), + Some(checkpoint_id.as_str()) + ); + let checkpoints = state + .manager + .list_checkpoints(uuid) + .await + .expect("completed checkpoint history"); + assert_eq!(checkpoints.len(), 1); + assert_eq!(checkpoints[0].id, checkpoint_id); + assert!(checkpoints[0].is_head); + state + .manager + .destroy(uuid) + .await + .expect("destroy after detached checkpoint completion"); + let destroyed = state.manager.get(uuid).expect("destroyed lifecycle"); + assert_eq!(destroyed.state, SandboxState::Destroyed); + assert!(destroyed.operation.is_none()); + } + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn crashed_checkpoint_phases_are_reconciled_from_durable_state() { + for (phase, expected_state) in [ + (OperationPhase::CheckpointPreparing, SandboxState::Running), + (OperationPhase::CheckpointPaused, SandboxState::Paused), + (OperationPhase::CheckpointPublished, SandboxState::Paused), + (OperationPhase::CheckpointHeadUpdated, SandboxState::Paused), + ] { + let restart_temp = tempfile::tempdir().expect("restart temp"); + let config = test_config(&restart_temp); + let restart_state = mock_state_from_config(config.clone()); + let created = created_json(&restart_state, &test_request()).await; + let restart_id = created["instance"]["id"] + .as_str() + .expect("restart id") + .to_string(); + let restart_uuid = Uuid::parse_str(&restart_id).expect("restart uuid"); + write_checkpoint_fixture(&restart_state, &restart_id).await; + persist_crashed_checkpoint_phase(&restart_state, restart_uuid, phase).await; + drop(restart_state); + + let restarted = mock_state_from_config(config); + let interrupted = restarted + .manager + .get(restart_uuid) + .expect("scanned interrupted lifecycle"); + assert_eq!(interrupted.state, expected_state); + assert_eq!( + interrupted.operation.and_then(|journal| journal.phase), + Some(phase) + ); + assert!(restarted.manager.backend_owner(restart_uuid).is_none()); + + let report = restarted.manager.reconcile_startup().await; + assert_eq!(report.attempted, 1); + assert_eq!(report.completed, 1); + assert!(report.failures.is_empty()); + let destroyed = restarted + .manager + .get(restart_uuid) + .expect("reconciled lifecycle"); + assert_eq!(destroyed.state, SandboxState::Destroyed); + assert!(destroyed.operation.is_none()); + let checkpoints = restarted + .manager + .list_checkpoints(restart_uuid) + .await + .expect("reconciled checkpoint history"); + assert!(checkpoints.is_empty()); + let checkpoint_dir = configured_state_dir(&restarted) + .join("checkpoints") + .join(&restart_id); + assert!(!checkpoint_dir.exists()); + } + } + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn guest_operations_wait_for_checkpoint_publication() { + let temp = tempfile::tempdir().expect("temp"); + let state = guest_mock_state(&temp); + let created = created_json(&state, &test_request()).await; + let id = created["instance"]["id"].as_str().expect("id"); + let uuid = Uuid::parse_str(id).expect("uuid"); + write_checkpoint_fixture(&state, id).await; + state + .manager + .write_file(uuid, "/tmp/existing".into(), b"before") + .await + .expect("seed guest file"); + let hook = crate::failpoint::TestFailpoint::new(&["checkpoint-after-publish-before-head"]); + let capture_state = state.clone(); + let capture_hook = hook.clone(); + let capture = tokio::spawn(async move { + capture_hook + .run(capture_state.manager.checkpoint(uuid)) + .await + }); + hook.wait_until_paused().await; + + let exec_state = state.clone(); + let mut exec = tokio::spawn(async move { + exec_state + .manager + .exec(uuid, "printf locked".into(), None, None, 5) + .await + }); + let read_state = state.clone(); + let mut read = tokio::spawn(async move { + read_state + .manager + .read_file(uuid, "/tmp/existing".into()) + .await + }); + let write_state = state.clone(); + let mut write = tokio::spawn(async move { + write_state + .manager + .write_file(uuid, "/tmp/after".into(), b"after") + .await + }); + assert!( + tokio::time::timeout(Duration::from_millis(20), &mut exec) + .await + .is_err(), + "guest exec must wait for checkpoint ownership" + ); + assert!( + tokio::time::timeout(Duration::from_millis(20), &mut read) + .await + .is_err(), + "guest read must wait for checkpoint ownership" + ); + assert!( + tokio::time::timeout(Duration::from_millis(20), &mut write) + .await + .is_err(), + "guest write must wait for checkpoint ownership" + ); + + hook.release(); + capture + .await + .expect("capture task") + .expect("checkpoint capture"); + assert_eq!( + exec.await.expect("exec task").expect("guest exec").stdout, + b"printf locked" + ); + assert_eq!( + read.await.expect("read task").expect("guest read"), + b"before" + ); + write.await.expect("write task").expect("guest write"); + } + + #[tokio::test] + async fn checkpoint_rejects_an_unfinished_lifecycle_journal() { + let temp = tempfile::tempdir().expect("temp"); + let config = test_config(&temp); + let storage: Arc = Arc::new(FileStorageProvider::with_images( + config.storage.images_dir.clone(), + config.storage.instances_dir.clone(), + )); + let state = build_test_state( + config, + test_policy(BackendKind::Mock), + spawners(BackendKind::Mock, Arc::new(MockSpawner)), + BackendKind::Mock, + storage, + ); + let created = created_json(&state, &test_request()).await; + let id = created["instance"]["id"].as_str().expect("id"); + let uuid = Uuid::parse_str(id).expect("uuid"); + let journal = { + let mut instances = state.instances.lock().expect("instances"); + let instance = instances.get_mut(&uuid).expect("instance"); + instance.begin_operation(OperationKind::Create); + state + .state_store + .persist(instance) + .expect("persist journal"); + instance.operation.clone().expect("journal") + }; + + let error = checkpoint(&state, id) + .await + .expect_err("unfinished lifecycle work must fail closed"); + + assert!(matches!(error, BlazeDaemonError::RecoveryRequired(_))); + assert_eq!( + state.instances.lock().expect("instances")[&uuid].operation, + Some(journal) + ); + assert_eq!( + state + .state_store + .load(uuid) + .expect("persisted instance") + .operation, + state.instances.lock().expect("instances")[&uuid].operation + ); + } + + #[tokio::test] + async fn checkpoint_rejects_a_non_running_lifecycle_state() { + let temp = tempfile::tempdir().expect("temp"); + let config = test_config(&temp); + let storage: Arc = Arc::new(FileStorageProvider::with_images( + config.storage.images_dir.clone(), + config.storage.instances_dir.clone(), + )); + let state = build_test_state( + config, + test_policy(BackendKind::Mock), + spawners(BackendKind::Mock, Arc::new(MockSpawner)), + BackendKind::Mock, + storage, + ); + let created = created_json(&state, &test_request()).await; + let id = created["instance"]["id"].as_str().expect("id"); + let uuid = Uuid::parse_str(id).expect("uuid"); + state.manager.destroy(uuid).await.expect("destroy"); + + let error = checkpoint(&state, id) + .await + .expect_err("checkpoint must require a running instance"); + + assert!(matches!(error, BlazeDaemonError::Conflict(_))); + assert_eq!(error.status_code(), 409); + } + #[tokio::test] async fn sandbox_guest_routes_use_owned_runtime() { let temp = tempfile::tempdir().expect("temp"); diff --git a/src/blaze/crates/blazed/src/checkpoint_store.rs b/src/blaze/crates/blazed/src/checkpoint_store.rs new file mode 100644 index 000000000..1e2f0460a --- /dev/null +++ b/src/blaze/crates/blazed/src/checkpoint_store.rs @@ -0,0 +1,2559 @@ +// SPDX-License-Identifier: Apache-2.0 +//! Filesystem-backed checkpoint catalog owned by the daemon. +//! +//! The catalog, sandbox directories, staging directories, committed +//! checkpoints, and artifacts are opened relative to retained directory +//! descriptors. Configured pathnames are retained only for diagnostics. + +use std::collections::{HashMap, HashSet}; +use std::fs::File; +use std::io::{Read, Seek, SeekFrom, Write}; +use std::path::{Path, PathBuf}; +use std::sync::{Arc, Mutex}; + +#[cfg(test)] +use std::sync::atomic::{AtomicUsize, Ordering}; + +use blaze_core::checkpoint::{ + CHECKPOINT_FORMAT_VERSION, CheckpointArtifact, CheckpointInfo, CheckpointMetadata, + CheckpointValidationError, CommitCheckpoint, REQUIRED_ARTIFACTS, validate_artifact_name, + validate_checkpoint_id, validate_checkpoint_manifest, validate_commit_checkpoint, +}; +use chrono::Utc; +use rustix::fs::{ + AtFlags, Dir, Mode, OFlags, RenameFlags, fchmod, fstat, fsync, mkdirat, openat, renameat, + renameat_with, unlinkat, +}; +use rustix::io::Errno; +use sha2::{Digest, Sha256}; +use thiserror::Error; +use uuid::Uuid; + +use crate::error::BlazeDaemonError; +use crate::state_store::{OwnedStateDirectory, StateStore}; + +const METADATA_FILE: &str = "metadata.json"; +const HEAD_FILE: &str = "HEAD"; +const STAGING_SUFFIX: &str = ".tmp"; +const TOMBSTONE_SUFFIX: &str = ".tombstone"; +const ABORT_TOMBSTONE_PREFIX: &str = ".abort."; +const CHECKPOINT_DIRECTORY_MODE: Mode = Mode::RWXU; +const CHECKPOINT_FILE_MODE: Mode = Mode::RUSR.union(Mode::WUSR); + +/// Failure while reading or mutating the daemon checkpoint catalog. +#[derive(Debug, Error)] +pub enum CheckpointStoreError { + /// A checkpoint record failed pure model validation. + #[error(transparent)] + Validation(#[from] CheckpointValidationError), + + /// Opening the namespace through the retained state root failed. + #[error("checkpoint catalog state-root operation failed: {0}")] + State(#[source] BlazeDaemonError), + + /// A catalog filesystem operation failed. + #[error("checkpoint catalog {operation} failed for {}: {source}", path.display())] + Io { + operation: &'static str, + path: PathBuf, + #[source] + source: std::io::Error, + }, + + /// A metadata file could not be encoded or decoded. + #[error("checkpoint metadata at {} is invalid: {source}", path.display())] + Json { + path: PathBuf, + #[source] + source: serde_json::Error, + }, + + /// The catalog layout violates an invariant required for safe mutation. + #[error("checkpoint catalog invariant failed: {0}")] + Invariant(String), +} + +/// Convenient result type for checkpoint catalog operations. +pub type Result = std::result::Result; + +/// Failure while creating a checkpoint stage. +#[derive(Debug, Error)] +#[error("{source}")] +pub struct CheckpointBeginError { + recovery_checkpoint_id: Option, + #[source] + source: Box, +} + +impl CheckpointBeginError { + /// Return the checkpoint whose stage cleanup could not be confirmed. + pub fn recovery_checkpoint_id(&self) -> Option<&str> { + self.recovery_checkpoint_id.as_deref() + } + + fn recovery_required(checkpoint_id: String, source: CheckpointStoreError) -> Self { + Self { + recovery_checkpoint_id: Some(checkpoint_id), + source: Box::new(source), + } + } +} + +impl From for CheckpointBeginError { + fn from(source: CheckpointStoreError) -> Self { + Self { + recovery_checkpoint_id: None, + source: Box::new(source), + } + } +} + +/// Result of creating a checkpoint stage. +pub type BeginResult = std::result::Result; + +/// Namespace outcome reported when checkpoint publication fails. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum CheckpointPublishOutcome { + /// The staging directory is known not to have been renamed. + KnownUnpublished, + /// The staging-to-catalog rename may have completed. + Unknown, +} + +/// Checkpoint publication failure with its observable namespace outcome. +#[derive(Debug, Error)] +#[error("{source}")] +pub struct CheckpointPublishError { + outcome: CheckpointPublishOutcome, + #[source] + source: CheckpointStoreError, +} + +impl CheckpointPublishError { + /// Return the strongest namespace outcome known at the failure boundary. + pub fn outcome(&self) -> CheckpointPublishOutcome { + self.outcome + } + + /// Return the underlying catalog error. + pub fn into_store_error(self) -> CheckpointStoreError { + self.source + } + + fn known_unpublished(source: CheckpointStoreError) -> Self { + Self { + outcome: CheckpointPublishOutcome::KnownUnpublished, + source, + } + } + + fn unknown(source: CheckpointStoreError) -> Self { + Self { + outcome: CheckpointPublishOutcome::Unknown, + source, + } + } +} + +/// Result of publishing a checkpoint staging directory. +pub type PublishResult = std::result::Result; + +/// Namespace outcome reported when moving checkpoint HEAD fails. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum CheckpointHeadOutcome { + /// HEAD retains its previous value and temporary cleanup is durable. + KnownUnchanged, + /// HEAD replacement may have completed, or temporary cleanup is uncertain. + Unknown, +} + +/// HEAD-update failure with its observable namespace outcome. +#[derive(Debug, Error)] +#[error("{source}")] +pub struct CheckpointHeadError { + outcome: CheckpointHeadOutcome, + #[source] + source: CheckpointStoreError, +} + +impl CheckpointHeadError { + /// Return the strongest namespace outcome known at the failure boundary. + pub fn outcome(&self) -> CheckpointHeadOutcome { + self.outcome + } + + /// Return the underlying catalog error. + pub fn into_store_error(self) -> CheckpointStoreError { + self.source + } + + fn known_unchanged(source: CheckpointStoreError) -> Self { + Self { + outcome: CheckpointHeadOutcome::KnownUnchanged, + source, + } + } + + fn unknown(source: CheckpointStoreError) -> Self { + Self { + outcome: CheckpointHeadOutcome::Unknown, + source, + } + } +} + +/// Result of atomically moving checkpoint HEAD. +pub type SetHeadResult = std::result::Result; + +/// Temporary checkpoint directory populated before atomic publication. +#[derive(Debug)] +pub struct CheckpointStage { + id: String, + sandbox_id: Uuid, + catalog: OwnedStateDirectory, + sandbox: OwnedStateDirectory, + directory: OwnedStateDirectory, + staging_name: String, +} + +impl CheckpointStage { + /// Generated checkpoint identifier. + pub fn id(&self) -> &str { + &self.id + } + + /// Resolve one artifact through the retained stage directory. + pub fn artifact_path(&self, name: &str) -> Result { + validate_artifact_name(name)?; + Ok(self.directory.path().join(name)) + } +} + +struct OwnedArtifact { + path: PathBuf, + file: File, +} + +struct VerifiedCheckpoint { + #[cfg(test)] + metadata: CheckpointMetadata, + directory: OwnedStateDirectory, + metadata_file: OwnedArtifact, + artifacts: Vec, +} + +struct LoadedCheckpointMetadata { + metadata: CheckpointMetadata, + directory: OwnedStateDirectory, + metadata_file: OwnedArtifact, + artifacts: Vec, +} + +pub(crate) struct PublishedCheckpoint { + catalog: OwnedStateDirectory, + sandbox: OwnedStateDirectory, + loaded: LoadedCheckpointMetadata, +} + +impl PublishedCheckpoint { + pub(crate) fn metadata(&self) -> &CheckpointMetadata { + &self.loaded.metadata + } + + fn require_linked(&self) -> Result<()> { + let sandbox_name = self.loaded.metadata.sandbox_id.to_string(); + require_linked_directory(&self.catalog, &sandbox_name, &self.sandbox)?; + self.loaded + .require_linked(&self.sandbox, &self.loaded.metadata.id) + } + + fn into_metadata(self) -> CheckpointMetadata { + self.loaded.metadata + } +} + +impl LoadedCheckpointMetadata { + fn require_linked(&self, sandbox: &OwnedStateDirectory, directory_name: &str) -> Result<()> { + validate_exact_entries( + &self.directory, + &[ + REQUIRED_ARTIFACTS[0], + REQUIRED_ARTIFACTS[1], + REQUIRED_ARTIFACTS[2], + METADATA_FILE, + ], + )?; + require_linked_file(&self.directory, METADATA_FILE, &self.metadata_file)?; + for (name, artifact) in REQUIRED_ARTIFACTS.iter().zip(&self.artifacts) { + require_linked_file(&self.directory, name, artifact)?; + } + require_linked_directory(sandbox, directory_name, &self.directory) + } +} + +impl VerifiedCheckpoint { + fn require_linked(&self, sandbox: &OwnedStateDirectory, checkpoint_id: &str) -> Result<()> { + validate_exact_entries( + &self.directory, + &[ + REQUIRED_ARTIFACTS[0], + REQUIRED_ARTIFACTS[1], + REQUIRED_ARTIFACTS[2], + METADATA_FILE, + ], + )?; + require_linked_file(&self.directory, METADATA_FILE, &self.metadata_file)?; + for (name, artifact) in REQUIRED_ARTIFACTS.iter().zip(&self.artifacts) { + require_linked_file(&self.directory, name, artifact)?; + } + require_linked_directory(sandbox, checkpoint_id, &self.directory) + } +} + +#[cfg(test)] +type BeforePublishRevalidation = Arc>>>; + +/// Filesystem-backed checkpoint catalog. +#[derive(Clone)] +pub struct CheckpointStore { + state_store: StateStore, + root: Arc>>, + #[cfg(test)] + before_publish_revalidation: BeforePublishRevalidation, + #[cfg(test)] + verified_checkpoint_calls: Arc, +} + +impl std::fmt::Debug for CheckpointStore { + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + formatter + .debug_struct("CheckpointStore") + .field("state_store", &self.state_store) + .finish_non_exhaustive() + } +} + +impl CheckpointStore { + /// Bind the catalog to the daemon's retained state-root owner. + pub fn new(state_store: StateStore) -> Self { + Self { + state_store, + root: Arc::new(Mutex::new(None)), + #[cfg(test)] + before_publish_revalidation: Arc::new(Mutex::new(None)), + #[cfg(test)] + verified_checkpoint_calls: Arc::new(AtomicUsize::new(0)), + } + } + + /// Create and durably expose a unique staging directory. + pub fn begin(&self, sandbox_id: Uuid) -> BeginResult { + let catalog = self.root()?; + let sandbox = self.ensure_sandbox_dir(&catalog, sandbox_id)?; + loop { + let id = format!("ckpt-{}", Uuid::new_v4()); + let staging_name = format!(".{id}{STAGING_SUFFIX}"); + let stage_path = sandbox.configured_path().join(&staging_name); + match mkdirat( + sandbox.descriptor(), + staging_name.as_str(), + CHECKPOINT_DIRECTORY_MODE, + ) { + Ok(()) => {} + Err(Errno::EXIST) => continue, + Err(source) => { + return Err(io_error( + "create checkpoint directory", + &stage_path, + std::io::Error::from(source), + ) + .into()); + } + } + let directory = + match checkpoint_store_failpoint("checkpoint-store-stage-open", &stage_path) + .and_then(|()| open_child_directory(&sandbox, &staging_name)) + { + Ok(directory) => directory, + Err(open_error) => { + let cleanup_result = checkpoint_store_failpoint( + "checkpoint-store-stage-open-cleanup-before-unlink", + &stage_path, + ) + .and_then(|()| { + unlinkat( + sandbox.descriptor(), + staging_name.as_str(), + AtFlags::REMOVEDIR, + ) + .map_err(|source| { + io_error( + "remove unopened checkpoint stage", + &stage_path, + std::io::Error::from(source), + ) + }) + }) + .and_then(|()| { + checkpoint_store_failpoint( + "checkpoint-store-stage-open-cleanup-parent-sync", + sandbox.configured_path(), + ) + .and_then(|()| sync_directory(&sandbox)) + }); + return match cleanup_result { + Ok(()) => Err(open_error.into()), + Err(cleanup_error) => Err(CheckpointBeginError::recovery_required( + id, + invariant(format!( + "checkpoint stage opening failed: {open_error}; \ + created-stage cleanup also failed: {cleanup_error}" + )), + )), + }; + } + }; + let sync_result = checkpoint_store_failpoint( + "checkpoint-store-stage-parent-sync", + sandbox.configured_path(), + ) + .and_then(|()| sync_directory(&sandbox)); + if let Err(sync_error) = sync_result { + return match self.abort_owned_stage(&sandbox, &staging_name, directory) { + Ok(()) => Err(sync_error.into()), + Err(cleanup_error) => Err(CheckpointBeginError::recovery_required( + id, + invariant(format!( + "checkpoint stage parent synchronization failed: \ + {sync_error}; owned-stage cleanup also failed: \ + {cleanup_error}" + )), + )), + }; + } + return Ok(CheckpointStage { + id, + sandbox_id, + catalog, + sandbox, + directory, + staging_name, + }); + } + } + + /// Hash, sync, and atomically publish a populated stage without moving HEAD. + #[cfg(test)] + pub fn publish( + &self, + stage: &CheckpointStage, + input: CommitCheckpoint, + ) -> PublishResult { + self.publish_retained(stage, input) + .map(PublishedCheckpoint::into_metadata) + } + + pub(crate) fn publish_retained( + &self, + stage: &CheckpointStage, + input: CommitCheckpoint, + ) -> PublishResult { + let loaded = (|| -> Result { + self.validate_stage(stage)?; + validate_commit_checkpoint(&stage.id, &input)?; + if let Some(parent) = &input.parent { + self.validated_chain_from(&stage.sandbox, stage.sandbox_id, parent)?; + } + if optional_child_directory(&stage.sandbox, &stage.id)?.is_some() { + return Err(invariant(format!( + "checkpoint publication target {} already exists", + stage.sandbox.configured_path().join(&stage.id).display() + ))); + } + validate_exact_entries(&stage.directory, &REQUIRED_ARTIFACTS)?; + + let mut opened_artifacts = Vec::with_capacity(REQUIRED_ARTIFACTS.len()); + for name in REQUIRED_ARTIFACTS { + let artifact = + open_required_file(&stage.directory, name, "open checkpoint artifact")?; + validate_checkpoint_artifact_owner(&artifact)?; + opened_artifacts.push((name, artifact)); + } + + let mut artifacts = Vec::with_capacity(REQUIRED_ARTIFACTS.len()); + for (name, artifact) in &mut opened_artifacts { + fchmod(&artifact.file, CHECKPOINT_FILE_MODE).map_err(|source| { + io_error( + "restrict checkpoint artifact permissions", + &artifact.path, + std::io::Error::from(source), + ) + })?; + artifact.file.sync_all().map_err(|source| { + io_error("sync checkpoint artifact", &artifact.path, source) + })?; + artifacts.push(hash_artifact(artifact, name)?); + } + + let metadata = CheckpointMetadata { + format_version: CHECKPOINT_FORMAT_VERSION, + id: stage.id.clone(), + parent: input.parent, + sandbox_id: stage.sandbox_id, + policy_name: input.policy_name, + image_digest: input.image_digest, + backend: input.backend, + backend_version: input.backend_version, + created_at: Utc::now(), + snapshot_kind: input.snapshot_kind, + artifacts, + }; + validate_checkpoint_manifest(&metadata, stage.sandbox_id, &stage.id)?; + let metadata_file = write_json_new(&stage.directory, METADATA_FILE, &metadata)?; + sync_directory(&stage.directory)?; + + #[cfg(test)] + self.run_before_publish_revalidation(); + + let loaded = LoadedCheckpointMetadata { + metadata, + directory: stage.directory.clone(), + metadata_file, + artifacts: opened_artifacts + .into_iter() + .map(|(_, artifact)| artifact) + .collect(), + }; + loaded.require_linked(&stage.sandbox, &stage.staging_name)?; + checkpoint_store_failpoint( + "checkpoint-store-publish-before-rename", + &stage.sandbox.configured_path().join(&stage.staging_name), + )?; + Ok(loaded) + })() + .map_err(CheckpointPublishError::known_unpublished)?; + + renameat_with( + stage.sandbox.descriptor(), + stage.staging_name.as_str(), + stage.sandbox.descriptor(), + stage.id.as_str(), + RenameFlags::NOREPLACE, + ) + .map_err(|source| { + let source = std::io::Error::from(source); + io_error( + "publish checkpoint directory", + stage.sandbox.configured_path().join(&stage.id), + source, + ) + }) + .map_err(CheckpointPublishError::unknown)?; + checkpoint_store_failpoint( + "checkpoint-store-publish-after-rename", + &stage.sandbox.configured_path().join(&stage.id), + ) + .map_err(CheckpointPublishError::unknown)?; + loaded + .require_linked(&stage.sandbox, &stage.id) + .map_err(CheckpointPublishError::unknown)?; + sync_directory(&stage.sandbox).map_err(CheckpointPublishError::unknown)?; + Ok(PublishedCheckpoint { + catalog: stage.catalog.clone(), + sandbox: stage.sandbox.clone(), + loaded, + }) + } + + /// Remove an unpublished stage owned by this process. + pub fn abort(&self, stage: CheckpointStage) -> Result<()> { + self.validate_stage(&stage)?; + self.abort_owned_stage(&stage.sandbox, &stage.staging_name, stage.directory) + } + + /// Read and validate one committed checkpoint and all artifact hashes. + #[cfg(test)] + pub fn verify(&self, sandbox_id: Uuid, checkpoint_id: &str) -> Result { + let catalog = self.root()?; + let sandbox = required_child_directory( + &catalog, + &sandbox_id.to_string(), + "open checkpoint sandbox directory", + )?; + Ok(self + .verified_checkpoint(&sandbox, sandbox_id, checkpoint_id)? + .metadata) + } + + /// List committed checkpoints and mark the lineage reachable from HEAD. + pub fn list(&self, sandbox_id: Uuid) -> Result> { + let catalog_root = self.root()?; + let Some(sandbox) = optional_child_directory(&catalog_root, &sandbox_id.to_string())? + else { + return Ok(Vec::new()); + }; + let catalog = self.load_catalog(&sandbox, sandbox_id)?; + let head = self.read_head_id_from(&sandbox)?; + let on_head_chain = match head.as_deref() { + Some(head) => lineage_from(&catalog, head)?, + None => HashSet::new(), + }; + + let mut checkpoints = Vec::with_capacity(catalog.len()); + for metadata in catalog.into_values() { + let size_bytes = metadata + .artifacts + .iter() + .try_fold(0_u64, |total, artifact| { + total.checked_add(artifact.size_bytes) + }) + .ok_or_else(|| { + invariant(format!( + "checkpoint {} artifact sizes overflow u64", + metadata.id + )) + })?; + checkpoints.push(CheckpointInfo { + id: metadata.id.clone(), + parent: metadata.parent, + created_at: metadata.created_at, + size_bytes, + is_head: head.as_deref() == Some(metadata.id.as_str()), + on_head_chain: on_head_chain.contains(&metadata.id), + }); + } + checkpoints.sort_by(|left, right| { + left.created_at + .cmp(&right.created_at) + .then_with(|| left.id.cmp(&right.id)) + }); + Ok(checkpoints) + } + + /// Atomically move HEAD to an already committed, verified checkpoint. + #[cfg(test)] + pub fn set_head(&self, sandbox_id: Uuid, checkpoint_id: &str) -> SetHeadResult<()> { + let catalog = self.root().map_err(CheckpointHeadError::known_unchanged)?; + let sandbox = required_child_directory( + &catalog, + &sandbox_id.to_string(), + "open checkpoint sandbox directory", + ) + .map_err(CheckpointHeadError::known_unchanged)?; + let verified = self + .verified_checkpoint(&sandbox, sandbox_id, checkpoint_id) + .map_err(CheckpointHeadError::known_unchanged)?; + let sandbox_name = sandbox_id.to_string(); + self.set_head_with_revalidation(&sandbox, checkpoint_id, || { + require_linked_directory(&catalog, &sandbox_name, &sandbox)?; + verified.require_linked(&sandbox, checkpoint_id) + }) + } + + pub(crate) fn set_head_published( + &self, + published: PublishedCheckpoint, + ) -> SetHeadResult { + let root = self.root().map_err(CheckpointHeadError::known_unchanged)?; + let checkpoint_id = published.metadata().id.clone(); + self.set_head_with_revalidation(&published.sandbox, &checkpoint_id, || { + if !same_directory(&root, &published.catalog)? { + return Err(invariant( + "published checkpoint belongs to a different catalog root", + )); + } + published.require_linked() + })?; + Ok(published.into_metadata()) + } + + fn set_head_with_revalidation( + &self, + sandbox: &OwnedStateDirectory, + checkpoint_id: &str, + mut revalidate: F, + ) -> SetHeadResult<()> + where + F: FnMut() -> Result<()>, + { + revalidate().map_err(CheckpointHeadError::known_unchanged)?; + // Opening with O_NOFOLLOW performs the complete type validation needed + // before atomic replacement. A missing HEAD is also valid. + let _existing_head = optional_file(sandbox, HEAD_FILE, "inspect existing checkpoint HEAD") + .map_err(CheckpointHeadError::known_unchanged)?; + + let temporary_name = format!(".HEAD.{}{STAGING_SUFFIX}", Uuid::new_v4()); + let mut temporary = create_new_file(sandbox, &temporary_name, "create temporary HEAD") + .map_err(CheckpointHeadError::known_unchanged)?; + let before_rename = (|| { + write_all( + &mut temporary.file, + &temporary.path, + checkpoint_id.as_bytes(), + )?; + write_all(&mut temporary.file, &temporary.path, b"\n")?; + temporary + .file + .sync_all() + .map_err(|source| io_error("sync temporary HEAD", &temporary.path, source))?; + revalidate()?; + checkpoint_store_failpoint("checkpoint-store-head-before-rename", &temporary.path) + })(); + if let Err(source) = before_rename { + let cleanup = + checkpoint_store_failpoint("checkpoint-store-head-cleanup", &temporary.path) + .and_then(|()| remove_file_if_exists(sandbox, &temporary_name)) + .and_then(|()| sync_directory(sandbox)); + return match cleanup { + Ok(()) => Err(CheckpointHeadError::known_unchanged(source)), + Err(cleanup) => Err(CheckpointHeadError::unknown(invariant(format!( + "{source}; temporary HEAD cleanup failed: {cleanup}" + )))), + }; + } + + renameat( + sandbox.descriptor(), + temporary_name.as_str(), + sandbox.descriptor(), + HEAD_FILE, + ) + .map_err(|source| { + io_error( + "publish checkpoint HEAD", + sandbox.configured_path().join(HEAD_FILE), + std::io::Error::from(source), + ) + }) + .map_err(CheckpointHeadError::unknown)?; + checkpoint_store_failpoint( + "checkpoint-store-head-after-rename", + &sandbox.configured_path().join(HEAD_FILE), + ) + .map_err(CheckpointHeadError::unknown)?; + require_linked_file(sandbox, HEAD_FILE, &temporary) + .map_err(CheckpointHeadError::unknown)?; + sync_directory(sandbox).map_err(CheckpointHeadError::unknown) + } + + /// Return the persisted HEAD, if present. + pub fn read_head(&self, sandbox_id: Uuid) -> Result> { + let catalog = self.root()?; + let Some(sandbox) = optional_child_directory(&catalog, &sandbox_id.to_string())? else { + return Ok(None); + }; + self.read_head_from(&sandbox, sandbox_id) + } + + /// Remove every checkpoint artifact owned by one sandbox. + /// + /// A missing sandbox directory is already clean. Any unexpected entry or + /// changed identity fails closed so the lifecycle owner can retain a + /// recoverable destroy record instead of deleting an unrelated object. + pub fn remove_sandbox(&self, sandbox_id: Uuid) -> Result<()> { + enum OwnedEntry { + Directory(String, OwnedStateDirectory), + File(String, OwnedArtifact), + } + + let catalog = self.root()?; + let name = sandbox_id.to_string(); + let Some(sandbox) = optional_child_directory(&catalog, &name)? else { + checkpoint_store_failpoint( + "checkpoint-store-sandbox-remove-parent-sync", + catalog.configured_path(), + )?; + return sync_directory(&catalog); + }; + let mut names: Vec<_> = directory_names(&sandbox, "scan sandbox checkpoint namespace")? + .into_iter() + .collect(); + names.sort(); + + let mut entries = Vec::with_capacity(names.len()); + for entry in names { + let kind = if entry == HEAD_FILE { + ScratchKind::File + } else if let Some(kind) = classify_scratch_name(&entry)? { + kind + } else { + validate_checkpoint_id(&entry)?; + ScratchKind::Directory + }; + match kind { + ScratchKind::Directory => entries.push(OwnedEntry::Directory( + entry.clone(), + required_child_directory(&sandbox, &entry, "open owned checkpoint directory")?, + )), + ScratchKind::File => entries.push(OwnedEntry::File( + entry.clone(), + open_required_file(&sandbox, &entry, "open owned checkpoint file")?, + )), + } + } + + for entry in entries { + match entry { + OwnedEntry::Directory(name, directory) => { + remove_owned_directory(&sandbox, &name, directory)?; + } + OwnedEntry::File(name, file) => remove_owned_file(&sandbox, &name, file)?, + } + } + sync_directory(&sandbox)?; + require_linked_directory(&catalog, &name, &sandbox)?; + checkpoint_store_failpoint( + "checkpoint-store-sandbox-remove-before-unlink", + sandbox.configured_path(), + )?; + unlinkat(catalog.descriptor(), name.as_str(), AtFlags::REMOVEDIR).map_err(|source| { + io_error( + "remove sandbox checkpoint namespace", + catalog.configured_path().join(&name), + std::io::Error::from(source), + ) + })?; + checkpoint_store_failpoint( + "checkpoint-store-sandbox-remove-parent-sync", + catalog.configured_path(), + )?; + sync_directory(&catalog) + } + + fn root(&self) -> Result { + let mut root = self + .root + .lock() + .map_err(|_| invariant("checkpoint root owner lock poisoned"))?; + if let Some(root) = root.as_ref() { + return Ok(root.clone()); + } + let opened = self + .state_store + .checkpoint_directory() + .map_err(CheckpointStoreError::State)?; + *root = Some(opened.clone()); + Ok(opened) + } + + fn ensure_sandbox_dir( + &self, + catalog: &OwnedStateDirectory, + sandbox_id: Uuid, + ) -> Result { + let name = sandbox_id.to_string(); + match create_child_directory(catalog, &name) { + Ok(directory) => { + checkpoint_store_failpoint( + "checkpoint-store-sandbox-parent-sync", + catalog.configured_path(), + )?; + sync_directory(catalog)?; + Ok(directory) + } + Err(CheckpointStoreError::Io { source, .. }) + if source.kind() == std::io::ErrorKind::AlreadyExists => + { + let directory = + required_child_directory(catalog, &name, "open checkpoint sandbox directory")?; + checkpoint_store_failpoint( + "checkpoint-store-sandbox-parent-sync", + catalog.configured_path(), + )?; + sync_directory(catalog)?; + Ok(directory) + } + Err(error) => Err(error), + } + } + + fn validate_stage(&self, stage: &CheckpointStage) -> Result<()> { + validate_checkpoint_id(&stage.id)?; + let root = self.root()?; + if !same_directory(&root, &stage.catalog)? { + return Err(invariant("checkpoint stage belongs to a different catalog")); + } + require_linked_directory(&root, &stage.sandbox_id.to_string(), &stage.sandbox)?; + require_linked_directory(&stage.sandbox, &stage.staging_name, &stage.directory)?; + if optional_child_directory(&stage.sandbox, &stage.id)?.is_some() { + return Err(invariant(format!( + "checkpoint publication target {} already exists", + stage.sandbox.configured_path().join(&stage.id).display() + ))); + } + Ok(()) + } + + fn abort_owned_stage( + &self, + sandbox: &OwnedStateDirectory, + staging_name: &str, + stage: OwnedStateDirectory, + ) -> Result<()> { + require_linked_directory(sandbox, staging_name, &stage)?; + let checkpoint_id = staging_name + .strip_prefix('.') + .and_then(|name| name.strip_suffix(STAGING_SUFFIX)) + .ok_or_else(|| invariant(format!("invalid staging name {staging_name:?}")))?; + validate_checkpoint_id(checkpoint_id)?; + let tombstone_name = format!( + "{ABORT_TOMBSTONE_PREFIX}{checkpoint_id}.{}{TOMBSTONE_SUFFIX}", + Uuid::new_v4() + ); + checkpoint_store_failpoint( + "checkpoint-store-abort-before-rename", + &sandbox.configured_path().join(staging_name), + )?; + renameat_with( + sandbox.descriptor(), + staging_name, + sandbox.descriptor(), + tombstone_name.as_str(), + RenameFlags::NOREPLACE, + ) + .map_err(|source| { + io_error( + "tombstone aborted checkpoint stage", + sandbox.configured_path().join(&tombstone_name), + std::io::Error::from(source), + ) + })?; + require_linked_directory(sandbox, &tombstone_name, &stage)?; + sync_directory(sandbox)?; + remove_owned_directory(sandbox, &tombstone_name, stage)?; + sync_directory(sandbox) + } + + fn verified_checkpoint( + &self, + sandbox: &OwnedStateDirectory, + sandbox_id: Uuid, + checkpoint_id: &str, + ) -> Result { + #[cfg(test)] + self.verified_checkpoint_calls + .fetch_add(1, Ordering::SeqCst); + + let LoadedCheckpointMetadata { + metadata, + directory, + metadata_file, + mut artifacts, + } = self.load_checkpoint_metadata(sandbox, sandbox_id, checkpoint_id)?; + + for (name, artifact) in REQUIRED_ARTIFACTS.iter().zip(&mut artifacts) { + let expected = metadata + .artifacts + .iter() + .find(|artifact| artifact.name == *name) + .ok_or_else(|| { + invariant(format!( + "validated checkpoint {checkpoint_id} has no record for {name}" + )) + })?; + let actual = hash_artifact(artifact, name)?; + if &actual != expected { + return Err(invariant(format!( + "checkpoint {checkpoint_id} artifact {name} failed integrity validation" + ))); + } + } + let verified = VerifiedCheckpoint { + #[cfg(test)] + metadata, + directory, + metadata_file, + artifacts, + }; + verified.require_linked(sandbox, checkpoint_id)?; + Ok(verified) + } + + fn load_checkpoint_metadata( + &self, + sandbox: &OwnedStateDirectory, + sandbox_id: Uuid, + checkpoint_id: &str, + ) -> Result { + validate_checkpoint_id(checkpoint_id)?; + let directory = required_child_directory( + sandbox, + checkpoint_id, + "open committed checkpoint directory", + )?; + validate_exact_entries( + &directory, + &[ + REQUIRED_ARTIFACTS[0], + REQUIRED_ARTIFACTS[1], + REQUIRED_ARTIFACTS[2], + METADATA_FILE, + ], + )?; + let mut metadata_file = + open_required_file(&directory, METADATA_FILE, "open checkpoint metadata")?; + let bytes = read_file(&mut metadata_file, "read checkpoint metadata")?; + let metadata: CheckpointMetadata = + serde_json::from_slice(&bytes).map_err(|source| CheckpointStoreError::Json { + path: metadata_file.path.clone(), + source, + })?; + validate_checkpoint_manifest(&metadata, sandbox_id, checkpoint_id)?; + let mut artifacts = Vec::with_capacity(REQUIRED_ARTIFACTS.len()); + for name in REQUIRED_ARTIFACTS { + artifacts.push(open_required_file( + &directory, + name, + "open checkpoint artifact", + )?); + } + let loaded = LoadedCheckpointMetadata { + metadata, + directory, + metadata_file, + artifacts, + }; + loaded.require_linked(sandbox, checkpoint_id)?; + Ok(loaded) + } + + fn validated_chain_from( + &self, + sandbox: &OwnedStateDirectory, + sandbox_id: Uuid, + checkpoint_id: &str, + ) -> Result> { + validate_checkpoint_id(checkpoint_id)?; + let mut current = checkpoint_id.to_string(); + let mut lineage = Vec::new(); + let mut seen = HashSet::new(); + loop { + if !seen.insert(current.clone()) { + return Err(invariant(format!( + "checkpoint parent cycle reaches {current}" + ))); + } + let metadata = self + .load_checkpoint_metadata(sandbox, sandbox_id, ¤t)? + .metadata; + lineage.push(current); + let Some(parent) = metadata.parent else { + break; + }; + current = parent; + } + Ok(lineage) + } + + fn load_catalog( + &self, + sandbox: &OwnedStateDirectory, + sandbox_id: Uuid, + ) -> Result> { + let mut catalog = HashMap::new(); + for name in directory_names(sandbox, "scan checkpoint catalog")? { + if !name.starts_with("ckpt-") { + continue; + } + validate_checkpoint_id(&name)?; + let metadata = self + .load_checkpoint_metadata(sandbox, sandbox_id, &name)? + .metadata; + catalog.insert(name, metadata); + } + Ok(catalog) + } + + fn read_head_id_from(&self, sandbox: &OwnedStateDirectory) -> Result> { + let Some(mut head) = optional_file(sandbox, HEAD_FILE, "open checkpoint HEAD")? else { + return Ok(None); + }; + let bytes = read_file(&mut head, "read checkpoint HEAD")?; + require_linked_file(sandbox, HEAD_FILE, &head)?; + let raw = std::str::from_utf8(&bytes) + .map_err(|error| invariant(format!("checkpoint HEAD is not UTF-8: {error}")))?; + let checkpoint_id = raw + .strip_suffix('\n') + .filter(|value| !value.contains('\n') && !value.contains('\r')) + .ok_or_else(|| invariant("checkpoint HEAD is not one canonical line"))?; + validate_checkpoint_id(checkpoint_id)?; + Ok(Some(checkpoint_id.to_string())) + } + + fn read_head_from( + &self, + sandbox: &OwnedStateDirectory, + sandbox_id: Uuid, + ) -> Result> { + let checkpoint_id = self.read_head_id_from(sandbox)?; + if let Some(checkpoint_id) = checkpoint_id.as_deref() { + let _verified = self.verified_checkpoint(sandbox, sandbox_id, checkpoint_id)?; + } + Ok(checkpoint_id) + } + + #[cfg(test)] + fn set_before_publish_revalidation(&self, hook: F) + where + F: FnOnce() + Send + 'static, + { + *self + .before_publish_revalidation + .lock() + .expect("checkpoint test hook lock") = Some(Box::new(hook)); + } + + #[cfg(test)] + fn verified_checkpoint_count(&self) -> usize { + self.verified_checkpoint_calls.load(Ordering::SeqCst) + } + + #[cfg(test)] + fn run_before_publish_revalidation(&self) { + if let Some(hook) = self + .before_publish_revalidation + .lock() + .expect("checkpoint test hook lock") + .take() + { + hook(); + } + } + + #[cfg(test)] + fn configured_root(&self) -> PathBuf { + self.root() + .expect("open checkpoint root") + .configured_path() + .to_path_buf() + } +} + +#[derive(Clone, Copy)] +enum ScratchKind { + Directory, + File, +} + +fn lineage_from( + catalog: &HashMap, + checkpoint_id: &str, +) -> Result> { + let mut current = checkpoint_id.to_string(); + let mut lineage = HashSet::new(); + loop { + if !lineage.insert(current.clone()) { + return Err(invariant(format!( + "checkpoint parent cycle reaches {current}" + ))); + } + let metadata = catalog.get(¤t).ok_or_else(|| { + invariant(format!( + "checkpoint lineage references missing parent {current}" + )) + })?; + let Some(parent) = &metadata.parent else { + break; + }; + current = parent.clone(); + } + Ok(lineage) +} + +fn create_child_directory(parent: &OwnedStateDirectory, name: &str) -> Result { + mkdirat(parent.descriptor(), name, CHECKPOINT_DIRECTORY_MODE).map_err(|source| { + io_error( + "create checkpoint directory", + parent.configured_path().join(name), + std::io::Error::from(source), + ) + })?; + match open_child_directory(parent, name) { + Ok(directory) => Ok(directory), + Err(error) => { + let _ = unlinkat(parent.descriptor(), name, AtFlags::REMOVEDIR); + Err(error) + } + } +} + +fn open_child_directory(parent: &OwnedStateDirectory, name: &str) -> Result { + let directory = openat( + parent.descriptor(), + name, + OFlags::RDONLY | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC, + Mode::empty(), + ) + .map_err(|source| { + io_error( + "open checkpoint directory", + parent.configured_path().join(name), + std::io::Error::from(source), + ) + })?; + Ok(OwnedStateDirectory::new( + parent.configured_path().join(name), + directory, + )) +} + +fn optional_child_directory( + parent: &OwnedStateDirectory, + name: &str, +) -> Result> { + match openat( + parent.descriptor(), + name, + OFlags::RDONLY | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC, + Mode::empty(), + ) { + Ok(directory) => Ok(Some(OwnedStateDirectory::new( + parent.configured_path().join(name), + directory, + ))), + Err(Errno::NOENT) => Ok(None), + Err(source) => Err(io_error( + "open checkpoint directory", + parent.configured_path().join(name), + std::io::Error::from(source), + )), + } +} + +fn required_child_directory( + parent: &OwnedStateDirectory, + name: &str, + operation: &'static str, +) -> Result { + optional_child_directory(parent, name)?.ok_or_else(|| { + io_error( + operation, + parent.configured_path().join(name), + std::io::Error::from(std::io::ErrorKind::NotFound), + ) + }) +} + +fn create_new_file( + directory: &OwnedStateDirectory, + name: &str, + operation: &'static str, +) -> Result { + let path = directory.configured_path().join(name); + let descriptor = openat( + directory.descriptor(), + name, + OFlags::WRONLY | OFlags::CREATE | OFlags::EXCL | OFlags::NOFOLLOW | OFlags::CLOEXEC, + CHECKPOINT_FILE_MODE, + ) + .map_err(|source| io_error(operation, &path, std::io::Error::from(source)))?; + Ok(OwnedArtifact { + path, + file: File::from(descriptor), + }) +} + +fn open_required_file( + directory: &OwnedStateDirectory, + name: &str, + operation: &'static str, +) -> Result { + optional_file(directory, name, operation)?.ok_or_else(|| { + io_error( + operation, + directory.configured_path().join(name), + std::io::Error::from(std::io::ErrorKind::NotFound), + ) + }) +} + +fn optional_file( + directory: &OwnedStateDirectory, + name: &str, + operation: &'static str, +) -> Result> { + let path = directory.configured_path().join(name); + let descriptor = match openat( + directory.descriptor(), + name, + OFlags::RDONLY | OFlags::NOFOLLOW | OFlags::CLOEXEC | OFlags::NONBLOCK, + Mode::empty(), + ) { + Ok(descriptor) => descriptor, + Err(Errno::NOENT) => return Ok(None), + Err(source) => return Err(io_error(operation, &path, std::io::Error::from(source))), + }; + let file = File::from(descriptor); + let metadata = file + .metadata() + .map_err(|source| io_error("inspect checkpoint file", &path, source))?; + if !metadata.is_file() { + return Err(invariant(format!( + "checkpoint file {} is not a regular file", + path.display() + ))); + } + Ok(Some(OwnedArtifact { path, file })) +} + +fn validate_checkpoint_artifact_owner(artifact: &OwnedArtifact) -> Result<()> { + let metadata = fstat(&artifact.file).map_err(|source| { + io_error( + "inspect checkpoint file", + &artifact.path, + std::io::Error::from(source), + ) + })?; + let expected_uid = unsafe { libc::geteuid() }; + if metadata.st_uid != expected_uid { + return Err(invariant(format!( + "checkpoint artifact {} is not owned by the daemon user", + artifact.path.display() + ))); + } + if metadata.st_nlink != 1 { + return Err(invariant(format!( + "checkpoint artifact {} must have exactly one hard link", + artifact.path.display() + ))); + } + Ok(()) +} + +fn validate_exact_entries(directory: &OwnedStateDirectory, expected: &[&str]) -> Result<()> { + let actual = directory_names(directory, "scan checkpoint directory")?; + let expected: HashSet = expected.iter().map(|name| (*name).to_string()).collect(); + if actual != expected { + let mut unexpected: Vec<_> = actual.difference(&expected).cloned().collect(); + let mut missing: Vec<_> = expected.difference(&actual).cloned().collect(); + unexpected.sort(); + missing.sort(); + return Err(invariant(format!( + "checkpoint directory {} has unexpected entries {:?} and missing entries {:?}", + directory.configured_path().display(), + unexpected, + missing + ))); + } + Ok(()) +} + +fn directory_names( + directory: &OwnedStateDirectory, + operation: &'static str, +) -> Result> { + // Open a fresh description so directory offsets from an earlier scan are + // never reused through the retained owner. + let scan = openat( + directory.descriptor(), + ".", + OFlags::RDONLY | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC, + Mode::empty(), + ) + .map_err(|source| { + io_error( + operation, + directory.configured_path(), + std::io::Error::from(source), + ) + })?; + let entries = Dir::read_from(&scan).map_err(|source| { + io_error( + operation, + directory.configured_path(), + std::io::Error::from(source), + ) + })?; + let mut names = HashSet::new(); + for entry in entries { + let entry = entry.map_err(|source| { + io_error( + operation, + directory.configured_path(), + std::io::Error::from(source), + ) + })?; + let Some(name) = entry.file_name().to_str().ok() else { + return Err(invariant(format!( + "checkpoint directory {} contains a non-UTF-8 name", + directory.configured_path().display() + ))); + }; + if name != "." && name != ".." { + names.insert(name.to_string()); + } + } + Ok(names) +} + +fn hash_artifact(artifact: &mut OwnedArtifact, name: &str) -> Result { + artifact + .file + .seek(SeekFrom::Start(0)) + .map_err(|source| io_error("rewind checkpoint artifact", &artifact.path, source))?; + let mut hasher = Sha256::new(); + let mut size_bytes = 0_u64; + let mut buffer = [0_u8; 64 * 1024]; + loop { + let read = artifact + .file + .read(&mut buffer) + .map_err(|source| io_error("read checkpoint artifact", &artifact.path, source))?; + if read == 0 { + break; + } + size_bytes = size_bytes + .checked_add(read as u64) + .ok_or_else(|| invariant(format!("checkpoint artifact {name} size overflow")))?; + hasher.update(&buffer[..read]); + } + Ok(CheckpointArtifact { + name: name.to_string(), + size_bytes, + sha256: format!("{:x}", hasher.finalize()), + }) +} + +fn write_json_new( + directory: &OwnedStateDirectory, + name: &str, + value: &T, +) -> Result { + let path = directory.configured_path().join(name); + let bytes = serde_json::to_vec_pretty(value) + .map_err(|source| CheckpointStoreError::Json { path, source })?; + let mut file = create_new_file(directory, name, "create checkpoint metadata")?; + write_all(&mut file.file, &file.path, &bytes)?; + write_all(&mut file.file, &file.path, b"\n")?; + file.file + .sync_all() + .map_err(|source| io_error("sync checkpoint metadata", &file.path, source))?; + Ok(file) +} + +fn read_file(file: &mut OwnedArtifact, operation: &'static str) -> Result> { + file.file + .seek(SeekFrom::Start(0)) + .map_err(|source| io_error("rewind checkpoint file", &file.path, source))?; + let mut bytes = Vec::new(); + file.file + .read_to_end(&mut bytes) + .map_err(|source| io_error(operation, &file.path, source))?; + Ok(bytes) +} + +fn write_all(file: &mut File, path: &Path, bytes: &[u8]) -> Result<()> { + file.write_all(bytes) + .map_err(|source| io_error("write checkpoint file", path, source)) +} + +fn require_linked_directory( + parent: &OwnedStateDirectory, + name: &str, + expected: &OwnedStateDirectory, +) -> Result<()> { + let linked = optional_child_directory(parent, name)?.ok_or_else(|| { + invariant(format!( + "checkpoint directory {} disappeared", + parent.configured_path().join(name).display() + )) + })?; + if !same_directory(&linked, expected)? { + return Err(invariant(format!( + "checkpoint directory {} changed identity", + parent.configured_path().join(name).display() + ))); + } + Ok(()) +} + +fn require_linked_file( + parent: &OwnedStateDirectory, + name: &str, + expected: &OwnedArtifact, +) -> Result<()> { + let linked = open_required_file(parent, name, "revalidate checkpoint file")?; + let expected_stat = fstat(&expected.file).map_err(|source| { + io_error( + "inspect checkpoint file", + &expected.path, + std::io::Error::from(source), + ) + })?; + let linked_stat = fstat(&linked.file).map_err(|source| { + io_error( + "inspect checkpoint file", + &linked.path, + std::io::Error::from(source), + ) + })?; + if expected_stat.st_dev != linked_stat.st_dev || expected_stat.st_ino != linked_stat.st_ino { + return Err(invariant(format!( + "checkpoint file {} changed identity", + parent.configured_path().join(name).display() + ))); + } + Ok(()) +} + +fn same_directory(left: &OwnedStateDirectory, right: &OwnedStateDirectory) -> Result { + let left = fstat(left.descriptor()).map_err(|source| { + io_error( + "inspect checkpoint directory", + left.configured_path(), + std::io::Error::from(source), + ) + })?; + let right = fstat(right.descriptor()).map_err(|source| { + io_error( + "inspect checkpoint directory", + right.configured_path(), + std::io::Error::from(source), + ) + })?; + Ok(left.st_dev == right.st_dev && left.st_ino == right.st_ino) +} + +fn sync_directory(directory: &OwnedStateDirectory) -> Result<()> { + fsync(directory.descriptor()).map_err(|source| { + io_error( + "sync checkpoint directory", + directory.configured_path(), + std::io::Error::from(source), + ) + }) +} + +fn remove_owned_file(parent: &OwnedStateDirectory, name: &str, file: OwnedArtifact) -> Result<()> { + require_linked_file(parent, name, &file)?; + unlinkat(parent.descriptor(), name, AtFlags::empty()).map_err(|source| { + io_error( + "remove checkpoint scratch file", + parent.configured_path().join(name), + std::io::Error::from(source), + ) + }) +} + +fn remove_file_if_exists(parent: &OwnedStateDirectory, name: &str) -> Result<()> { + match unlinkat(parent.descriptor(), name, AtFlags::empty()) { + Ok(()) | Err(Errno::NOENT) => Ok(()), + Err(source) => Err(io_error( + "remove checkpoint temporary file", + parent.configured_path().join(name), + std::io::Error::from(source), + )), + } +} + +fn remove_owned_directory( + parent: &OwnedStateDirectory, + name: &str, + directory: OwnedStateDirectory, +) -> Result<()> { + require_linked_directory(parent, name, &directory)?; + let mut entries: Vec<_> = directory_names(&directory, "scan checkpoint scratch directory")? + .into_iter() + .collect(); + entries.sort(); + let files = entries + .into_iter() + .map(|entry| { + let file = open_required_file(&directory, &entry, "open checkpoint scratch entry")?; + Ok((entry, file)) + }) + .collect::>>()?; + for (entry, file) in files { + remove_owned_file(&directory, &entry, file)?; + } + sync_directory(&directory)?; + require_linked_directory(parent, name, &directory)?; + unlinkat(parent.descriptor(), name, AtFlags::REMOVEDIR).map_err(|source| { + io_error( + "remove checkpoint scratch directory", + parent.configured_path().join(name), + std::io::Error::from(source), + ) + }) +} + +fn classify_scratch_name(name: &str) -> Result> { + if let Some(checkpoint_id) = name + .strip_prefix('.') + .and_then(|name| name.strip_suffix(STAGING_SUFFIX)) + .filter(|name| name.starts_with("ckpt-")) + { + validate_checkpoint_id(checkpoint_id)?; + return Ok(Some(ScratchKind::Directory)); + } + if let Some(nonce) = name + .strip_prefix(".HEAD.") + .and_then(|name| name.strip_suffix(STAGING_SUFFIX)) + { + parse_uuid_component(nonce, "checkpoint HEAD staging")?; + return Ok(Some(ScratchKind::File)); + } + if let Some(body) = name + .strip_prefix(ABORT_TOMBSTONE_PREFIX) + .and_then(|name| name.strip_suffix(TOMBSTONE_SUFFIX)) + { + let (checkpoint_id, nonce) = body + .rsplit_once('.') + .ok_or_else(|| invariant(format!("invalid checkpoint tombstone name {name:?}")))?; + validate_checkpoint_id(checkpoint_id)?; + parse_uuid_component(nonce, "checkpoint tombstone")?; + return Ok(Some(ScratchKind::Directory)); + } + Ok(None) +} + +fn parse_uuid_component(value: &str, label: &str) -> Result { + let uuid = Uuid::parse_str(value) + .map_err(|error| invariant(format!("invalid {label} identifier {value:?}: {error}")))?; + if value != uuid.to_string() { + return Err(invariant(format!( + "{label} identifier {value:?} is not canonical" + ))); + } + Ok(uuid) +} + +fn io_error( + operation: &'static str, + path: impl AsRef, + source: std::io::Error, +) -> CheckpointStoreError { + CheckpointStoreError::Io { + operation, + path: path.as_ref().to_path_buf(), + source, + } +} + +fn invariant(message: impl Into) -> CheckpointStoreError { + CheckpointStoreError::Invariant(message.into()) +} + +fn checkpoint_store_failpoint(name: &'static str, path: &Path) -> Result<()> { + crate::failpoint::storage(name).map_err(|error| { + io_error( + "run checkpoint store failpoint", + path, + std::io::Error::other(error.to_string()), + ) + }) +} + +#[cfg(test)] +mod tests { + use std::fs; + use std::os::unix::fs::{MetadataExt, PermissionsExt}; + + use blaze_core::backend::{BackendKind, SnapshotKind}; + + use super::*; + + fn store(temp: &tempfile::TempDir) -> CheckpointStore { + let state_root = temp.path().join("state"); + fs::create_dir(&state_root).expect("state root"); + CheckpointStore::new(StateStore::new(state_root)) + } + + fn commit_input(parent: Option) -> CommitCheckpoint { + CommitCheckpoint { + parent, + policy_name: "default".to_string(), + image_digest: "sha256:test".to_string(), + backend: BackendKind::Mock, + backend_version: Some("mock-v1".to_string()), + snapshot_kind: SnapshotKind::Full, + } + } + + fn populate(stage: &CheckpointStage, suffix: &str) { + for name in REQUIRED_ARTIFACTS { + let mut artifact = create_new_file(&stage.directory, name, "create test artifact") + .expect("create artifact"); + artifact + .file + .write_all(format!("{name}-{suffix}").as_bytes()) + .expect("write artifact"); + } + } + + fn publish( + store: &CheckpointStore, + sandbox_id: Uuid, + parent: Option, + move_head: bool, + ) -> String { + let stage = store.begin(sandbox_id).expect("begin checkpoint"); + let id = stage.id().to_string(); + populate(&stage, &id); + store + .publish(&stage, commit_input(parent)) + .expect("publish checkpoint"); + if move_head { + store.set_head(sandbox_id, &id).expect("move HEAD"); + } + id + } + + #[test] + fn publish_verify_and_list_preserve_the_head_boundary() { + let temp = tempfile::tempdir().expect("tempdir"); + let store = store(&temp); + let sandbox_id = Uuid::new_v4(); + let root = publish(&store, sandbox_id, None, true); + let unreachable = publish(&store, sandbox_id, Some(root.clone()), false); + + assert_eq!(store.read_head(sandbox_id).expect("HEAD"), Some(root)); + store + .verify(sandbox_id, &unreachable) + .expect("published checkpoint"); + let listed = store.list(sandbox_id).expect("list checkpoints"); + assert_eq!(listed.len(), 2); + assert_eq!(listed.iter().filter(|info| info.is_head).count(), 1); + assert!( + listed + .iter() + .any(|info| info.id == unreachable && !info.on_head_chain) + ); + } + + #[test] + fn checkpoint_tree_uses_owner_only_permissions() { + if std::env::var_os("BLAZE_CHECKPOINT_MODE_CHILD").is_none() { + return; + } + let temp = tempfile::tempdir().expect("tempdir"); + let store = store(&temp); + let sandbox_id = Uuid::new_v4(); + let stage = store.begin(sandbox_id).expect("begin checkpoint"); + populate(&stage, "private"); + let root = store.configured_root(); + let sandbox = root.join(sandbox_id.to_string()); + let staging = sandbox.join(&stage.staging_name); + + assert_mode(&root, 0o700); + assert_mode(&sandbox, 0o700); + assert_mode(&staging, 0o700); + + for name in REQUIRED_ARTIFACTS { + fs::set_permissions(staging.join(name), fs::Permissions::from_mode(0o666)) + .expect("make backend artifact permissive"); + } + let checkpoint_id = stage.id().to_string(); + store + .publish(&stage, commit_input(None)) + .expect("publish checkpoint"); + store + .set_head(sandbox_id, &checkpoint_id) + .expect("set HEAD"); + + let committed = sandbox.join(&checkpoint_id); + assert_mode(&committed, 0o700); + for name in REQUIRED_ARTIFACTS { + assert_mode(&committed.join(name), 0o600); + } + assert_mode(&committed.join(METADATA_FILE), 0o600); + assert_mode(&sandbox.join(HEAD_FILE), 0o600); + } + + #[test] + fn publish_rejects_multiply_linked_artifacts_without_changing_permissions() { + let temp = tempfile::tempdir().expect("tempdir"); + let store = store(&temp); + let sandbox_id = Uuid::new_v4(); + let stage = store.begin(sandbox_id).expect("begin checkpoint"); + populate(&stage, "hard-linked"); + let vmstate = stage.artifact_path("vmstate.snap").expect("VM state path"); + let rootfs = stage.artifact_path("rootfs.snap").expect("rootfs path"); + let external = temp.path().join("external-rootfs"); + fs::set_permissions(&vmstate, fs::Permissions::from_mode(0o666)) + .expect("make preceding artifact permissive"); + fs::set_permissions(&rootfs, fs::Permissions::from_mode(0o640)) + .expect("set observable mode"); + fs::hard_link(&rootfs, &external).expect("link artifact outside checkpoint tree"); + + let error = store + .publish(&stage, commit_input(None)) + .expect_err("multiply linked artifact must fail closed"); + + assert!(error.to_string().contains("exactly one hard link")); + assert_eq!( + fs::metadata(&vmstate).expect("VM state metadata").mode() & 0o777, + 0o666, + "validation must finish before any artifact permissions change" + ); + assert_eq!( + fs::metadata(&external).expect("external metadata").mode() & 0o777, + 0o640 + ); + assert!(store.list(sandbox_id).expect("list").is_empty()); + } + + #[test] + fn sandbox_removal_accepts_an_interrupted_internal_rootfs_link() { + let temp = tempfile::tempdir().expect("tempdir"); + let store = store(&temp); + let sandbox_id = Uuid::new_v4(); + let stage = store.begin(sandbox_id).expect("begin checkpoint"); + let rootfs = stage.artifact_path("rootfs.snap").expect("rootfs path"); + let temporary = stage + .directory + .configured_path() + .join(".rootfs.snap.capture-interrupted.tmp"); + fs::write(&temporary, b"partial rootfs").expect("write temporary rootfs"); + fs::hard_link(&temporary, &rootfs).expect("link captured rootfs"); + + store.remove_sandbox(sandbox_id).expect("remove sandbox"); + + assert!(!stage.directory.configured_path().exists()); + } + + #[test] + fn owner_only_modes_ignore_a_permissive_umask() { + let temp = tempfile::tempdir().expect("tempdir"); + let script = "umask 000; \"$1\" --exact checkpoint_store::tests::checkpoint_tree_uses_owner_only_permissions --nocapture"; + let output = std::process::Command::new("/bin/sh") + .arg("-c") + .arg(script) + .arg("sh") + .arg(std::env::current_exe().expect("test binary")) + .env("BLAZE_CHECKPOINT_MODE_CHILD", "1") + .env("TMPDIR", temp.path()) + .output() + .expect("run child test with a permissive umask"); + assert!( + output.status.success(), + "child test failed: stdout={} stderr={}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + } + + fn assert_mode(path: &Path, expected: u32) { + assert_eq!( + fs::symlink_metadata(path) + .unwrap_or_else(|error| panic!("inspect {}: {error}", path.display())) + .mode() + & 0o777, + expected, + "unexpected permissions for {}", + path.display() + ); + } + + #[test] + fn list_uses_committed_metadata_without_rehashing_artifacts() { + let temp = tempfile::tempdir().expect("tempdir"); + let store = store(&temp); + let sandbox_id = Uuid::new_v4(); + let root = publish(&store, sandbox_id, None, true); + let head = publish(&store, sandbox_id, Some(root.clone()), true); + let expected = store.list(sandbox_id).expect("list intact checkpoints"); + let sandbox = store.configured_root().join(sandbox_id.to_string()); + + fs::write(sandbox.join(&root).join("rootfs.snap"), b"corrupted root") + .expect("corrupt historical checkpoint artifact"); + fs::write(sandbox.join(&head).join("memory.snap"), b"corrupted head") + .expect("corrupt HEAD checkpoint artifact"); + + assert_eq!( + store.list(sandbox_id).expect("list from metadata"), + expected + ); + let verify_error = store + .verify(sandbox_id, &root) + .expect_err("verification must hash historical artifacts"); + assert!( + verify_error + .to_string() + .contains("failed integrity validation") + ); + let set_head_error = store + .set_head(sandbox_id, &root) + .expect_err("setting HEAD must hash the target artifacts"); + assert_eq!( + set_head_error.outcome(), + CheckpointHeadOutcome::KnownUnchanged + ); + assert!( + set_head_error + .to_string() + .contains("failed integrity validation") + ); + assert!( + store.read_head(sandbox_id).is_err(), + "reading HEAD must retain full artifact verification" + ); + } + + #[test] + fn publish_does_not_rehash_non_head_ancestor_artifacts() { + let temp = tempfile::tempdir().expect("tempdir"); + let store = store(&temp); + let sandbox_id = Uuid::new_v4(); + let root = publish(&store, sandbox_id, None, true); + let head = publish(&store, sandbox_id, Some(root.clone()), true); + let sandbox = store.configured_root().join(sandbox_id.to_string()); + + fs::write(sandbox.join(&root).join("rootfs.snap"), b"corrupted root") + .expect("corrupt non-HEAD ancestor artifact"); + + assert_eq!( + store.read_head(sandbox_id).expect("read intact HEAD"), + Some(head.clone()) + ); + let next = publish(&store, sandbox_id, Some(head), true); + assert_eq!( + store + .read_head(sandbox_id) + .expect("read newly published HEAD"), + Some(next) + ); + let verify_error = store + .verify(sandbox_id, &root) + .expect_err("explicit verification must hash ancestor artifacts"); + assert!( + verify_error + .to_string() + .contains("failed integrity validation") + ); + } + + #[test] + fn publish_metadata_only_lineage_validation_rejects_a_missing_parent() { + let temp = tempfile::tempdir().expect("tempdir"); + let store = store(&temp); + let sandbox_id = Uuid::new_v4(); + let missing_parent = format!("ckpt-{}", Uuid::new_v4()); + let stage = store.begin(sandbox_id).expect("begin checkpoint"); + populate(&stage, "missing-parent"); + + let error = store + .publish(&stage, commit_input(Some(missing_parent))) + .expect_err("missing parent must prevent publication"); + + assert_eq!(error.outcome(), CheckpointPublishOutcome::KnownUnpublished); + assert!( + error + .to_string() + .contains("open committed checkpoint directory") + ); + } + + #[test] + fn publish_metadata_only_lineage_validation_rejects_a_parent_cycle() { + let temp = tempfile::tempdir().expect("tempdir"); + let store = store(&temp); + let sandbox_id = Uuid::new_v4(); + let root = publish(&store, sandbox_id, None, true); + let head = publish(&store, sandbox_id, Some(root.clone()), true); + let sandbox = store.configured_root().join(sandbox_id.to_string()); + let root_metadata_path = sandbox.join(&root).join(METADATA_FILE); + let mut root_metadata: CheckpointMetadata = + serde_json::from_slice(&fs::read(&root_metadata_path).expect("read root metadata")) + .expect("decode root metadata"); + root_metadata.parent = Some(head.clone()); + fs::write( + &root_metadata_path, + serde_json::to_vec(&root_metadata).expect("encode cyclic root metadata"), + ) + .expect("write cyclic root metadata"); + let stage = store.begin(sandbox_id).expect("begin checkpoint"); + populate(&stage, "parent-cycle"); + + let error = store + .publish(&stage, commit_input(Some(head))) + .expect_err("parent cycle must prevent publication"); + + assert_eq!(error.outcome(), CheckpointPublishOutcome::KnownUnpublished); + assert!( + error + .to_string() + .contains("checkpoint parent cycle reaches") + ); + } + + #[test] + fn sandbox_removal_clears_scratch_and_committed_history() { + let temp = tempfile::tempdir().expect("tempdir"); + let store = store(&temp); + let sandbox_id = Uuid::new_v4(); + let committed = publish(&store, sandbox_id, None, true); + let stage = store.begin(sandbox_id).expect("begin checkpoint"); + let sandbox_dir = store.configured_root().join(sandbox_id.to_string()); + let stage_path = sandbox_dir.join(&stage.staging_name); + let temporary_head = sandbox_dir.join(format!(".HEAD.{}{STAGING_SUFFIX}", Uuid::new_v4())); + fs::write(&temporary_head, b"temporary").expect("write temporary HEAD"); + + store.remove_sandbox(sandbox_id).expect("remove sandbox"); + + assert!(!stage_path.exists()); + assert!(!temporary_head.exists()); + assert!(!sandbox_dir.join(committed).exists()); + assert_eq!(store.read_head(sandbox_id).expect("HEAD"), None); + assert!(!sandbox_dir.exists()); + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn sandbox_parent_sync_is_retried_for_an_existing_directory() { + let temp = tempfile::tempdir().expect("tempdir"); + let store = store(&temp); + let sandbox_id = Uuid::new_v4(); + let hook = crate::failpoint::TestFailpoint::new(&["checkpoint-store-sandbox-parent-sync"]); + + let first_error = hook + .run(async { store.begin(sandbox_id) }) + .await + .expect_err("initial parent sync must fail"); + assert!( + first_error + .to_string() + .contains("checkpoint-store-sandbox-parent-sync") + ); + assert!( + store + .configured_root() + .join(sandbox_id.to_string()) + .is_dir(), + "the failed parent sync leaves the newly created directory" + ); + + let retry_error = hook + .run(async { store.begin(sandbox_id) }) + .await + .expect_err("retry must synchronize the catalog again"); + assert!( + retry_error + .to_string() + .contains("checkpoint-store-sandbox-parent-sync") + ); + + let stage = store.begin(sandbox_id).expect("unarmed retry"); + store.abort(stage).expect("discard retry stage"); + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn stage_parent_sync_failure_removes_the_owned_stage() { + let temp = tempfile::tempdir().expect("tempdir"); + let store = store(&temp); + let sandbox_id = Uuid::new_v4(); + let hook = crate::failpoint::TestFailpoint::new(&["checkpoint-store-stage-parent-sync"]); + + let error = hook + .run(async { store.begin(sandbox_id) }) + .await + .expect_err("stage parent sync must fail"); + assert!( + error + .to_string() + .contains("checkpoint-store-stage-parent-sync") + ); + + let sandbox = store.configured_root().join(sandbox_id.to_string()); + assert!( + fs::read_dir(&sandbox) + .expect("checkpoint sandbox") + .next() + .is_none(), + "failed stage creation must not leave scratch entries" + ); + + let stage = store.begin(sandbox_id).expect("unarmed retry"); + store.abort(stage).expect("discard retry stage"); + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn sandbox_removal_retries_parent_sync_when_the_namespace_is_absent() { + let temp = tempfile::tempdir().expect("tempdir"); + let store = store(&temp); + let sandbox_id = Uuid::new_v4(); + let stage = store.begin(sandbox_id).expect("begin checkpoint"); + store.abort(stage).expect("discard stage"); + let sandbox = store.configured_root().join(sandbox_id.to_string()); + let hook = + crate::failpoint::TestFailpoint::new(&["checkpoint-store-sandbox-remove-parent-sync"]); + + let first_error = hook + .run(async { store.remove_sandbox(sandbox_id) }) + .await + .expect_err("initial parent sync must fail"); + assert!( + first_error + .to_string() + .contains("checkpoint-store-sandbox-remove-parent-sync") + ); + assert!(!sandbox.exists(), "the namespace was already unlinked"); + + let retry_error = hook + .run(async { store.remove_sandbox(sandbox_id) }) + .await + .expect_err("retry must synchronize the catalog again"); + assert!( + retry_error + .to_string() + .contains("checkpoint-store-sandbox-remove-parent-sync") + ); + + store + .remove_sandbox(sandbox_id) + .expect("unarmed retry synchronizes the catalog"); + } + + #[test] + fn state_root_replacement_does_not_redirect_catalog_creation() { + let temp = tempfile::tempdir().expect("tempdir"); + let store = store(&temp); + let configured_state = temp.path().join("state"); + let retained_state = temp.path().join("retained-state"); + fs::rename(&configured_state, &retained_state).expect("move retained state root"); + fs::create_dir(&configured_state).expect("replacement state root"); + + let sandbox_id = Uuid::new_v4(); + let stage = store + .begin(sandbox_id) + .expect("begin through retained root"); + populate(&stage, "retained-state"); + + assert!( + retained_state + .join("checkpoints") + .join(sandbox_id.to_string()) + .is_dir() + ); + assert!(!configured_state.join("checkpoints").exists()); + } + + #[test] + fn catalog_replacement_does_not_redirect_later_operations() { + let temp = tempfile::tempdir().expect("tempdir"); + let store = store(&temp); + let first = Uuid::new_v4(); + let first_stage = store.begin(first).expect("open catalog"); + store.abort(first_stage).expect("discard first stage"); + let configured_catalog = temp.path().join("state/checkpoints"); + let retained_catalog = temp.path().join("retained-checkpoints"); + fs::rename(&configured_catalog, &retained_catalog).expect("move retained catalog"); + fs::create_dir(&configured_catalog).expect("replacement catalog"); + + let second = Uuid::new_v4(); + let stage = store.begin(second).expect("begin through retained catalog"); + populate(&stage, "retained-catalog"); + + assert!(retained_catalog.join(second.to_string()).is_dir()); + assert!(!configured_catalog.join(second.to_string()).exists()); + } + + #[test] + fn sandbox_replacement_is_detected_before_publication() { + let temp = tempfile::tempdir().expect("tempdir"); + let store = store(&temp); + let sandbox_id = Uuid::new_v4(); + let stage = store.begin(sandbox_id).expect("begin checkpoint"); + populate(&stage, "sandbox-owner"); + let configured = store.configured_root().join(sandbox_id.to_string()); + let retained = store.configured_root().join("retained-sandbox"); + fs::rename(&configured, &retained).expect("move retained sandbox"); + fs::create_dir(&configured).expect("replacement sandbox"); + + let error = store + .publish(&stage, commit_input(None)) + .expect_err("sandbox replacement must fail closed"); + + assert!(error.to_string().contains("changed identity")); + assert!(!configured.join(stage.id()).exists()); + assert!(retained.join(&stage.staging_name).is_dir()); + } + + #[test] + fn stage_replacement_is_detected_before_publication() { + let temp = tempfile::tempdir().expect("tempdir"); + let store = store(&temp); + let sandbox_id = Uuid::new_v4(); + let stage = store.begin(sandbox_id).expect("begin checkpoint"); + populate(&stage, "stage-owner"); + let sandbox = store.configured_root().join(sandbox_id.to_string()); + let configured_stage = sandbox.join(&stage.staging_name); + let retained_stage = sandbox.join("retained-stage"); + fs::rename(&configured_stage, &retained_stage).expect("move retained stage"); + fs::create_dir(&configured_stage).expect("replacement stage"); + fs::write(configured_stage.join("sentinel"), b"replacement") + .expect("write replacement sentinel"); + + let error = store + .publish(&stage, commit_input(None)) + .expect_err("stage replacement must fail closed"); + + assert_eq!(error.outcome(), CheckpointPublishOutcome::KnownUnpublished); + assert!(error.to_string().contains("changed identity")); + assert!(!sandbox.join(stage.id()).exists()); + let error = store + .abort(stage) + .expect_err("replacement must prevent retained-stage cleanup"); + assert!(error.to_string().contains("changed identity")); + assert_eq!( + fs::read(configured_stage.join("sentinel")).expect("read replacement sentinel"), + b"replacement" + ); + assert!(retained_stage.is_dir()); + } + + #[test] + fn artifact_replacement_during_publication_is_detected() { + let temp = tempfile::tempdir().expect("tempdir"); + let store = store(&temp); + let sandbox_id = Uuid::new_v4(); + let stage = store.begin(sandbox_id).expect("begin checkpoint"); + populate(&stage, "artifact-owner"); + let rootfs = store + .configured_root() + .join(sandbox_id.to_string()) + .join(&stage.staging_name) + .join("rootfs.snap"); + let retained = temp.path().join("retained-rootfs.snap"); + store.set_before_publish_revalidation(move || { + fs::rename(&rootfs, &retained).expect("move retained artifact"); + fs::write(&rootfs, b"replacement").expect("replacement artifact"); + }); + + let error = store + .publish(&stage, commit_input(None)) + .expect_err("artifact replacement must fail closed"); + + assert!(error.to_string().contains("changed identity")); + assert!(store.list(sandbox_id).expect("list").is_empty()); + } + + #[cfg(unix)] + #[test] + fn verify_rejects_artifact_symlinks() { + use std::os::unix::fs::symlink; + + let temp = tempfile::tempdir().expect("tempdir"); + let store = store(&temp); + let sandbox_id = Uuid::new_v4(); + let checkpoint_id = publish(&store, sandbox_id, None, true); + let artifact = store + .configured_root() + .join(sandbox_id.to_string()) + .join(&checkpoint_id) + .join("rootfs.snap"); + fs::remove_file(&artifact).expect("remove artifact"); + let outside = temp.path().join("outside"); + fs::write(&outside, b"outside").expect("write outside file"); + symlink(&outside, &artifact).expect("link artifact"); + + assert!(store.verify(sandbox_id, &checkpoint_id).is_err()); + } + + #[cfg(unix)] + #[test] + fn begin_rejects_a_symlinked_catalog_root() { + use std::os::unix::fs::symlink; + + let temp = tempfile::tempdir().expect("tempdir"); + let state = temp.path().join("state"); + let actual = temp.path().join("actual"); + fs::create_dir(&state).expect("state root"); + fs::create_dir(&actual).expect("actual root"); + symlink(&actual, state.join("checkpoints")).expect("link root"); + let store = CheckpointStore::new(StateStore::new(state)); + + assert!(store.begin(Uuid::new_v4()).is_err()); + } + + #[cfg(not(feature = "test-failpoints"))] + #[test] + fn production_checkpoint_store_boundary_hooks_are_inert() { + let temp = tempfile::tempdir().expect("tempdir"); + let store = store(&temp); + let sandbox_id = Uuid::new_v4(); + let root = publish(&store, sandbox_id, None, true); + + assert_eq!(store.read_head(sandbox_id).expect("HEAD"), Some(root)); + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn publish_boundary_error_leaves_a_committed_unreachable_checkpoint() { + let temp = tempfile::tempdir().expect("tempdir"); + let store = store(&temp); + let sandbox_id = Uuid::new_v4(); + let stage = store.begin(sandbox_id).expect("begin checkpoint"); + let checkpoint_id = stage.id().to_string(); + populate(&stage, "publish-boundary"); + let hook = crate::failpoint::TestFailpoint::new(&["checkpoint-store-publish-after-rename"]); + + let error = hook + .run(async { store.publish(&stage, commit_input(None)) }) + .await + .expect_err("publish boundary must return a store error"); + + assert!( + error + .to_string() + .contains("checkpoint-store-publish-after-rename") + ); + assert_eq!(error.outcome(), CheckpointPublishOutcome::Unknown); + store + .verify(sandbox_id, &checkpoint_id) + .expect("renamed checkpoint remains committed"); + assert_eq!(store.read_head(sandbox_id).expect("HEAD"), None); + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn publish_pre_rename_error_reports_a_known_unpublished_stage() { + let temp = tempfile::tempdir().expect("tempdir"); + let store = store(&temp); + let sandbox_id = Uuid::new_v4(); + let stage = store.begin(sandbox_id).expect("begin checkpoint"); + let checkpoint_id = stage.id().to_string(); + let staging_name = stage.staging_name.clone(); + populate(&stage, "pre-rename-boundary"); + let hook = + crate::failpoint::TestFailpoint::new(&["checkpoint-store-publish-before-rename"]); + + let error = hook + .run(async { store.publish(&stage, commit_input(None)) }) + .await + .expect_err("pre-rename boundary must return a store error"); + + assert_eq!(error.outcome(), CheckpointPublishOutcome::KnownUnpublished); + assert!( + error + .to_string() + .contains("checkpoint-store-publish-before-rename") + ); + let sandbox = store.configured_root().join(sandbox_id.to_string()); + assert!(sandbox.join(&staging_name).is_dir()); + assert!(sandbox.join(&staging_name).join(METADATA_FILE).is_file()); + assert!(!sandbox.join(checkpoint_id).exists()); + store + .abort(stage) + .expect("abort retained unpublished stage"); + assert!(!sandbox.join(staging_name).exists()); + } + + #[test] + fn publish_rename_error_reports_an_unknown_outcome() { + let temp = tempfile::tempdir().expect("tempdir"); + let store = store(&temp); + let sandbox_id = Uuid::new_v4(); + let stage = store.begin(sandbox_id).expect("begin checkpoint"); + let checkpoint_id = stage.id().to_string(); + let staging_name = stage.staging_name.clone(); + populate(&stage, "rename-collision"); + let sandbox = store.configured_root().join(sandbox_id.to_string()); + let target = sandbox.join(&checkpoint_id); + let collision = target.clone(); + store.set_before_publish_revalidation(move || { + fs::create_dir(&collision).expect("create publication collision"); + fs::write(collision.join("sentinel"), b"collision").expect("write collision sentinel"); + }); + + let error = store + .publish(&stage, commit_input(None)) + .expect_err("rename collision must fail publication"); + + assert_eq!(error.outcome(), CheckpointPublishOutcome::Unknown); + assert!(sandbox.join(staging_name).is_dir()); + assert_eq!( + fs::read(target.join("sentinel")).expect("read collision sentinel"), + b"collision" + ); + } + + #[test] + fn published_witness_sets_head_without_full_verification() { + let temp = tempfile::tempdir().expect("tempdir"); + let store = store(&temp); + let sandbox_id = Uuid::new_v4(); + let stage = store.begin(sandbox_id).expect("begin checkpoint"); + let checkpoint_id = stage.id().to_string(); + populate(&stage, "retained-publication"); + + let published = store + .publish_retained(&stage, commit_input(None)) + .expect("publish checkpoint with retained owners"); + assert_eq!(store.verified_checkpoint_count(), 0); + + let metadata = store + .set_head_published(published) + .expect("advance HEAD from published witness"); + assert_eq!(metadata.id, checkpoint_id); + assert_eq!( + store.verified_checkpoint_count(), + 0, + "the publication witness must avoid a second payload scan" + ); + let head = store + .configured_root() + .join(sandbox_id.to_string()) + .join(HEAD_FILE); + assert_eq!( + fs::read_to_string(head).expect("read HEAD").trim(), + checkpoint_id + ); + + store + .set_head(sandbox_id, &checkpoint_id) + .expect("public HEAD update performs full verification"); + assert_eq!(store.verified_checkpoint_count(), 1); + } + + #[test] + fn published_witness_rejects_replaced_artifact_identity() { + let temp = tempfile::tempdir().expect("tempdir"); + let store = store(&temp); + let sandbox_id = Uuid::new_v4(); + let existing_head = publish(&store, sandbox_id, None, true); + let stage = store.begin(sandbox_id).expect("begin checkpoint"); + let checkpoint_id = stage.id().to_string(); + populate(&stage, "replaced-after-publication"); + let published = store + .publish_retained(&stage, commit_input(Some(existing_head.clone()))) + .expect("publish checkpoint with retained owners"); + + let sandbox = store.configured_root().join(sandbox_id.to_string()); + let artifact = sandbox.join(&checkpoint_id).join("rootfs.snap"); + let displaced = sandbox.join("displaced-rootfs.snap"); + let bytes = fs::read(&artifact).expect("read published rootfs"); + fs::rename(&artifact, &displaced).expect("move retained rootfs"); + fs::write(&artifact, &bytes).expect("write same-content replacement"); + + let error = store + .set_head_published(published) + .expect_err("replacement must invalidate the publication witness"); + assert_eq!(error.outcome(), CheckpointHeadOutcome::KnownUnchanged); + assert!(error.to_string().contains("changed identity")); + assert_eq!( + fs::read_to_string(sandbox.join(HEAD_FILE)) + .expect("read unchanged HEAD") + .trim(), + existing_head + ); + assert_eq!(fs::read(&artifact).expect("read replacement rootfs"), bytes); + assert!(displaced.is_file()); + assert!( + fs::read_dir(&sandbox) + .expect("checkpoint sandbox") + .filter_map(std::result::Result::ok) + .all(|entry| !entry.file_name().to_string_lossy().starts_with(".HEAD.")), + "identity rejection must not leave temporary HEAD state" + ); + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn head_pre_rename_error_removes_scratch_and_reports_known_unchanged() { + let temp = tempfile::tempdir().expect("tempdir"); + let store = store(&temp); + let sandbox_id = Uuid::new_v4(); + let existing_head = publish(&store, sandbox_id, None, true); + let checkpoint_id = publish(&store, sandbox_id, Some(existing_head.clone()), false); + let hook = crate::failpoint::TestFailpoint::new(&["checkpoint-store-head-before-rename"]); + + let error = hook + .run(async { store.set_head(sandbox_id, &checkpoint_id) }) + .await + .expect_err("pre-rename HEAD boundary must fail"); + + assert_eq!(error.outcome(), CheckpointHeadOutcome::KnownUnchanged); + assert!( + error + .to_string() + .contains("checkpoint-store-head-before-rename") + ); + assert_eq!( + store.read_head(sandbox_id).expect("HEAD"), + Some(existing_head.clone()) + ); + let checkpoints = store.list(sandbox_id).expect("checkpoint catalog"); + assert_eq!(checkpoints.len(), 2); + assert!( + checkpoints + .iter() + .any(|checkpoint| checkpoint.id == existing_head && checkpoint.is_head) + ); + assert!( + checkpoints + .iter() + .any(|checkpoint| checkpoint.id == checkpoint_id && !checkpoint.is_head) + ); + let sandbox = store.configured_root().join(sandbox_id.to_string()); + assert!( + fs::read_dir(sandbox) + .expect("checkpoint sandbox") + .filter_map(std::result::Result::ok) + .all(|entry| !entry.file_name().to_string_lossy().starts_with(".HEAD.")), + "known-unchanged failure must remove its temporary HEAD" + ); + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn head_pre_rename_cleanup_error_reports_an_unknown_outcome() { + let temp = tempfile::tempdir().expect("tempdir"); + let store = store(&temp); + let sandbox_id = Uuid::new_v4(); + let checkpoint_id = publish(&store, sandbox_id, None, false); + let hook = crate::failpoint::TestFailpoint::new(&[ + "checkpoint-store-head-before-rename", + "checkpoint-store-head-cleanup", + ]); + + let error = hook + .run(async { store.set_head(sandbox_id, &checkpoint_id) }) + .await + .expect_err("failed pre-rename cleanup must be uncertain"); + + assert_eq!(error.outcome(), CheckpointHeadOutcome::Unknown); + assert!(error.to_string().contains("temporary HEAD cleanup failed")); + assert_eq!(store.read_head(sandbox_id).expect("HEAD"), None); + let sandbox = store.configured_root().join(sandbox_id.to_string()); + assert_eq!( + fs::read_dir(&sandbox) + .expect("checkpoint sandbox") + .filter_map(std::result::Result::ok) + .filter(|entry| entry.file_name().to_string_lossy().starts_with(".HEAD.")) + .count(), + 1, + "failed cleanup must remain observable for recovery" + ); + store.remove_sandbox(sandbox_id).expect("remove sandbox"); + assert!(!sandbox.exists()); + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn head_boundary_error_leaves_the_new_head_visible() { + let temp = tempfile::tempdir().expect("tempdir"); + let store = store(&temp); + let sandbox_id = Uuid::new_v4(); + let checkpoint_id = publish(&store, sandbox_id, None, false); + let hook = crate::failpoint::TestFailpoint::new(&["checkpoint-store-head-after-rename"]); + + let error = hook + .run(async { store.set_head(sandbox_id, &checkpoint_id) }) + .await + .expect_err("HEAD boundary must return a store error"); + + assert_eq!(error.outcome(), CheckpointHeadOutcome::Unknown); + assert!( + error + .to_string() + .contains("checkpoint-store-head-after-rename") + ); + assert_eq!( + store.read_head(sandbox_id).expect("HEAD"), + Some(checkpoint_id) + ); + } +} diff --git a/src/blaze/crates/blazed/src/error.rs b/src/blaze/crates/blazed/src/error.rs index 40015abb3..a35156c19 100644 --- a/src/blaze/crates/blazed/src/error.rs +++ b/src/blaze/crates/blazed/src/error.rs @@ -118,6 +118,7 @@ impl BlazeDaemonError { BlazeDaemonError::HttpStatus { status, .. } => *status, BlazeDaemonError::Core(blaze_core::BlazeError::PolicyEvalError { .. }) | BlazeDaemonError::Core(blaze_core::BlazeError::InvalidStateTransition { .. }) => 422, + BlazeDaemonError::Core(blaze_core::BlazeError::OperationInProgress { .. }) => 409, BlazeDaemonError::Core(blaze_core::BlazeError::BackendUnavailable { .. }) => 503, BlazeDaemonError::Guest(crate::guest::GuestError::InvalidArgument(_)) => 400, BlazeDaemonError::Guest(crate::guest::GuestError::Timeout(_)) => 504, diff --git a/src/blaze/crates/blazed/src/failpoint.rs b/src/blaze/crates/blazed/src/failpoint.rs index 15edb0bc9..31f9b4490 100644 --- a/src/blaze/crates/blazed/src/failpoint.rs +++ b/src/blaze/crates/blazed/src/failpoint.rs @@ -1,8 +1,6 @@ // SPDX-License-Identifier: Apache-2.0 //! Feature-gated fault hooks for daemon-level integration verification. -#![allow(dead_code)] // Call sites land with their owning lifecycle commits. - #[cfg(test)] use std::cell::RefCell; #[cfg(test)] @@ -17,9 +15,14 @@ use tokio::sync::Notify; const FAILPOINTS_ENV: &str = "BLAZE_TEST_FAILPOINTS"; const FAILPOINT_FILE_ENV: &str = "BLAZE_TEST_FAILPOINT_FILE"; +#[cfg(test)] +tokio::task_local! { + static TEST_FAILPOINTS: Option>; +} + #[cfg(test)] thread_local! { - static TEST_FAILPOINTS: RefCell>> = + static BLOCKING_TEST_FAILPOINTS: RefCell>> = const { RefCell::new(None) }; } @@ -40,14 +43,14 @@ pub(crate) struct TestFailpoint { } #[cfg(test)] -struct TestFailpointScope { +struct BlockingTestFailpointScope { previous: Option>, } #[cfg(test)] -impl Drop for TestFailpointScope { +impl Drop for BlockingTestFailpointScope { fn drop(&mut self) { - TEST_FAILPOINTS.with(|current| { + BLOCKING_TEST_FAILPOINTS.with(|current| { current.replace(self.previous.take()); }); } @@ -70,9 +73,9 @@ impl TestFailpoint { /// Run one future with this failpoint set in its test-thread context. pub(crate) async fn run(&self, future: F) -> F::Output { - let previous = TEST_FAILPOINTS.with(|current| current.replace(Some(self.state.clone()))); - let _scope = TestFailpointScope { previous }; - future.await + TEST_FAILPOINTS + .scope(Some(self.state.clone()), future) + .await } /// Wait until the scoped future reaches a pause failpoint. @@ -93,6 +96,52 @@ impl TestFailpoint { } } +/// Run blocking work while preserving the active unit-test failpoint context. +/// +/// Tokio's blocking pool uses different threads, so thread-local test hooks +/// would otherwise become invisible at filesystem durability boundaries. +pub(crate) fn spawn_blocking(operation: F) -> tokio::task::JoinHandle +where + F: FnOnce() -> R + Send + 'static, + R: Send + 'static, +{ + #[cfg(test)] + { + let context = task_test_context() + .or_else(|| BLOCKING_TEST_FAILPOINTS.with(|current| current.borrow().clone())); + tokio::task::spawn_blocking(move || { + let previous = BLOCKING_TEST_FAILPOINTS.with(|current| current.replace(context)); + let _scope = BlockingTestFailpointScope { previous }; + operation() + }) + } + + #[cfg(not(test))] + { + tokio::task::spawn_blocking(operation) + } +} + +// Preserve the active unit-test failpoint context in detached supervision. +#[cfg(test)] +pub(crate) fn spawn(future: F) -> tokio::task::JoinHandle +where + F: Future + Send + 'static, + R: Send + 'static, +{ + let context = task_test_context(); + tokio::spawn(TEST_FAILPOINTS.scope(context, future)) +} + +#[cfg(not(test))] +pub(crate) fn spawn(future: F) -> tokio::task::JoinHandle +where + F: std::future::Future + Send + 'static, + R: Send + 'static, +{ + tokio::spawn(future) +} + /// Log that a test-only binary is accepting failpoint configuration. pub(crate) fn announce() { tracing::warn!( @@ -169,6 +218,29 @@ pub(crate) async fn pause(name: &str) { } } +/// Hold a blocking durability operation at a test-only boundary. +pub(crate) fn pause_blocking(name: &str) { + #[cfg(test)] + if let Some(state) = test_state(name) { + state.paused.store(true, Ordering::Release); + state.paused_notify.notify_waiters(); + tracing::warn!(failpoint = name, "test failpoint paused"); + while !state.released.load(Ordering::Acquire) { + std::thread::sleep(std::time::Duration::from_millis(1)); + } + tracing::warn!(failpoint = name, "test failpoint released"); + return; + } + + if armed(name) { + tracing::warn!(failpoint = name, "test failpoint paused"); + while armed(name) { + std::thread::sleep(std::time::Duration::from_millis(10)); + } + tracing::warn!(failpoint = name, "test failpoint released"); + } +} + fn hit(name: &str) -> bool { if armed(name) { tracing::warn!(failpoint = name, "test failpoint triggered"); @@ -192,11 +264,19 @@ fn armed(name: &str) -> bool { #[cfg(test)] fn test_state(name: &str) -> Option> { - TEST_FAILPOINTS - .with(|current| current.borrow().clone()) + task_test_context() + .or_else(|| BLOCKING_TEST_FAILPOINTS.with(|current| current.borrow().clone())) .filter(|state| !state.released.load(Ordering::Acquire) && state.names.contains(&name)) } +#[cfg(test)] +fn task_test_context() -> Option> { + TEST_FAILPOINTS + .try_with(|current| current.clone()) + .ok() + .flatten() +} + fn configured(name: &str, inline: &str, file: &str) -> bool { inline .split(|character: char| character == ',' || character.is_whitespace()) @@ -207,7 +287,9 @@ fn configured(name: &str, inline: &str, file: &str) -> bool { #[cfg(test)] mod tests { - use super::configured; + use std::time::Duration; + + use super::{TestFailpoint, configured, pause, pause_blocking, spawn, spawn_blocking}; #[test] fn configuration_matches_complete_tokens_from_both_sources() { @@ -215,4 +297,72 @@ mod tests { assert!(configured("after-publish", "", "start\nafter-publish")); assert!(!configured("publish", "before-publish", "after-publish")); } + + #[tokio::test] + async fn blocking_failpoint_keeps_a_single_worker_runtime_responsive() { + let hook = TestFailpoint::new(&["blocking-runtime-heartbeat"]); + let task_hook = hook.clone(); + let task = tokio::spawn(async move { + task_hook + .run(async { + spawn_blocking(|| pause_blocking("blocking-runtime-heartbeat")) + .await + .expect("blocking failpoint task"); + }) + .await; + }); + hook.wait_until_paused().await; + + tokio::time::timeout( + Duration::from_millis(250), + tokio::time::sleep(Duration::from_millis(1)), + ) + .await + .expect("blocking work must not occupy the async runtime worker"); + + hook.release(); + task.await.expect("scoped blocking task"); + } + + #[tokio::test(flavor = "current_thread")] + async fn detached_spawn_keeps_failpoint_context_after_parent_abort() { + let hook = TestFailpoint::new(&["detached-child-context"]); + let (ready_tx, ready_rx) = tokio::sync::oneshot::channel(); + let (continue_tx, continue_rx) = tokio::sync::oneshot::channel(); + let (finished_tx, finished_rx) = tokio::sync::oneshot::channel(); + let parent_hook = hook.clone(); + let parent = tokio::spawn(async move { + parent_hook + .run(async move { + let child = spawn(async move { + let _ = ready_tx.send(()); + continue_rx.await.expect("continue detached child"); + pause("detached-child-context").await; + let _ = finished_tx.send(()); + }); + drop(child); + std::future::pending::<()>().await; + }) + .await; + }); + + ready_rx.await.expect("detached child started"); + parent.abort(); + assert!( + parent + .await + .expect_err("parent task must be cancelled") + .is_cancelled() + ); + continue_tx.send(()).expect("continue detached child"); + tokio::time::timeout(Duration::from_millis(250), hook.wait_until_paused()) + .await + .expect("detached child must retain its failpoint context"); + + hook.release(); + tokio::time::timeout(Duration::from_millis(250), finished_rx) + .await + .expect("detached child must finish after release") + .expect("detached child completion signal"); + } } diff --git a/src/blaze/crates/blazed/src/failpoint_disabled.rs b/src/blaze/crates/blazed/src/failpoint_disabled.rs index a43b1ef25..fe1374a06 100644 --- a/src/blaze/crates/blazed/src/failpoint_disabled.rs +++ b/src/blaze/crates/blazed/src/failpoint_disabled.rs @@ -1,8 +1,6 @@ // SPDX-License-Identifier: Apache-2.0 //! No-op hooks used when daemon verification support is disabled. -#![allow(dead_code)] // Call sites land with their owning lifecycle commits. - /// Keep daemon startup independent from verification-only configuration. pub(crate) fn announce() {} @@ -29,6 +27,27 @@ pub(crate) fn state(_name: &str) -> crate::error::Result<()> { /// Never pause production requests. pub(crate) async fn pause(_name: &str) {} +/// Run filesystem work on Tokio's blocking pool. +pub(crate) fn spawn_blocking(operation: F) -> tokio::task::JoinHandle +where + F: FnOnce() -> R + Send + 'static, + R: Send + 'static, +{ + tokio::task::spawn_blocking(operation) +} + +// Spawn detached supervision in production builds. +pub(crate) fn spawn(future: F) -> tokio::task::JoinHandle +where + F: std::future::Future + Send + 'static, + R: Send + 'static, +{ + tokio::spawn(future) +} + +/// Never pause production blocking operations. +pub(crate) fn pause_blocking(_name: &str) {} + #[cfg(test)] mod tests { #[tokio::test] @@ -39,5 +58,9 @@ mod tests { super::guest("any").expect("guest hook"); super::state("any").expect("state hook"); super::pause("any").await; + super::pause_blocking("any"); + super::spawn_blocking(|| ()) + .await + .expect("blocking operation"); } } diff --git a/src/blaze/crates/blazed/src/file_provider.rs b/src/blaze/crates/blazed/src/file_provider.rs index 51ba6fb75..29de4cfb6 100644 --- a/src/blaze/crates/blazed/src/file_provider.rs +++ b/src/blaze/crates/blazed/src/file_provider.rs @@ -3,11 +3,16 @@ //! rootfs and memory files on a local filesystem. Base images and mutable //! instance slots use separate roots. -use std::path::{Path, PathBuf}; +use std::ffi::OsString; +use std::os::fd::OwnedFd; +use std::path::{Component, Path, PathBuf}; use std::sync::Arc; use async_trait::async_trait; -use rustix::fs::{Mode, OFlags, open, openat}; +use rustix::fs::{ + AtFlags, Mode, OFlags, RenameFlags, fstat, fsync, open, openat, renameat_with, statat, unlinkat, +}; +use uuid::Uuid; use blaze_core::error::{BlazeError, Result}; use blaze_core::storage::{ @@ -27,6 +32,7 @@ pub struct FileStorageProvider { pub(crate) struct ArtifactSyncOpenHook { opened: tokio::sync::Notify, resume: tokio::sync::Notify, + capture_finished: tokio::sync::Notify, } #[cfg(test)] @@ -35,6 +41,7 @@ impl ArtifactSyncOpenHook { Self { opened: tokio::sync::Notify::new(), resume: tokio::sync::Notify::new(), + capture_finished: tokio::sync::Notify::new(), } } @@ -45,6 +52,25 @@ impl ArtifactSyncOpenHook { pub(crate) fn resume(&self) { self.resume.notify_one(); } + + #[cfg(feature = "test-failpoints")] + pub(crate) async fn wait_until_capture_finished(&self) { + self.capture_finished.notified().await; + } +} + +struct CaptureCompletion { + #[cfg(test)] + hook: Option>, +} + +impl CaptureCompletion { + fn finish(self) { + #[cfg(test)] + if let Some(hook) = self.hook { + hook.capture_finished.notify_one(); + } + } } impl FileStorageProvider { @@ -125,6 +151,126 @@ impl RequiredPathType { } } +struct UnpublishedCheckpoint { + parent: OwnedFd, + temporary_file: std::fs::File, + identity: Option, + temporary: OsString, + target: OsString, + committed: bool, +} + +impl UnpublishedCheckpoint { + fn new( + parent: OwnedFd, + temporary_file: std::fs::File, + temporary: OsString, + target: OsString, + ) -> Self { + Self { + parent, + temporary_file, + identity: None, + temporary, + target, + committed: false, + } + } + + fn parent(&self) -> &OwnedFd { + &self.parent + } + + fn temporary_file(&self) -> &std::fs::File { + &self.temporary_file + } + + fn retain_identity(&mut self) -> std::io::Result<()> { + let stat = fstat(&self.temporary_file).map_err(std::io::Error::from)?; + self.identity = Some(stat); + Ok(()) + } + + fn candidate_matches(&self, name: &std::ffi::OsStr) -> std::io::Result { + let identity = self + .identity + .as_ref() + .ok_or_else(|| std::io::Error::other("checkpoint temporary identity is unavailable"))?; + match statat(&self.parent, name, AtFlags::SYMLINK_NOFOLLOW) { + Ok(stat) => Ok(stat.st_dev == identity.st_dev && stat.st_ino == identity.st_ino), + Err(rustix::io::Errno::NOENT) => Ok(false), + Err(error) => Err(std::io::Error::from(error)), + } + } + + fn require_temporary_identity(&self) -> std::io::Result<()> { + if self.candidate_matches(&self.temporary)? { + return Ok(()); + } + Err(std::io::Error::other( + "checkpoint temporary file changed identity before publication", + )) + } + + fn publish_noreplace(&self) -> std::io::Result<()> { + self.require_temporary_identity()?; + let rename_error = renameat_with( + &self.parent, + &self.temporary, + &self.parent, + &self.target, + RenameFlags::NOREPLACE, + ) + .err() + .map(std::io::Error::from); + + let temporary_matches = self.candidate_matches(&self.temporary)?; + let target_matches = self.candidate_matches(&self.target)?; + match (temporary_matches, target_matches, rename_error) { + (false, true, _) => Ok(()), + (true, false, Some(error)) => Err(error), + (true, false, None) => Err(std::io::Error::other( + "checkpoint rename reported success but retained the temporary name", + )), + (false, false, Some(error)) => Err(std::io::Error::other(format!( + "checkpoint rename failed and the retained file lost both candidate names: {error}" + ))), + (false, false, None) => Err(std::io::Error::other( + "checkpoint rename reported success but the retained file lost both candidate names", + )), + (true, true, Some(error)) => Err(std::io::Error::other(format!( + "checkpoint rename failed with both candidate names linked to the retained file: {error}" + ))), + (true, true, None) => Err(std::io::Error::other( + "checkpoint rename reported success with both candidate names linked to the retained file", + )), + } + } + + fn commit(&mut self) { + self.committed = true; + } +} + +impl Drop for UnpublishedCheckpoint { + fn drop(&mut self) { + if self.committed { + return; + } + let mut removed = false; + for name in [&self.temporary, &self.target] { + if self.candidate_matches(name).unwrap_or(false) + && unlinkat(&self.parent, name, AtFlags::empty()).is_ok() + { + removed = true; + } + } + if removed { + let _ = fsync(&self.parent); + } + } +} + async fn require_slot_path( instance_id: &str, path: &Path, @@ -346,6 +492,47 @@ impl StorageProvider for FileStorageProvider { Ok(()) } + fn supports_checkpoint_capture(&self) -> bool { + true + } + + async fn capture_checkpoint(&self, slot: &StorageSlot, target: &Path) -> Result<()> { + let (source, source_path) = self.checkpoint_source(slot).await?; + let (target_parent, target) = checkpoint_target(target).await?; + let target_parent_owner = open_checkpoint_target_parent(&target_parent, &target).await?; + #[cfg(test)] + if let Some(hook) = &self.artifact_sync_open_hook { + hook.opened.notify_one(); + hook.resume.notified().await; + } + + let target_name = target + .file_name() + .expect("validated checkpoint target") + .to_os_string(); + let temporary_name = checkpoint_temporary_name(&target); + let completion = CaptureCompletion { + #[cfg(test)] + hook: self.artifact_sync_open_hook.clone(), + }; + let result = capture_rootfs( + source, + target_parent_owner, + temporary_name, + target_name, + completion, + ) + .await; + result.map_err(|error| BlazeError::StorageError { + msg: format!( + "capture checkpoint for '{}': copy {} to {}: {error}", + slot.id, + source_path.display(), + target.display() + ), + }) + } + fn pool_status(&self) -> PoolStatus { PoolStatus::default() } @@ -409,6 +596,46 @@ where })? } +impl FileStorageProvider { + async fn checkpoint_source(&self, slot: &StorageSlot) -> Result<(tokio::fs::File, PathBuf)> { + let canonical = self.slot_for_id(&slot.id)?; + let instance_path = canonical.instance_dir.clone(); + let directory = open_required_slot_path( + &slot.id, + &canonical.instance_dir, + RequiredPathType::Directory, + move || { + open( + &instance_path, + OFlags::RDONLY | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC, + Mode::empty(), + ) + }, + ) + .await?; + let directory = Arc::new(directory); + let open_directory = Arc::clone(&directory); + let source = open_required_slot_path( + &slot.id, + &canonical.rootfs_path, + RequiredPathType::File, + move || { + openat( + &*open_directory, + "rootfs.ext4", + OFlags::RDONLY | OFlags::NOFOLLOW | OFlags::CLOEXEC | OFlags::NONBLOCK, + Mode::empty(), + ) + }, + ) + .await?; + Ok(( + tokio::fs::File::from_std(std::fs::File::from(source)), + canonical.rootfs_path, + )) + } +} + async fn create_or_copy( source: &std::path::Path, target: &std::path::Path, @@ -425,6 +652,311 @@ async fn create_or_copy( Ok(()) } +async fn canonical_plain_path(path: &Path, required_type: RequiredPathType) -> Result { + let metadata = + tokio::fs::symlink_metadata(path) + .await + .map_err(|error| BlazeError::StorageError { + msg: format!("inspect checkpoint path {}: {error}", path.display()), + })?; + if !required_type.matches(&metadata) || metadata.file_type().is_symlink() { + return Err(BlazeError::StorageError { + msg: format!( + "checkpoint path {} is not a plain {}", + path.display(), + required_type.description() + ), + }); + } + tokio::fs::canonicalize(path) + .await + .map_err(|error| BlazeError::StorageError { + msg: format!("canonicalize checkpoint path {}: {error}", path.display()), + }) +} + +async fn checkpoint_target(target: &Path) -> Result<(PathBuf, PathBuf)> { + if !matches!(target.components().next_back(), Some(Component::Normal(_))) { + return Err(BlazeError::StorageError { + msg: format!( + "checkpoint target {} must end in a file name", + target.display() + ), + }); + } + let parent = target.parent().ok_or_else(|| BlazeError::StorageError { + msg: format!( + "checkpoint target {} has no parent directory", + target.display() + ), + })?; + let parent = if is_retained_directory_adapter(parent) { + let metadata = + tokio::fs::metadata(parent) + .await + .map_err(|error| BlazeError::StorageError { + msg: format!( + "inspect retained checkpoint directory {}: {error}", + parent.display() + ), + })?; + if !metadata.is_dir() { + return Err(BlazeError::StorageError { + msg: format!( + "retained checkpoint path {} is not a directory", + parent.display() + ), + }); + } + parent.to_path_buf() + } else { + canonical_plain_path(parent, RequiredPathType::Directory).await? + }; + let file_name = target.file_name().ok_or_else(|| BlazeError::StorageError { + msg: format!("checkpoint target {} has no file name", target.display()), + })?; + let target = parent.join(file_name); + Ok((parent, target)) +} + +#[cfg(target_os = "linux")] +fn is_retained_directory_adapter(path: &Path) -> bool { + path.parent() == Some(Path::new("/proc/self/fd")) + && path + .file_name() + .and_then(|name| name.to_str()) + .map(|name| !name.is_empty() && name.bytes().all(|byte| byte.is_ascii_digit())) + .unwrap_or(false) +} + +#[cfg(not(target_os = "linux"))] +fn is_retained_directory_adapter(_path: &Path) -> bool { + false +} + +async fn open_checkpoint_target_parent(parent: &Path, target: &Path) -> Result { + let parent_path = parent.to_path_buf(); + let target_path = target.to_path_buf(); + let target_name = target + .file_name() + .expect("validated checkpoint target") + .to_os_string(); + let follow_retained_adapter = is_retained_directory_adapter(parent); + crate::failpoint::spawn_blocking(move || { + let base_flags = OFlags::RDONLY | OFlags::DIRECTORY | OFlags::CLOEXEC; + let flags = if follow_retained_adapter { + base_flags + } else { + base_flags | OFlags::NOFOLLOW + }; + let parent_owner = + open(&parent_path, flags, Mode::empty()).map_err(|error| BlazeError::StorageError { + msg: format!( + "open checkpoint target directory {}: {error}", + parent_path.display() + ), + })?; + match statat(&parent_owner, &target_name, AtFlags::SYMLINK_NOFOLLOW) { + Ok(_) => Err(BlazeError::StorageError { + msg: format!("checkpoint target {} already exists", target_path.display()), + }), + Err(rustix::io::Errno::NOENT) => Ok(parent_owner), + Err(error) => Err(BlazeError::StorageError { + msg: format!( + "inspect checkpoint target {}: {error}", + target_path.display() + ), + }), + } + }) + .await + .map_err(|error| BlazeError::StorageError { + msg: format!( + "open checkpoint target directory {}: blocking task failed: {error}", + parent.display() + ), + })? +} + +fn checkpoint_temporary_name(target: &Path) -> OsString { + let mut name = OsString::from("."); + name.push(target.file_name().expect("validated checkpoint target")); + name.push(format!(".capture-{}.tmp", Uuid::new_v4())); + name +} + +async fn capture_rootfs( + source_file: tokio::fs::File, + target_parent: OwnedFd, + temporary_name: OsString, + target_name: OsString, + completion: CaptureCompletion, +) -> std::io::Result<()> { + let source_file = source_file.into_std().await; + crate::failpoint::spawn_blocking(move || { + let result = (|| { + if !source_file.metadata()?.is_file() { + return Err(std::io::Error::other( + "checkpoint source owner is not a regular file", + )); + } + let temporary_file = openat( + &target_parent, + &temporary_name, + OFlags::WRONLY + | OFlags::CREATE + | OFlags::EXCL + | OFlags::NOFOLLOW + | OFlags::CLOEXEC + | OFlags::NONBLOCK, + Mode::RUSR.union(Mode::WUSR), + ) + .map(std::fs::File::from) + .map_err(std::io::Error::from)?; + let mut cleanup = UnpublishedCheckpoint::new( + target_parent, + temporary_file, + temporary_name, + target_name, + ); + cleanup.retain_identity()?; + copy_sparse_file(&source_file, cleanup.temporary_file())?; + cleanup.temporary_file().sync_all()?; + crate::failpoint::pause_blocking("storage-capture-before-publish"); + cleanup.publish_noreplace()?; + crate::failpoint::storage("storage-capture-after-publish") + .map_err(|error| std::io::Error::other(error.to_string()))?; + fsync(cleanup.parent()).map_err(std::io::Error::from)?; + cleanup.commit(); + Ok(()) + })(); + completion.finish(); + result + }) + .await + .map_err(|error| std::io::Error::other(format!("checkpoint capture task failed: {error}")))? +} + +fn copy_sparse_file(source: &std::fs::File, target: &std::fs::File) -> std::io::Result<()> { + copy_sparse_file_with_seek(source, target, |file, position| { + rustix::fs::seek(file, position) + }) +} + +fn copy_sparse_file_with_seek( + source: &std::fs::File, + target: &std::fs::File, + mut seek: F, +) -> std::io::Result<()> +where + F: FnMut(&std::fs::File, rustix::fs::SeekFrom) -> std::result::Result, +{ + const COPY_BUFFER_SIZE: usize = 64 * 1024; + + let logical_len = source.metadata()?.len(); + let mut position = 0_u64; + let mut buffer = [0_u8; COPY_BUFFER_SIZE]; + + while position < logical_len { + let data = match seek(source, rustix::fs::SeekFrom::Data(position)) { + Ok(data) => data, + Err(rustix::io::Errno::NXIO) => break, + Err(error) if sparse_seek_is_unsupported(error) => { + return copy_sparse_file_by_scanning(source, target, logical_len); + } + Err(error) => return Err(error.into()), + }; + if data >= logical_len { + break; + } + let hole = match seek(source, rustix::fs::SeekFrom::Hole(data)) { + Ok(hole) => hole.min(logical_len), + Err(error) if sparse_seek_is_unsupported(error) => { + return copy_sparse_file_by_scanning(source, target, logical_len); + } + Err(error) => return Err(error.into()), + }; + if hole <= data { + return Err(std::io::Error::other(format!( + "invalid sparse extent {data}..{hole} for file length {logical_len}" + ))); + } + + let mut offset = data; + while offset < hole { + let remaining = hole - offset; + let requested = usize::try_from(remaining.min(COPY_BUFFER_SIZE as u64)) + .map_err(|_| std::io::Error::other("sparse extent exceeds platform limits"))?; + let read = rustix::io::pread(source, &mut buffer[..requested], offset) + .map_err(std::io::Error::from)?; + if read == 0 { + return Err(std::io::Error::new( + std::io::ErrorKind::UnexpectedEof, + format!("sparse extent ended before offset {hole}"), + )); + } + write_all_at(target, &buffer[..read], offset)?; + offset += read as u64; + } + position = hole; + } + + rustix::fs::ftruncate(target, logical_len).map_err(std::io::Error::from) +} + +fn sparse_seek_is_unsupported(error: rustix::io::Errno) -> bool { + error == rustix::io::Errno::INVAL || error == rustix::io::Errno::NOTSUP +} + +fn copy_sparse_file_by_scanning( + source: &std::fs::File, + target: &std::fs::File, + logical_len: u64, +) -> std::io::Result<()> { + const COPY_BUFFER_SIZE: usize = 64 * 1024; + + // A seek implementation may report unsupported after earlier extents were + // copied. Reset the private temporary file before rebuilding it so skipped + // zero blocks cannot retain stale bytes from that partial attempt. + rustix::fs::ftruncate(target, 0).map_err(std::io::Error::from)?; + let mut buffer = [0_u8; COPY_BUFFER_SIZE]; + let mut offset = 0_u64; + while offset < logical_len { + let remaining = logical_len - offset; + let requested = usize::try_from(remaining.min(COPY_BUFFER_SIZE as u64)) + .map_err(|_| std::io::Error::other("checkpoint file exceeds platform limits"))?; + let read = rustix::io::pread(source, &mut buffer[..requested], offset) + .map_err(std::io::Error::from)?; + if read == 0 { + return Err(std::io::Error::new( + std::io::ErrorKind::UnexpectedEof, + format!("checkpoint source ended before offset {logical_len}"), + )); + } + if buffer[..read].iter().any(|byte| *byte != 0) { + write_all_at(target, &buffer[..read], offset)?; + } + offset += read as u64; + } + rustix::fs::ftruncate(target, logical_len).map_err(std::io::Error::from) +} + +fn write_all_at(target: &std::fs::File, buffer: &[u8], offset: u64) -> std::io::Result<()> { + let mut written = 0; + while written < buffer.len() { + let count = rustix::io::pwrite(target, &buffer[written..], offset + written as u64) + .map_err(std::io::Error::from)?; + if count == 0 { + return Err(std::io::Error::new( + std::io::ErrorKind::WriteZero, + "failed to write sparse checkpoint extent", + )); + } + written += count; + } + Ok(()) +} + fn validate_instance_id(instance_id: &str) -> Result<()> { if instance_id.is_empty() || instance_id.contains('/') @@ -444,6 +976,26 @@ fn validate_instance_id(instance_id: &str) -> Result<()> { mod tests { use super::*; + async fn checkpoint_fixture( + instance_id: &str, + ) -> (tempfile::TempDir, FileStorageProvider, StorageSlot, PathBuf) { + let temp = tempfile::TempDir::new().unwrap(); + let instances = temp.path().join("instances"); + let checkpoints = temp.path().join("checkpoints"); + tokio::fs::create_dir(&instances).await.unwrap(); + tokio::fs::create_dir(&checkpoints).await.unwrap(); + let provider = FileStorageProvider::new(instances); + let slot = provider + .acquire(&AcquireOpts { + instance_id: instance_id.to_string(), + rootfs_size: 64, + mem_size: 32, + }) + .await + .unwrap(); + (temp, provider, slot, checkpoints) + } + #[tokio::test] async fn probe_existing_dir_returns_true() { let tmp = tempfile::TempDir::new().unwrap(); @@ -847,4 +1399,695 @@ mod tests { .expect_err("missing artifact must fail the sweep item"); assert!(error.to_string().contains("mem.diff"), "{error}"); } + + #[tokio::test] + async fn checkpoint_capture_is_explicit_and_independent() { + let (_temp, provider, slot, checkpoints) = checkpoint_fixture("capture-independent").await; + tokio::fs::write(&slot.rootfs_path, b"captured-rootfs") + .await + .unwrap(); + let target = checkpoints.join("rootfs.snap"); + + assert!(provider.supports_checkpoint_capture()); + provider.capture_checkpoint(&slot, &target).await.unwrap(); + tokio::fs::write(&slot.rootfs_path, b"changed-live-rootfs") + .await + .unwrap(); + + assert_eq!(tokio::fs::read(&target).await.unwrap(), b"captured-rootfs"); + } + + #[tokio::test] + async fn checkpoint_capture_does_not_replace_the_live_rootfs() { + let (_temp, provider, slot, checkpoints) = checkpoint_fixture("capture-read-only").await; + tokio::fs::write(&slot.rootfs_path, b"live-rootfs") + .await + .unwrap(); + + provider + .capture_checkpoint(&slot, &checkpoints.join("rootfs.snap")) + .await + .unwrap(); + + assert_eq!( + tokio::fs::read(&slot.rootfs_path).await.unwrap(), + b"live-rootfs" + ); + } + + #[tokio::test] + async fn checkpoint_capture_ignores_forged_slot_paths() { + let (temp, provider, slot, checkpoints) = checkpoint_fixture("capture-canonical").await; + tokio::fs::write(&slot.rootfs_path, b"canonical-rootfs") + .await + .unwrap(); + let forged_source = temp.path().join("forged-rootfs"); + tokio::fs::write(&forged_source, b"forged-rootfs") + .await + .unwrap(); + let mut forged = slot.clone(); + forged.rootfs_path = forged_source; + forged.mem_path = temp.path().join("forged-memory"); + forged.mem_diff_path = temp.path().join("forged-memory-diff"); + forged.rootfs_diff_path = temp.path().join("forged-rootfs-diff"); + forged.instance_dir = temp.path().to_path_buf(); + let target = checkpoints.join("rootfs.snap"); + + provider.capture_checkpoint(&forged, &target).await.unwrap(); + + assert_eq!(tokio::fs::read(&target).await.unwrap(), b"canonical-rootfs"); + } + + #[tokio::test] + async fn checkpoint_capture_retains_the_opened_source_artifact() { + let temp = tempfile::TempDir::new().unwrap(); + let instances = temp.path().join("instances"); + let checkpoints = temp.path().join("checkpoints"); + tokio::fs::create_dir(&instances).await.unwrap(); + tokio::fs::create_dir(&checkpoints).await.unwrap(); + let hook = Arc::new(ArtifactSyncOpenHook::new()); + let provider = Arc::new(FileStorageProvider::with_artifact_sync_open_hook( + instances.clone(), + instances, + Arc::clone(&hook), + )); + let slot = provider + .acquire(&AcquireOpts { + instance_id: "capture-source-owner".into(), + rootfs_size: 64, + mem_size: 32, + }) + .await + .unwrap(); + tokio::fs::write(&slot.rootfs_path, b"opened-source") + .await + .unwrap(); + let target = checkpoints.join("rootfs.snap"); + + let capture_provider = Arc::clone(&provider); + let capture_slot = slot.clone(); + let capture_target = target.clone(); + let capture = tokio::spawn(async move { + capture_provider + .capture_checkpoint(&capture_slot, &capture_target) + .await + }); + hook.wait_until_open().await; + let retained = slot.instance_dir.join("retained-rootfs.ext4"); + tokio::fs::rename(&slot.rootfs_path, &retained) + .await + .unwrap(); + tokio::fs::write(&slot.rootfs_path, b"replacement-source") + .await + .unwrap(); + hook.resume(); + + capture.await.unwrap().unwrap(); + assert_eq!(tokio::fs::read(&target).await.unwrap(), b"opened-source"); + assert_eq!( + tokio::fs::read(&slot.rootfs_path).await.unwrap(), + b"replacement-source" + ); + } + + #[cfg(target_os = "linux")] + #[test] + fn checkpoint_sparse_copy_falls_back_when_extent_seeks_are_unsupported() { + use std::io::{Read, Seek, Write}; + use std::os::unix::fs::MetadataExt; + + const LOGICAL_LEN: u64 = 64 * 1024 * 1024; + const FIRST_OFFSET: u64 = 8 * 1024; + const LAST_OFFSET: u64 = 48 * 1024 * 1024 + 91; + const FIRST_DATA: &[u8] = b"portable-first-extent"; + const LAST_DATA: &[u8] = b"portable-last-extent"; + + let temp = tempfile::tempdir().expect("temp"); + let source_path = temp.path().join("source.img"); + let mut source = std::fs::OpenOptions::new() + .read(true) + .write(true) + .create_new(true) + .open(&source_path) + .expect("source"); + source.set_len(LOGICAL_LEN).expect("logical source length"); + source + .seek(std::io::SeekFrom::Start(FIRST_OFFSET)) + .expect("first offset"); + source.write_all(FIRST_DATA).expect("first data"); + source + .seek(std::io::SeekFrom::Start(LAST_OFFSET)) + .expect("last offset"); + source.write_all(LAST_DATA).expect("last data"); + source.sync_all().expect("source sync"); + + for (name, unsupported) in [ + ("invalid", rustix::io::Errno::INVAL), + ("not-supported", rustix::io::Errno::NOTSUP), + ] { + let target_path = temp.path().join(format!("target-{name}.img")); + let target = std::fs::OpenOptions::new() + .read(true) + .write(true) + .create_new(true) + .open(&target_path) + .expect("target"); + copy_sparse_file_with_seek(&source, &target, |_, _| Err(unsupported)) + .expect("portable sparse copy"); + target.sync_all().expect("target sync"); + + let metadata = target.metadata().expect("target metadata"); + assert_eq!(metadata.len(), LOGICAL_LEN); + assert!( + metadata.blocks().saturating_mul(512) < LOGICAL_LEN / 4, + "fallback allocated {} bytes for a {LOGICAL_LEN}-byte sparse source", + metadata.blocks().saturating_mul(512) + ); + + let mut captured = std::fs::File::open(&target_path).expect("captured target"); + let mut first = vec![0; FIRST_DATA.len()]; + captured + .seek(std::io::SeekFrom::Start(FIRST_OFFSET)) + .expect("captured first offset"); + captured + .read_exact(&mut first) + .expect("captured first data"); + assert_eq!(first, FIRST_DATA); + let mut last = vec![0; LAST_DATA.len()]; + captured + .seek(std::io::SeekFrom::Start(LAST_OFFSET)) + .expect("captured last offset"); + captured.read_exact(&mut last).expect("captured last data"); + assert_eq!(last, LAST_DATA); + let mut hole = [1_u8; 4096]; + captured + .seek(std::io::SeekFrom::Start(24 * 1024 * 1024)) + .expect("captured hole offset"); + captured.read_exact(&mut hole).expect("captured hole"); + assert!(hole.iter().all(|byte| *byte == 0)); + } + } + + #[cfg(target_os = "linux")] + #[tokio::test] + async fn checkpoint_capture_preserves_sparse_extents() { + use std::io::{Read, Seek, Write}; + use std::os::unix::fs::MetadataExt; + + const LOGICAL_LEN: u64 = 64 * 1024 * 1024; + const FIRST_OFFSET: u64 = 4 * 1024; + const LAST_OFFSET: u64 = 48 * 1024 * 1024 + 137; + const FIRST_DATA: &[u8] = b"first-checkpoint-extent"; + const LAST_DATA: &[u8] = b"last-checkpoint-extent"; + + let (_temp, provider, slot, checkpoints) = checkpoint_fixture("capture-sparse").await; + let mut source = std::fs::OpenOptions::new() + .write(true) + .open(&slot.rootfs_path) + .unwrap(); + source.set_len(LOGICAL_LEN).unwrap(); + source.seek(std::io::SeekFrom::Start(FIRST_OFFSET)).unwrap(); + source.write_all(FIRST_DATA).unwrap(); + source.seek(std::io::SeekFrom::Start(LAST_OFFSET)).unwrap(); + source.write_all(LAST_DATA).unwrap(); + source.sync_all().unwrap(); + let source_blocks = source.metadata().unwrap().blocks(); + drop(source); + + let target = checkpoints.join("rootfs.snap"); + provider.capture_checkpoint(&slot, &target).await.unwrap(); + + let metadata = std::fs::metadata(&target).unwrap(); + assert_eq!(metadata.len(), LOGICAL_LEN); + assert!( + metadata.blocks().saturating_mul(512) < LOGICAL_LEN / 4, + "checkpoint allocated {} bytes for a {LOGICAL_LEN}-byte sparse source", + metadata.blocks().saturating_mul(512) + ); + assert!( + metadata.blocks() <= source_blocks.saturating_add(32), + "checkpoint used {} blocks for a source using {source_blocks} blocks", + metadata.blocks() + ); + + let mut live = std::fs::OpenOptions::new() + .write(true) + .open(&slot.rootfs_path) + .unwrap(); + live.seek(std::io::SeekFrom::Start(FIRST_OFFSET)).unwrap(); + live.write_all(&[b'x'; FIRST_DATA.len()]).unwrap(); + live.sync_all().unwrap(); + + let mut captured = std::fs::File::open(&target).unwrap(); + let mut first = vec![0; FIRST_DATA.len()]; + captured + .seek(std::io::SeekFrom::Start(FIRST_OFFSET)) + .unwrap(); + captured.read_exact(&mut first).unwrap(); + assert_eq!(first, FIRST_DATA); + let mut last = vec![0; LAST_DATA.len()]; + captured + .seek(std::io::SeekFrom::Start(LAST_OFFSET)) + .unwrap(); + captured.read_exact(&mut last).unwrap(); + assert_eq!(last, LAST_DATA); + let mut hole = [1_u8; 4096]; + captured + .seek(std::io::SeekFrom::Start(16 * 1024 * 1024)) + .unwrap(); + captured.read_exact(&mut hole).unwrap(); + assert!(hole.iter().all(|byte| *byte == 0)); + } + + #[cfg(target_os = "linux")] + #[tokio::test] + async fn checkpoint_capture_preserves_an_all_hole_rootfs() { + use std::io::{Read, Seek}; + use std::os::unix::fs::MetadataExt; + + const LOGICAL_LEN: u64 = 64 * 1024 * 1024; + + let (_temp, provider, slot, checkpoints) = checkpoint_fixture("capture-all-hole").await; + let source = std::fs::OpenOptions::new() + .write(true) + .open(&slot.rootfs_path) + .unwrap(); + source.set_len(LOGICAL_LEN).unwrap(); + source.sync_all().unwrap(); + let source_blocks = source.metadata().unwrap().blocks(); + drop(source); + + let target = checkpoints.join("rootfs.snap"); + provider.capture_checkpoint(&slot, &target).await.unwrap(); + + let metadata = std::fs::metadata(&target).unwrap(); + assert_eq!(metadata.len(), LOGICAL_LEN); + assert!( + metadata.blocks().saturating_mul(512) < LOGICAL_LEN / 16, + "all-hole checkpoint allocated {} bytes", + metadata.blocks().saturating_mul(512) + ); + assert!( + metadata.blocks() <= source_blocks.saturating_add(8), + "all-hole checkpoint used {} blocks for a source using {source_blocks} blocks", + metadata.blocks() + ); + + let mut captured = std::fs::File::open(&target).unwrap(); + let mut zeros = [1_u8; 4096]; + captured + .seek(std::io::SeekFrom::Start(LOGICAL_LEN / 2)) + .unwrap(); + captured.read_exact(&mut zeros).unwrap(); + assert!(zeros.iter().all(|byte| *byte == 0)); + } + + #[cfg(target_os = "linux")] + #[tokio::test] + async fn checkpoint_capture_retains_the_opened_target_directory() { + use std::os::fd::AsRawFd; + + let temp = tempfile::TempDir::new().unwrap(); + let instances = temp.path().join("instances"); + let checkpoints = temp.path().join("checkpoints"); + tokio::fs::create_dir(&instances).await.unwrap(); + tokio::fs::create_dir(&checkpoints).await.unwrap(); + let checkpoint_owner = open( + &checkpoints, + OFlags::RDONLY | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC, + Mode::empty(), + ) + .unwrap(); + let stable_parent = + PathBuf::from(format!("/proc/self/fd/{}", checkpoint_owner.as_raw_fd())); + let hook = Arc::new(ArtifactSyncOpenHook::new()); + let provider = Arc::new(FileStorageProvider::with_artifact_sync_open_hook( + instances.clone(), + instances, + Arc::clone(&hook), + )); + let slot = provider + .acquire(&AcquireOpts { + instance_id: "capture-target-owner".into(), + rootfs_size: 64, + mem_size: 32, + }) + .await + .unwrap(); + tokio::fs::write(&slot.rootfs_path, b"retained-target") + .await + .unwrap(); + let target = stable_parent.join("rootfs.snap"); + + let capture_provider = Arc::clone(&provider); + let capture_slot = slot.clone(); + let capture = tokio::spawn(async move { + capture_provider + .capture_checkpoint(&capture_slot, &target) + .await + }); + hook.wait_until_open().await; + let retained = temp.path().join("retained-checkpoints"); + tokio::fs::rename(&checkpoints, &retained).await.unwrap(); + tokio::fs::create_dir(&checkpoints).await.unwrap(); + hook.resume(); + + capture.await.unwrap().unwrap(); + assert_eq!( + tokio::fs::read(retained.join("rootfs.snap")).await.unwrap(), + b"retained-target" + ); + assert!(!checkpoints.join("rootfs.snap").exists()); + } + + #[cfg(unix)] + #[tokio::test] + async fn checkpoint_capture_rejects_a_linked_rootfs() { + use std::os::unix::fs::symlink; + + let (temp, provider, slot, checkpoints) = checkpoint_fixture("capture-linked-source").await; + tokio::fs::remove_file(&slot.rootfs_path).await.unwrap(); + let external = temp.path().join("external-rootfs"); + tokio::fs::write(&external, b"external").await.unwrap(); + symlink(&external, &slot.rootfs_path).unwrap(); + let target = checkpoints.join("rootfs.snap"); + + provider + .capture_checkpoint(&slot, &target) + .await + .expect_err("linked rootfs must not be captured"); + + assert!(!target.exists()); + assert_eq!(tokio::fs::read(external).await.unwrap(), b"external"); + } + + #[cfg(unix)] + #[tokio::test] + async fn checkpoint_capture_rejects_a_linked_slot_directory() { + use std::os::unix::fs::symlink; + + let (temp, provider, slot, checkpoints) = checkpoint_fixture("capture-linked-slot").await; + tokio::fs::remove_dir_all(&slot.instance_dir).await.unwrap(); + let external = temp.path().join("external-slot"); + tokio::fs::create_dir(&external).await.unwrap(); + tokio::fs::write(external.join("rootfs.ext4"), b"external") + .await + .unwrap(); + symlink(&external, &slot.instance_dir).unwrap(); + let target = checkpoints.join("rootfs.snap"); + + provider + .capture_checkpoint(&slot, &target) + .await + .expect_err("linked slot directory must be rejected"); + + assert!(!target.exists()); + assert_eq!( + tokio::fs::read(external.join("rootfs.ext4")).await.unwrap(), + b"external" + ); + } + + #[cfg(unix)] + #[tokio::test] + async fn checkpoint_capture_rejects_a_linked_target_parent() { + use std::os::unix::fs::symlink; + + let temp = tempfile::TempDir::new().unwrap(); + let instances = temp.path().join("instances"); + let external = temp.path().join("external-checkpoints"); + tokio::fs::create_dir(&instances).await.unwrap(); + tokio::fs::create_dir(&external).await.unwrap(); + let linked_parent = temp.path().join("linked-checkpoints"); + symlink(&external, &linked_parent).unwrap(); + let provider = FileStorageProvider::new(instances); + let slot = provider + .acquire(&AcquireOpts { + instance_id: "capture-linked-parent".into(), + rootfs_size: 64, + mem_size: 32, + }) + .await + .unwrap(); + let target = linked_parent.join("rootfs.snap"); + + provider + .capture_checkpoint(&slot, &target) + .await + .expect_err("linked target parent must be rejected"); + + assert!(!external.join("rootfs.snap").exists()); + } + + #[tokio::test] + async fn checkpoint_capture_preserves_an_existing_target() { + let (_temp, provider, slot, checkpoints) = + checkpoint_fixture("capture-existing-target").await; + tokio::fs::write(&slot.rootfs_path, b"new-checkpoint") + .await + .unwrap(); + let target = checkpoints.join("rootfs.snap"); + tokio::fs::write(&target, b"existing-checkpoint") + .await + .unwrap(); + + provider + .capture_checkpoint(&slot, &target) + .await + .expect_err("capture must never replace an existing target"); + + assert_eq!( + tokio::fs::read(&target).await.unwrap(), + b"existing-checkpoint" + ); + } + + #[test] + fn unpublished_checkpoint_cleans_target_after_an_unreported_rename() { + let temp = tempfile::TempDir::new().unwrap(); + let temporary_name = OsString::from("temporary"); + let target_name = OsString::from("target"); + let parent = open( + temp.path(), + OFlags::RDONLY | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC, + Mode::empty(), + ) + .unwrap(); + let temporary_file = openat( + &parent, + &temporary_name, + OFlags::WRONLY | OFlags::CREATE | OFlags::EXCL | OFlags::NOFOLLOW | OFlags::CLOEXEC, + Mode::RUSR.union(Mode::WUSR), + ) + .map(std::fs::File::from) + .unwrap(); + let mut cleanup = UnpublishedCheckpoint::new( + parent, + temporary_file, + temporary_name.clone(), + target_name.clone(), + ); + cleanup.retain_identity().unwrap(); + std::fs::rename( + temp.path().join(&temporary_name), + temp.path().join(&target_name), + ) + .unwrap(); + + drop(cleanup); + + assert!(!temp.path().join(&temporary_name).exists()); + assert!(!temp.path().join(&target_name).exists()); + } + + #[test] + fn unpublished_checkpoint_does_not_remove_a_replacement_target() { + let temp = tempfile::TempDir::new().unwrap(); + let temporary_name = OsString::from("temporary"); + let target_name = OsString::from("target"); + let retained_name = OsString::from("retained"); + let parent = open( + temp.path(), + OFlags::RDONLY | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC, + Mode::empty(), + ) + .unwrap(); + let temporary_file = openat( + &parent, + &temporary_name, + OFlags::WRONLY | OFlags::CREATE | OFlags::EXCL | OFlags::NOFOLLOW | OFlags::CLOEXEC, + Mode::RUSR.union(Mode::WUSR), + ) + .map(std::fs::File::from) + .unwrap(); + let mut cleanup = UnpublishedCheckpoint::new( + parent, + temporary_file, + temporary_name.clone(), + target_name.clone(), + ); + cleanup.retain_identity().unwrap(); + std::fs::rename( + temp.path().join(&temporary_name), + temp.path().join(&target_name), + ) + .unwrap(); + std::fs::rename( + temp.path().join(&target_name), + temp.path().join(&retained_name), + ) + .unwrap(); + std::fs::write(temp.path().join(&target_name), b"replacement").unwrap(); + + drop(cleanup); + + assert_eq!( + std::fs::read(temp.path().join(&target_name)).unwrap(), + b"replacement" + ); + assert!(temp.path().join(&retained_name).exists()); + } + + #[tokio::test] + async fn checkpoint_capture_does_not_replace_a_racing_target() { + let temp = tempfile::TempDir::new().unwrap(); + let instances = temp.path().join("instances"); + let checkpoints = temp.path().join("checkpoints"); + tokio::fs::create_dir(&instances).await.unwrap(); + tokio::fs::create_dir(&checkpoints).await.unwrap(); + let hook = Arc::new(ArtifactSyncOpenHook::new()); + let provider = Arc::new(FileStorageProvider::with_artifact_sync_open_hook( + instances.clone(), + instances, + Arc::clone(&hook), + )); + let slot = provider + .acquire(&AcquireOpts { + instance_id: "capture-racing-target".into(), + rootfs_size: 64, + mem_size: 32, + }) + .await + .unwrap(); + tokio::fs::write(&slot.rootfs_path, b"new-checkpoint") + .await + .unwrap(); + let target = checkpoints.join("rootfs.snap"); + + let capture_provider = Arc::clone(&provider); + let capture_slot = slot.clone(); + let capture_target = target.clone(); + let capture = tokio::spawn(async move { + capture_provider + .capture_checkpoint(&capture_slot, &capture_target) + .await + }); + hook.wait_until_open().await; + tokio::fs::write(&target, b"racing-checkpoint") + .await + .unwrap(); + hook.resume(); + + capture + .await + .unwrap() + .expect_err("capture must not replace a target created after validation"); + assert_eq!( + tokio::fs::read(&target).await.unwrap(), + b"racing-checkpoint" + ); + let mut entries = tokio::fs::read_dir(&checkpoints).await.unwrap(); + assert_eq!( + entries.next_entry().await.unwrap().unwrap().file_name(), + "rootfs.snap" + ); + assert!(entries.next_entry().await.unwrap().is_none()); + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn cancelled_checkpoint_capture_finishes_its_blocking_publication() { + let temp = tempfile::TempDir::new().unwrap(); + let instances = temp.path().join("instances"); + let checkpoints = temp.path().join("checkpoints"); + tokio::fs::create_dir(&instances).await.unwrap(); + tokio::fs::create_dir(&checkpoints).await.unwrap(); + let completion = Arc::new(ArtifactSyncOpenHook::new()); + let provider = Arc::new(FileStorageProvider::with_artifact_sync_open_hook( + instances.clone(), + instances, + Arc::clone(&completion), + )); + let slot = provider + .acquire(&AcquireOpts { + instance_id: "capture-cancelled-publication".into(), + rootfs_size: 64, + mem_size: 32, + }) + .await + .unwrap(); + tokio::fs::write(&slot.rootfs_path, b"complete-checkpoint") + .await + .unwrap(); + let target = checkpoints.join("rootfs.snap"); + let hook = crate::failpoint::TestFailpoint::new(&["storage-capture-before-publish"]); + + let capture = tokio::spawn({ + let hook = hook.clone(); + let provider = Arc::clone(&provider); + let slot = slot.clone(); + let target = target.clone(); + async move { hook.run(provider.capture_checkpoint(&slot, &target)).await } + }); + completion.wait_until_open().await; + completion.resume(); + hook.wait_until_paused().await; + capture.abort(); + assert!(capture.await.unwrap_err().is_cancelled()); + hook.release(); + + tokio::time::timeout( + std::time::Duration::from_secs(2), + completion.wait_until_capture_finished(), + ) + .await + .expect("blocking publication transaction must finish after caller cancellation"); + assert_eq!( + tokio::fs::read(&target).await.unwrap(), + b"complete-checkpoint" + ); + let mut entries = tokio::fs::read_dir(&checkpoints).await.unwrap(); + assert_eq!( + entries.next_entry().await.unwrap().unwrap().file_name(), + "rootfs.snap" + ); + assert!(entries.next_entry().await.unwrap().is_none()); + } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn checkpoint_capture_cleans_temporary_data_after_failure() { + let (_temp, provider, slot, checkpoints) = checkpoint_fixture("capture-cleanup").await; + tokio::fs::write(&slot.rootfs_path, b"complete-temporary-copy") + .await + .unwrap(); + let target = checkpoints.join("rootfs.snap"); + let hook = crate::failpoint::TestFailpoint::new(&["storage-capture-after-publish"]); + + hook.run(provider.capture_checkpoint(&slot, &target)) + .await + .expect_err("armed capture must roll back its unpublished target"); + + assert!(!target.exists()); + assert!( + tokio::fs::read_dir(&checkpoints) + .await + .unwrap() + .next_entry() + .await + .unwrap() + .is_none(), + "capture failure must remove its temporary file" + ); + } } diff --git a/src/blaze/crates/blazed/src/main.rs b/src/blaze/crates/blazed/src/main.rs index aa0961ccc..e2b419775 100644 --- a/src/blaze/crates/blazed/src/main.rs +++ b/src/blaze/crates/blazed/src/main.rs @@ -5,6 +5,7 @@ //! exposed via the HTTP API; this binary only handles daemon lifecycle. mod api; +mod checkpoint_store; mod cli; mod daemon; mod error; diff --git a/src/blaze/crates/blazed/src/sandbox.rs b/src/blaze/crates/blazed/src/sandbox.rs index 747e4ef1e..511984343 100644 --- a/src/blaze/crates/blazed/src/sandbox.rs +++ b/src/blaze/crates/blazed/src/sandbox.rs @@ -1,6 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 //! Managed sandbox lifecycle and runtime ownership. +mod checkpoint; mod manager; mod storage_sync; pub(crate) mod template; diff --git a/src/blaze/crates/blazed/src/sandbox/checkpoint.rs b/src/blaze/crates/blazed/src/sandbox/checkpoint.rs new file mode 100644 index 000000000..f561e5c7f --- /dev/null +++ b/src/blaze/crates/blazed/src/sandbox/checkpoint.rs @@ -0,0 +1,703 @@ +// SPDX-License-Identifier: Apache-2.0 +//! Durable checkpoint capture and listing. + +use std::sync::Arc; + +use blaze_core::backend::{SnapshotKind, SnapshotRequest}; +use blaze_core::checkpoint::{CheckpointInfo, CheckpointMetadata, CommitCheckpoint}; +use blaze_core::lifecycle::{OperationPhase, SandboxInstance, SandboxState}; +use tokio::sync::OwnedMutexGuard; +use uuid::Uuid; + +use crate::checkpoint_store::{ + CheckpointHeadOutcome, CheckpointPublishOutcome, CheckpointStage, PublishedCheckpoint, +}; +use crate::error::{BlazeDaemonError, Result}; +use crate::spawner::DynBackendInstance; + +use super::manager::SandboxManager; + +enum PublishBoundaryResult { + Published { + instance: SandboxInstance, + published: Box, + }, + KnownUnpublished { + stage: CheckpointStage, + error: BlazeDaemonError, + }, + RecoveryRequired { + error: BlazeDaemonError, + }, +} + +enum HeadBoundaryResult { + Updated { + instance: SandboxInstance, + metadata: Box, + }, + KnownUnchanged { + error: BlazeDaemonError, + }, + RecoveryRequired { + error: BlazeDaemonError, + }, +} + +impl SandboxManager { + /// Capture a self-contained checkpoint and resume the existing backend. + pub async fn checkpoint(self: &Arc, id: Uuid) -> Result { + let operation = self.operation_lock(id).lock_owned().await; + let manager = Arc::clone(self); + crate::failpoint::spawn(async move { manager.checkpoint_supervised(id, operation).await }) + .await + .map_err(|error| { + let recovery = self.mark_recovery(id).err(); + BlazeDaemonError::RecoveryRequired(format!( + "checkpoint supervisor stopped unexpectedly: {error}{}", + recovery + .map(|error| format!("; recovery state persistence failed: {error}")) + .unwrap_or_default() + )) + })? + } + + async fn checkpoint_supervised( + self: Arc, + id: Uuid, + operation: OwnedMutexGuard<()>, + ) -> Result { + let manager = Arc::clone(&self); + let result = + match crate::failpoint::spawn(async move { manager.checkpoint_worker(id).await }).await + { + Ok(result) => result, + Err(error) => { + let recovery = self.mark_recovery(id).err(); + Err(BlazeDaemonError::RecoveryRequired(format!( + "checkpoint worker stopped unexpectedly: {error}{}", + recovery + .map(|error| format!("; recovery state persistence failed: {error}")) + .unwrap_or_default() + ))) + } + }; + drop(operation); + result + } + + async fn checkpoint_worker(self: Arc, id: Uuid) -> Result { + let mut instance = self.get(id)?; + if let Some(journal) = &instance.operation { + return Err(BlazeDaemonError::RecoveryRequired(format!( + "instance {id} has unfinished {} operation", + journal.kind + ))); + } + if instance.state != SandboxState::Running { + return Err(BlazeDaemonError::Conflict(format!( + "instance {id} is {}, expected running", + instance.state + ))); + } + + let backend = self.backend_owner(id).ok_or_else(|| { + BlazeDaemonError::Conflict(format!("instance {id} has no backend owner")) + })?; + if !backend.supports_checkpoint_capture() || !self.storage.supports_checkpoint_capture() { + return Err(BlazeDaemonError::UnsupportedOperation(format!( + "instance {id} backend {} and configured storage do not support checkpoint capture", + backend.backend() + ))); + } + if backend.instance_id() != id || backend.backend() != instance.backend { + self.mark_recovery(id)?; + return Err(BlazeDaemonError::RecoveryRequired(format!( + "instance {id} backend owner identity does not match durable state" + ))); + } + let backend_version = backend.version().map(str::to_string); + if backend_version + .as_deref() + .is_some_and(|version| version.trim().is_empty()) + { + return Err(BlazeDaemonError::UnsupportedOperation(format!( + "instance {id} backend {} does not report a usable checkpoint version", + backend.backend() + ))); + } + self.require_live_backend(id, &backend).await?; + let storage = self.storage.reconstruct(&id.to_string()).await?; + + // Validate the current parent before pausing the backend. The operation + // guard keeps HEAD stable until publication, while the blocking pool + // keeps full artifact verification off Tokio's async worker threads. + let read_manager = Arc::clone(&self); + let read_head = crate::failpoint::spawn_blocking(move || { + crate::failpoint::pause_blocking("checkpoint-before-read-head"); + read_manager + .checkpoints + .read_head(id) + .map_err(checkpoint_store_error) + }) + .await + .map_err(|error| { + BlazeDaemonError::Internal(format!( + "checkpoint parent validation blocking task: {error}" + )) + })?; + let parent = read_head?; + + let begin_store = self.checkpoints.clone(); + let begin = crate::failpoint::spawn_blocking(move || { + crate::failpoint::pause_blocking("checkpoint-before-stage-begin"); + begin_store.begin(id) + }) + .await + .map_err(|error| { + let recovery = self.mark_recovery(id).err(); + BlazeDaemonError::RecoveryRequired(format!( + "checkpoint stage creation blocking task stopped unexpectedly: {error}{}", + recovery + .map(|error| format!("; recovery state persistence failed: {error}")) + .unwrap_or_default() + )) + })?; + let stage = match begin { + Ok(stage) => stage, + Err(error) => { + let Some(checkpoint_id) = error.recovery_checkpoint_id().map(str::to_owned) else { + return Err(checkpoint_store_error(error)); + }; + if let Err(journal) = instance.begin_checkpoint_operation(checkpoint_id) { + let recovery = self.mark_recovery(id).err(); + return Err(BlazeDaemonError::RecoveryRequired(format!( + "checkpoint stage creation failed: {error}; recovery journal failed: \ + {journal}{}", + recovery + .map(|error| format!("; recovery state persistence failed: {error}")) + .unwrap_or_default() + ))); + } + let recovery = self.mark_instance_recovery(instance).err(); + return Err(BlazeDaemonError::RecoveryRequired(format!( + "checkpoint stage creation failed and cleanup could not be confirmed: \ + {error}{}", + recovery + .map(|error| format!("; recovery state persistence failed: {error}")) + .unwrap_or_default() + ))); + } + }; + let checkpoint_id = stage.id().to_string(); + let snapshot_path = match stage.artifact_path("vmstate.snap") { + Ok(path) => path, + Err(error) => { + let _ = self.abort_checkpoint_stage(stage).await; + return Err(checkpoint_store_error(error)); + } + }; + let memory_path = match stage.artifact_path("memory.snap") { + Ok(path) => path, + Err(error) => { + let _ = self.abort_checkpoint_stage(stage).await; + return Err(checkpoint_store_error(error)); + } + }; + let rootfs_path = match stage.artifact_path("rootfs.snap") { + Ok(path) => path, + Err(error) => { + let _ = self.abort_checkpoint_stage(stage).await; + return Err(checkpoint_store_error(error)); + } + }; + if let Err(error) = crate::failpoint::state("checkpoint-begin-state") { + let _ = self.abort_checkpoint_stage(stage).await; + return Err(error); + } + if let Err(error) = instance.begin_checkpoint_operation(checkpoint_id.clone()) { + let _ = self.abort_checkpoint_stage(stage).await; + return Err(error.into()); + } + if let Err(error) = crate::failpoint::state("checkpoint-begin-state-commit") + .and_then(|_| self.persist_and_retain(instance.clone())) + { + if let Err(cleanup) = self.abort_checkpoint_stage(stage).await { + let recovery = self.mark_instance_recovery(instance).err(); + return Err(BlazeDaemonError::RecoveryRequired(format!( + "checkpoint intent state commit failed: {error}; checkpoint staging cleanup \ + failed: {cleanup}{}", + recovery + .map(|error| format!("; recovery state persistence failed: {error}")) + .unwrap_or_default() + ))); + } + return Err(error); + } + crate::failpoint::pause("checkpoint-after-begin").await; + + let paused = match crate::failpoint::backend("checkpoint-pause") { + Ok(()) => backend.pause().await, + Err(error) => Err(error), + }; + if let Err(error) = paused { + return self + .finish_failed_unpublished_checkpoint(id, &backend, stage, error.into()) + .await; + } + + if let Err(error) = instance + .transition(SandboxState::Paused) + .and_then(|_| instance.advance_checkpoint_phase(OperationPhase::CheckpointPaused)) + { + return self + .finish_failed_unpublished_checkpoint(id, &backend, stage, error.into()) + .await; + } + if let Err(error) = crate::failpoint::state("checkpoint-paused-state") + .and_then(|_| self.persist_and_retain(instance.clone())) + { + return self + .finish_failed_unpublished_checkpoint(id, &backend, stage, error) + .await; + } + crate::failpoint::pause("checkpoint-after-pause").await; + + let snapshot = SnapshotRequest { + snapshot_path, + mem_path: memory_path, + kind: SnapshotKind::Full, + }; + let snapshot_result = match crate::failpoint::backend("checkpoint-snapshot") { + Ok(()) => backend.snapshot(snapshot).await, + Err(error) => Err(error), + }; + if let Err(error) = snapshot_result { + return self + .finish_failed_unpublished_checkpoint(id, &backend, stage, error.into()) + .await; + } + + let flushed = match crate::failpoint::storage("checkpoint-storage-flush") { + Ok(()) => self.storage.sync_artifacts(&storage).await, + Err(error) => Err(error), + }; + if let Err(error) = flushed { + return self + .finish_failed_unpublished_checkpoint(id, &backend, stage, error.into()) + .await; + } + + let captured = match crate::failpoint::storage("checkpoint-rootfs-capture") { + Ok(()) => { + self.storage + .capture_checkpoint(&storage, &rootfs_path) + .await + } + Err(error) => Err(error), + }; + if let Err(error) = captured { + return self + .finish_failed_unpublished_checkpoint(id, &backend, stage, error.into()) + .await; + } + + let publish_manager = Arc::clone(&self); + let publish = crate::failpoint::spawn_blocking(move || { + if let Err(error) = crate::failpoint::storage("checkpoint-publish") { + return PublishBoundaryResult::KnownUnpublished { + stage, + error: error.into(), + }; + } + let published = publish_manager.checkpoints.publish_retained( + &stage, + CommitCheckpoint { + parent, + policy_name: instance.policy_name.clone(), + image_digest: instance.image_digest.clone(), + backend: instance.backend, + backend_version, + snapshot_kind: SnapshotKind::Full, + }, + ); + let published = match published { + Ok(published) => published, + Err(error) if error.outcome() == CheckpointPublishOutcome::KnownUnpublished => { + return PublishBoundaryResult::KnownUnpublished { + stage, + error: checkpoint_store_error(error.into_store_error()), + }; + } + Err(error) => { + let error = blocking_recovery_error( + &publish_manager, + &instance, + checkpoint_store_error(error.into_store_error()), + "publication with uncertain outcome", + ); + return PublishBoundaryResult::RecoveryRequired { error }; + } + }; + + crate::failpoint::pause_blocking("checkpoint-after-store-publish-before-state"); + if let Err(error) = + instance.advance_checkpoint_phase(OperationPhase::CheckpointPublished) + { + let error = blocking_recovery_error( + &publish_manager, + &instance, + error.into(), + "published journal update", + ); + return PublishBoundaryResult::RecoveryRequired { error }; + } + if let Err(error) = crate::failpoint::state("checkpoint-published-state") + .and_then(|_| publish_manager.persist_and_retain(instance.clone())) + { + let error = blocking_recovery_error( + &publish_manager, + &instance, + error, + "published state commit", + ); + return PublishBoundaryResult::RecoveryRequired { error }; + } + PublishBoundaryResult::Published { + instance, + published: Box::new(published), + } + }) + .await; + let (mut instance, published) = match publish { + Ok(PublishBoundaryResult::Published { + instance, + published, + }) => (instance, published), + Ok(PublishBoundaryResult::KnownUnpublished { stage, error }) => { + return self + .finish_failed_unpublished_checkpoint(id, &backend, stage, error) + .await; + } + Ok(PublishBoundaryResult::RecoveryRequired { error }) => { + return self + .resume_after_blocking_boundary_failure(&backend, error) + .await; + } + Err(error) => { + let error = + blocking_join_recovery_error(&self, id, "checkpoint publication", error); + return self + .resume_after_blocking_boundary_failure(&backend, error) + .await; + } + }; + crate::failpoint::pause("checkpoint-after-publish-before-head").await; + + let head_manager = Arc::clone(&self); + let head = crate::failpoint::spawn_blocking(move || { + if let Err(error) = crate::failpoint::storage("checkpoint-head-update") { + return HeadBoundaryResult::KnownUnchanged { + error: error.into(), + }; + } + let metadata = match head_manager.checkpoints.set_head_published(*published) { + Ok(metadata) => metadata, + Err(error) => { + return match error.outcome() { + CheckpointHeadOutcome::KnownUnchanged => { + HeadBoundaryResult::KnownUnchanged { + error: checkpoint_store_error(error.into_store_error()), + } + } + CheckpointHeadOutcome::Unknown => { + let error = blocking_recovery_error( + &head_manager, + &instance, + checkpoint_store_error(error.into_store_error()), + "HEAD update with uncertain outcome", + ); + HeadBoundaryResult::RecoveryRequired { error } + } + }; + } + }; + + crate::failpoint::pause_blocking("checkpoint-after-store-head-before-state"); + if let Err(error) = + instance.advance_checkpoint_phase(OperationPhase::CheckpointHeadUpdated) + { + let error = blocking_recovery_error( + &head_manager, + &instance, + error.into(), + "HEAD journal update", + ); + return HeadBoundaryResult::RecoveryRequired { error }; + } + if let Err(error) = crate::failpoint::state("checkpoint-head-state") + .and_then(|_| head_manager.persist_and_retain(instance.clone())) + { + let error = + blocking_recovery_error(&head_manager, &instance, error, "HEAD state commit"); + return HeadBoundaryResult::RecoveryRequired { error }; + } + HeadBoundaryResult::Updated { + instance, + metadata: Box::new(metadata), + } + }) + .await; + let (mut instance, metadata) = match head { + Ok(HeadBoundaryResult::Updated { instance, metadata }) => (instance, *metadata), + Ok(HeadBoundaryResult::KnownUnchanged { error }) => { + return self + .finish_failed_published_checkpoint(id, &backend, error) + .await; + } + Ok(HeadBoundaryResult::RecoveryRequired { error }) => { + return self + .resume_after_blocking_boundary_failure(&backend, error) + .await; + } + Err(error) => { + let error = + blocking_join_recovery_error(&self, id, "checkpoint HEAD update", error); + return self + .resume_after_blocking_boundary_failure(&backend, error) + .await; + } + }; + crate::failpoint::pause("checkpoint-after-head").await; + + let resumed = match crate::failpoint::backend("checkpoint-resume") { + Ok(()) => backend.resume().await, + Err(error) => Err(error), + }; + if let Err(error) = resumed { + self.mark_recovery(id)?; + return Err(BlazeDaemonError::RecoveryRequired(format!( + "checkpoint {checkpoint_id} became HEAD, but backend resume failed: {error}" + ))); + } + if let Err(error) = self.verify_backend_ready(id, &backend).await { + self.mark_recovery(id)?; + return Err(error); + } + + if let Err(error) = instance + .transition(SandboxState::Checkpointed) + .and_then(|_| instance.transition(SandboxState::Running)) + { + self.mark_recovery(id)?; + return Err(BlazeDaemonError::RecoveryRequired(format!( + "checkpoint runtime resumed, but lifecycle transition failed: {error}" + ))); + } + instance.last_checkpoint = Some(checkpoint_id); + instance.finish_operation(); + if let Err(error) = crate::failpoint::state("checkpoint-final-state") + .and_then(|_| self.persist_and_retain(instance)) + { + self.mark_recovery(id)?; + return Err(BlazeDaemonError::RecoveryRequired(format!( + "checkpoint completed, but final lifecycle state could not be committed: {error}" + ))); + } + Ok(metadata) + } + + /// List every committed checkpoint and its HEAD reachability. + pub async fn list_checkpoints(self: &Arc, id: Uuid) -> Result> { + let operation = self.operation_lock(id).lock_owned().await; + self.get(id)?; + let manager = Arc::clone(self); + crate::failpoint::spawn_blocking(move || { + let _operation = operation; + crate::failpoint::pause_blocking("checkpoint-before-store-list"); + manager.checkpoints.list(id).map_err(checkpoint_store_error) + }) + .await + .map_err(|error| { + BlazeDaemonError::Internal(format!("checkpoint list blocking task: {error}")) + })? + } + + async fn finish_failed_unpublished_checkpoint( + &self, + id: Uuid, + backend: &DynBackendInstance, + stage: CheckpointStage, + original: BlazeDaemonError, + ) -> Result { + let compensation = self + .resume_and_clear_checkpoint(id, backend, Some(stage)) + .await; + match compensation { + Ok(()) => Err(original), + Err(compensation) => Err(BlazeDaemonError::RecoveryRequired(format!( + "{original}; checkpoint compensation failed: {compensation}" + ))), + } + } + + async fn finish_failed_published_checkpoint( + &self, + id: Uuid, + backend: &DynBackendInstance, + original: BlazeDaemonError, + ) -> Result { + let compensation = self.resume_and_clear_checkpoint(id, backend, None).await; + match compensation { + Ok(()) => Err(original), + Err(compensation) => Err(BlazeDaemonError::RecoveryRequired(format!( + "{original}; checkpoint compensation failed: {compensation}" + ))), + } + } + + async fn resume_after_blocking_boundary_failure( + &self, + backend: &DynBackendInstance, + original: BlazeDaemonError, + ) -> Result { + let resume = self.resume_backend(backend).await; + Err(BlazeDaemonError::RecoveryRequired(format!( + "{original}{}", + resume + .err() + .map(|error| format!("; backend resume failed: {error}")) + .unwrap_or_default() + ))) + } + + async fn resume_and_clear_checkpoint( + &self, + id: Uuid, + backend: &DynBackendInstance, + unpublished_stage: Option, + ) -> Result<()> { + if let Err(error) = self.resume_backend(backend).await { + let recovery = self.mark_recovery(id).err(); + return Err(BlazeDaemonError::RecoveryRequired(format!( + "backend resume failed: {error}{}", + recovery + .map(|error| format!("; recovery state persistence failed: {error}")) + .unwrap_or_default() + ))); + } + if let Some(stage) = unpublished_stage + && let Err(error) = self.abort_checkpoint_stage(stage).await + { + let recovery = self.mark_recovery(id).err(); + return Err(BlazeDaemonError::RecoveryRequired(format!( + "checkpoint staging cleanup failed: {error}{}", + recovery + .map(|error| format!("; recovery state persistence failed: {error}")) + .unwrap_or_default() + ))); + } + + let mut instance = self.get(id)?; + if instance.state == SandboxState::Paused { + instance.transition(SandboxState::Running)?; + } + instance.finish_operation(); + if let Err(error) = self.persist_and_retain(instance) { + let recovery = self.mark_recovery(id).err(); + return Err(BlazeDaemonError::RecoveryRequired(format!( + "checkpoint compensation state commit failed: {error}{}", + recovery + .map(|error| format!("; recovery state persistence failed: {error}")) + .unwrap_or_default() + ))); + } + Ok(()) + } + + async fn abort_checkpoint_stage(&self, stage: CheckpointStage) -> Result<()> { + let checkpoints = self.checkpoints.clone(); + crate::failpoint::spawn_blocking(move || { + crate::failpoint::pause_blocking("checkpoint-before-stage-abort"); + checkpoints.abort(stage).map_err(checkpoint_store_error) + }) + .await + .map_err(|error| { + BlazeDaemonError::Internal(format!( + "checkpoint staging cleanup blocking task stopped unexpectedly: {error}" + )) + })? + } + + async fn resume_backend(&self, backend: &DynBackendInstance) -> Result<()> { + match crate::failpoint::backend("checkpoint-compensation-resume") { + Ok(()) => backend.resume().await?, + Err(error) => return Err(error.into()), + } + self.verify_backend_ready(backend.instance_id(), backend) + .await + } + + async fn require_live_backend(&self, id: Uuid, backend: &DynBackendInstance) -> Result<()> { + match backend.try_wait().await { + Ok(None) => Ok(()), + Ok(Some(result)) => { + self.mark_recovery(id)?; + Err(BlazeDaemonError::RecoveryRequired(format!( + "instance {id} backend exited before checkpoint capture \ + (exit={:?}, signal={:?})", + result.exit_code, result.signal + ))) + } + Err(error) => { + self.mark_recovery(id)?; + Err(BlazeDaemonError::RecoveryRequired(format!( + "instance {id} backend liveness is unknown: {error}" + ))) + } + } + } + + async fn verify_backend_ready(&self, id: Uuid, backend: &DynBackendInstance) -> Result<()> { + self.require_live_backend(id, backend).await?; + self.wait_for_guest_ready(backend, "checkpoint-guest-ready") + .await?; + self.require_live_backend(id, backend).await + } +} + +fn blocking_recovery_error( + manager: &SandboxManager, + instance: &SandboxInstance, + original: BlazeDaemonError, + boundary: &str, +) -> BlazeDaemonError { + let recovery = manager.mark_instance_recovery(instance.clone()); + BlazeDaemonError::RecoveryRequired(format!( + "checkpoint {boundary} failed: {original}{}", + recovery + .err() + .map(|error| format!("; recovery state persistence failed: {error}")) + .unwrap_or_default() + )) +} + +fn blocking_join_recovery_error( + manager: &SandboxManager, + id: Uuid, + boundary: &str, + join: tokio::task::JoinError, +) -> BlazeDaemonError { + let recovery = manager.mark_recovery(id); + BlazeDaemonError::RecoveryRequired(format!( + "{boundary} blocking task failed: {join}{}", + recovery + .err() + .map(|error| format!("; recovery state persistence failed: {error}")) + .unwrap_or_default() + )) +} + +fn checkpoint_store_error(error: impl std::fmt::Display) -> BlazeDaemonError { + BlazeDaemonError::Internal(format!("checkpoint store: {error}")) +} diff --git a/src/blaze/crates/blazed/src/sandbox/manager.rs b/src/blaze/crates/blazed/src/sandbox/manager.rs index 1f9e32e95..c37ab7612 100644 --- a/src/blaze/crates/blazed/src/sandbox/manager.rs +++ b/src/blaze/crates/blazed/src/sandbox/manager.rs @@ -15,6 +15,7 @@ use tokio::sync::{Mutex as AsyncMutex, OwnedMutexGuard, Semaphore}; use tokio_util::sync::CancellationToken; use uuid::Uuid; +use crate::checkpoint_store::CheckpointStore; use crate::error::{BlazeDaemonError, Result}; use crate::guest::{GuestClient, GuestExecResult, MAX_GUEST_FILE_BYTES}; use crate::metrics::Metrics; @@ -81,8 +82,9 @@ pub struct SandboxManager { pub(super) storage_sync_permits: Arc, spawners: Arc, active_backend: BackendKind, - storage: Arc, + pub(super) storage: Arc, state_store: StateStore, + pub(super) checkpoints: CheckpointStore, rootfs_size: u64, mem_size: u64, metrics: Arc, @@ -129,6 +131,7 @@ impl SandboxManager { let instances = Arc::new(Mutex::new(instances)); let backend_instances = Arc::new(Mutex::new(HashMap::new())); let metrics = Arc::new(Metrics::new()); + let checkpoints = CheckpointStore::new(state_store.clone()); let resources = SandboxManagerResources { #[cfg(test)] instances: instances.clone(), @@ -147,6 +150,7 @@ impl SandboxManager { active_backend, storage, state_store, + checkpoints, rootfs_size, mem_size, metrics, @@ -477,10 +481,21 @@ impl SandboxManager { } /// Idempotently destroy one sandbox and its owned runtime resources. - pub async fn destroy(&self, id: Uuid) -> Result { - let operation_lock = self.operation_lock(id); - let _operation = operation_lock.lock().await; - self.destroy_locked(id).await + /// + /// The supervised task retains per-sandbox serialization after a caller + /// disconnects, so blocking filesystem cleanup cannot race a retry. + pub async fn destroy(self: &Arc, id: Uuid) -> Result { + let manager = Arc::clone(self); + crate::failpoint::spawn(async move { + let operation = manager.operation_lock(id).lock_owned().await; + let result = manager.destroy_locked(id).await; + drop(operation); + result + }) + .await + .map_err(|error| { + BlazeDaemonError::Internal(format!("destroy supervisor failed: {error}")) + })? } async fn destroy_locked(&self, id: Uuid) -> Result { @@ -576,6 +591,28 @@ impl SandboxManager { ))); } + let checkpoints = self.checkpoints.clone(); + let checkpoint_cleanup = crate::failpoint::spawn_blocking(move || { + crate::failpoint::pause_blocking("checkpoint-before-store-remove"); + checkpoints.remove_sandbox(id) + }) + .await; + let checkpoint_cleanup_error = match checkpoint_cleanup { + Ok(Ok(())) => None, + Ok(Err(error)) => Some(error.to_string()), + Err(error) => Some(format!("blocking task failed: {error}")), + }; + if let Some(error) = checkpoint_cleanup_error { + let recovery = self.mark_recovery(id).err(); + return Err(BlazeDaemonError::RecoveryRequired(format!( + "destroy {id}: backend stopped but checkpoint cleanup failed: {error}; \ + storage retained{}", + recovery + .map(|error| format!("; recovery state persistence failed: {error}")) + .unwrap_or_default() + ))); + } + if let Err(error) = self.storage.release_by_id(&id.to_string()).await { let recovery = self.mark_recovery(id).err(); return Err(BlazeDaemonError::RecoveryRequired(format!( @@ -696,7 +733,7 @@ impl SandboxManager { )) } - async fn wait_for_guest_ready( + pub(super) async fn wait_for_guest_ready( &self, backend: &DynBackendInstance, failpoint: &str, @@ -885,11 +922,19 @@ impl SandboxManager { } } - fn mark_recovery(&self, id: Uuid) -> Result<()> { + pub(super) fn mark_recovery(&self, id: Uuid) -> Result<()> { self.mark_instance_recovery(self.get(id)?) } - fn mark_instance_recovery(&self, mut instance: SandboxInstance) -> Result<()> { + pub(super) fn persist_and_retain(&self, instance: SandboxInstance) -> Result<()> { + self.state_store.persist(&instance)?; + if let Some(error) = self.retain_instance(instance) { + return Err(BlazeDaemonError::RecoveryRequired(error)); + } + Ok(()) + } + + pub(super) fn mark_instance_recovery(&self, mut instance: SandboxInstance) -> Result<()> { if instance.state != SandboxState::RecoveryRequired { instance.transition(SandboxState::RecoveryRequired)?; } @@ -918,7 +963,7 @@ impl SandboxManager { } } - fn retain_instance(&self, instance: SandboxInstance) -> Option { + pub(super) fn retain_instance(&self, instance: SandboxInstance) -> Option { match self.instances.lock() { Ok(mut instances) => { instances.insert(instance.id, instance); diff --git a/src/blaze/crates/blazed/src/spawner.rs b/src/blaze/crates/blazed/src/spawner.rs index 0d3bd15d6..b18ecd95e 100644 --- a/src/blaze/crates/blazed/src/spawner.rs +++ b/src/blaze/crates/blazed/src/spawner.rs @@ -16,7 +16,7 @@ use std::time::Duration; use std::time::Instant; use async_trait::async_trait; -use blaze_core::backend::{BackendKind, SpawnRequest}; +use blaze_core::backend::{BackendKind, SnapshotRequest, SpawnRequest}; #[cfg(test)] use blaze_core::guest_protocol::DEFAULT_MAX_RESPONSE_BYTES; use blaze_core::{BlazeError, Result}; @@ -60,8 +60,23 @@ pub struct SpawnResult { /// Owned runtime instance returned by a backend spawner. #[async_trait] pub trait BackendInstance: Send + Sync { + /// Stable sandbox identifier. + /// + /// The nil default prevents legacy or test-only owners from claiming a + /// real sandbox identity until they explicitly implement this contract. + fn instance_id(&self) -> Uuid { + Uuid::nil() + } /// Concrete backend implementation. fn backend(&self) -> BackendKind; + /// Backend version frozen into checkpoint metadata when available. + fn version(&self) -> Option<&str> { + None + } + /// Whether pause, resume, and full snapshot capture are implemented. + fn supports_checkpoint_capture(&self) -> bool { + false + } /// Guest transport endpoint, or an empty path for guestless backends. fn guest_socket_path(&self) -> &Path { Path::new("") @@ -72,6 +87,24 @@ pub trait BackendInstance: Send + Sync { /// Once an exit is observed, later calls continue to report a completed /// result even though the underlying handle has already been consumed. async fn try_wait(&self) -> Result>; + /// Pause guest execution for a consistent snapshot. + async fn pause(&self) -> Result<()> { + Err(BlazeError::BackendError { + msg: format!("{} does not support checkpoint pause", self.backend()), + }) + } + /// Resume guest execution after snapshot capture. + async fn resume(&self) -> Result<()> { + Err(BlazeError::BackendError { + msg: format!("{} does not support checkpoint resume", self.backend()), + }) + } + /// Write a self-contained snapshot. + async fn snapshot(&self, _request: SnapshotRequest) -> Result<()> { + Err(BlazeError::BackendError { + msg: format!("{} does not support checkpoint capture", self.backend()), + }) + } /// Terminate the process and release all backend-owned resources. async fn kill(&self) -> Result<()>; } @@ -123,10 +156,22 @@ struct RuntimeOwnedBackend { #[async_trait] impl BackendInstance for RuntimeOwnedBackend { + fn instance_id(&self) -> Uuid { + self.inner.instance_id() + } + fn backend(&self) -> BackendKind { self.inner.backend() } + fn version(&self) -> Option<&str> { + self.inner.version() + } + + fn supports_checkpoint_capture(&self) -> bool { + self.inner.supports_checkpoint_capture() + } + fn guest_socket_path(&self) -> &Path { self.inner.guest_socket_path() } @@ -135,6 +180,18 @@ impl BackendInstance for RuntimeOwnedBackend { self.inner.try_wait().await } + async fn pause(&self) -> Result<()> { + self.inner.pause().await + } + + async fn resume(&self) -> Result<()> { + self.inner.resume().await + } + + async fn snapshot(&self, request: SnapshotRequest) -> Result<()> { + self.inner.snapshot(request).await + } + async fn kill(&self) -> Result<()> { self.inner.kill().await } @@ -473,10 +530,22 @@ async fn spawn_mock_instance(instance_id: Uuid) -> Result { #[async_trait] impl BackendInstance for MockInstance { + fn instance_id(&self) -> Uuid { + self.instance_id + } + fn backend(&self) -> BackendKind { BackendKind::Mock } + fn version(&self) -> Option<&str> { + Some("mock-v1") + } + + fn supports_checkpoint_capture(&self) -> bool { + true + } + async fn try_wait(&self) -> Result> { let task = { let mut task = self.task.lock().await; @@ -502,6 +571,36 @@ impl BackendInstance for MockInstance { })) } + async fn pause(&self) -> Result<()> { + Ok(()) + } + + async fn resume(&self) -> Result<()> { + Ok(()) + } + + async fn snapshot(&self, request: SnapshotRequest) -> Result<()> { + for path in [&request.snapshot_path, &request.mem_path] { + if let Some(parent) = path + .parent() + .filter(|parent| !parent.as_os_str().is_empty()) + { + tokio::fs::create_dir_all(parent).await?; + } + } + let vmstate = serde_json::to_vec(&serde_json::json!({ + "format": "blaze-mock-v1", + "instance_id": self.instance_id, + "kind": request.kind, + })) + .map_err(|error| BlazeError::BackendError { + msg: format!("serialize mock VM state: {error}"), + })?; + tokio::fs::write(&request.snapshot_path, vmstate).await?; + tokio::fs::write(&request.mem_path, b"blaze-mock-memory-v1").await?; + Ok(()) + } + async fn kill(&self) -> Result<()> { if self.killed.load(Ordering::Acquire) { return Ok(()); @@ -548,6 +647,7 @@ impl BackendSpawner for GuestMockSpawner { struct GuestMockInstance { instance_id: Uuid, guest_socket_path: PathBuf, + files: Arc>>>, cancellation: CancellationToken, task: Mutex>>, killed: AtomicBool, @@ -588,6 +688,7 @@ async fn spawn_guest_mock_instance( Ok(Arc::new(GuestMockInstance { instance_id, guest_socket_path: socket, + files, cancellation, task: Mutex::new(Some(task)), killed: AtomicBool::new(false), @@ -597,10 +698,22 @@ async fn spawn_guest_mock_instance( #[cfg(test)] #[async_trait] impl BackendInstance for GuestMockInstance { + fn instance_id(&self) -> Uuid { + self.instance_id + } + fn backend(&self) -> BackendKind { BackendKind::Mock } + fn version(&self) -> Option<&str> { + Some("guest-mock-v1") + } + + fn supports_checkpoint_capture(&self) -> bool { + true + } + fn guest_socket_path(&self) -> &Path { &self.guest_socket_path } @@ -627,6 +740,33 @@ impl BackendInstance for GuestMockInstance { } } + async fn pause(&self) -> Result<()> { + Ok(()) + } + + async fn resume(&self) -> Result<()> { + Ok(()) + } + + async fn snapshot(&self, request: SnapshotRequest) -> Result<()> { + let vmstate = serde_json::to_vec(&serde_json::json!({ + "format": "blaze-guest-mock-v1", + "instance_id": self.instance_id, + "kind": request.kind, + })) + .map_err(|error| BlazeError::BackendError { + msg: format!("serialize guest mock VM state: {error}"), + })?; + let memory = serde_json::to_vec(&*self.files.lock().await).map_err(|error| { + BlazeError::BackendError { + msg: format!("serialize guest mock memory: {error}"), + } + })?; + tokio::fs::write(&request.snapshot_path, vmstate).await?; + tokio::fs::write(&request.mem_path, memory).await?; + Ok(()) + } + async fn kill(&self) -> Result<()> { if self.killed.load(Ordering::Acquire) { return Ok(()); @@ -1141,7 +1281,7 @@ mod tests { #[cfg(target_os = "linux")] use std::time::Duration; - use blaze_core::backend::SpawnRequest; + use blaze_core::backend::{SnapshotKind, SnapshotRequest, SpawnRequest}; use blaze_core::policy::BackendConfigs; use blaze_core::storage::StorageSlot; @@ -1149,6 +1289,23 @@ mod tests { use super::*; + struct UnsupportedInstance; + + #[async_trait] + impl BackendInstance for UnsupportedInstance { + fn backend(&self) -> BackendKind { + BackendKind::Bubblewrap + } + + async fn try_wait(&self) -> Result> { + Ok(None) + } + + async fn kill(&self) -> Result<()> { + Ok(()) + } + } + fn request(root: &Path) -> BackendSpawnRequest { let id = Uuid::new_v4(); let slot_dir = root.join("slot"); @@ -1331,6 +1488,59 @@ mod tests { instance.kill().await.expect("idempotent kill"); } + #[tokio::test] + async fn checkpoint_capture_defaults_fail_closed() { + let temp = tempfile::tempdir().expect("temp"); + let instance = UnsupportedInstance; + let request = SnapshotRequest { + snapshot_path: temp.path().join("vmstate.snap"), + mem_path: temp.path().join("memory.snap"), + kind: SnapshotKind::Full, + }; + + assert_eq!(instance.instance_id(), Uuid::nil()); + assert_eq!(instance.version(), None); + assert!(!instance.supports_checkpoint_capture()); + assert!(instance.pause().await.is_err()); + assert!(instance.resume().await.is_err()); + assert!(instance.snapshot(request).await.is_err()); + } + + #[tokio::test] + async fn mock_instance_captures_self_contained_state() { + let temp = tempfile::tempdir().expect("temp"); + let spawn = request(temp.path()); + let instance_id = spawn.instance_id; + let instance = MockSpawner.spawn(spawn).await.expect("spawn"); + let snapshot_path = temp.path().join("checkpoint/vmstate.snap"); + let mem_path = temp.path().join("checkpoint/memory.snap"); + + assert_eq!(instance.instance_id(), instance_id); + assert_eq!(instance.version(), Some("mock-v1")); + assert!(instance.supports_checkpoint_capture()); + instance.pause().await.expect("pause"); + instance + .snapshot(SnapshotRequest { + snapshot_path: snapshot_path.clone(), + mem_path: mem_path.clone(), + kind: SnapshotKind::Full, + }) + .await + .expect("snapshot"); + instance.resume().await.expect("resume"); + + let vmstate: serde_json::Value = + serde_json::from_slice(&std::fs::read(&snapshot_path).expect("VM state")) + .expect("VM state JSON"); + assert_eq!(vmstate["instance_id"], instance_id.to_string()); + assert_eq!(vmstate["kind"], "full"); + assert_eq!( + std::fs::read(&mem_path).expect("memory"), + b"blaze-mock-memory-v1" + ); + instance.kill().await.expect("kill"); + } + #[cfg(target_os = "linux")] #[tokio::test] async fn child_termination_requests_graceful_exit_first() { diff --git a/src/blaze/crates/blazed/src/state_store.rs b/src/blaze/crates/blazed/src/state_store.rs index 042d5f2da..902d663be 100644 --- a/src/blaze/crates/blazed/src/state_store.rs +++ b/src/blaze/crates/blazed/src/state_store.rs @@ -23,6 +23,8 @@ use crate::error::{BlazeDaemonError, Result}; const STATE_FILE: &str = "state.json"; const TEMP_STATE_FILE: &str = "state.json.tmp"; +const CHECKPOINT_DIRECTORY: &str = "checkpoints"; +const CHECKPOINT_DIRECTORY_MODE: Mode = Mode::RWXU; /// Central access point for the daemon state directory. /// @@ -56,6 +58,21 @@ pub(crate) struct OwnedRunDir { inner: Arc, } +/// Cloneable owner of a directory derived from the retained state root. +/// +/// Checkpoint catalog code uses this handle instead of reopening configured +/// pathnames after startup validation. +#[derive(Clone)] +pub(crate) struct OwnedStateDirectory { + inner: Arc, +} + +struct OwnedStateDirectoryInner { + configured_path: PathBuf, + stable_path: PathBuf, + directory: OwnedFd, +} + struct OwnedRunDirInner { instance_id: Uuid, configured_path: PathBuf, @@ -83,6 +100,15 @@ impl fmt::Debug for OwnedRunDir { } } +impl fmt::Debug for OwnedStateDirectory { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("OwnedStateDirectory") + .field("configured_path", &self.inner.configured_path) + .finish_non_exhaustive() + } +} + impl StateStore { /// Open and exclusively own the configured state directory. pub fn open(root: PathBuf) -> Result { @@ -120,6 +146,32 @@ impl StateStore { }) } + /// Create or open the daemon-owned checkpoint namespace relative to the + /// retained state-root object. + pub(crate) fn checkpoint_directory(&self) -> Result { + match mkdirat( + &self.inner.root, + CHECKPOINT_DIRECTORY, + CHECKPOINT_DIRECTORY_MODE, + ) { + Ok(()) | Err(Errno::EXIST) => {} + Err(error) => return Err(std::io::Error::from(error).into()), + } + let directory = openat( + &self.inner.root, + CHECKPOINT_DIRECTORY, + OFlags::RDONLY | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC, + Mode::empty(), + ) + .map_err(std::io::Error::from)?; + crate::failpoint::state("checkpoint-state-root-sync")?; + fsync(&self.inner.root).map_err(std::io::Error::from)?; + Ok(OwnedStateDirectory::new( + self.inner.configured_root.join(CHECKPOINT_DIRECTORY), + directory, + )) + } + /// Return the retained owner for one known sandbox directory. pub(crate) fn run_dir(&self, id: Uuid) -> Result { self.cached_run_dir(id)?.ok_or_else(|| { @@ -996,6 +1048,34 @@ impl OwnedRunDir { } } +impl OwnedStateDirectory { + pub(crate) fn new(configured_path: PathBuf, directory: OwnedFd) -> Self { + #[cfg(target_os = "linux")] + let stable_path = PathBuf::from(format!("/proc/self/fd/{}", directory.as_raw_fd())); + #[cfg(not(target_os = "linux"))] + let stable_path = configured_path.clone(); + Self { + inner: Arc::new(OwnedStateDirectoryInner { + configured_path, + stable_path, + directory, + }), + } + } + + pub(crate) fn path(&self) -> &Path { + &self.inner.stable_path + } + + pub(crate) fn configured_path(&self) -> &Path { + &self.inner.configured_path + } + + pub(crate) fn descriptor(&self) -> &OwnedFd { + &self.inner.directory + } +} + #[cfg(test)] mod tests { use blaze_core::backend::BackendKind; @@ -2151,4 +2231,42 @@ mod tests { assert!(matches!(error, BlazeDaemonError::Conflict(_))); drop(owner); } + + #[cfg(feature = "test-failpoints")] + #[tokio::test] + async fn checkpoint_catalog_retries_state_root_sync_when_existing() { + let temporary = tempfile::tempdir().expect("temporary directory"); + let root = temporary.path().join("state"); + std::fs::create_dir(&root).expect("state directory"); + let store = StateStore::new(root.clone()); + let hook = crate::failpoint::TestFailpoint::new(&["checkpoint-state-root-sync"]); + + let first_error = hook + .run(async { store.checkpoint_directory() }) + .await + .expect_err("initial state-root sync must fail"); + assert!( + first_error + .to_string() + .contains("checkpoint-state-root-sync") + ); + assert!( + root.join(CHECKPOINT_DIRECTORY).is_dir(), + "the failed parent sync leaves the newly created catalog" + ); + + let retry_error = hook + .run(async { store.checkpoint_directory() }) + .await + .expect_err("retry must synchronize the state root again"); + assert!( + retry_error + .to_string() + .contains("checkpoint-state-root-sync") + ); + + store + .checkpoint_directory() + .expect("unarmed retry synchronizes the state root"); + } } diff --git a/src/blaze/docs/design/lifecycle-state-consistency.md b/src/blaze/docs/design/lifecycle-state-consistency.md index 33381feaf..b135e4f18 100644 --- a/src/blaze/docs/design/lifecycle-state-consistency.md +++ b/src/blaze/docs/design/lifecycle-state-consistency.md @@ -2,18 +2,20 @@ [中文版](lifecycle-state-consistency_zh.md) -Blaze has two related lifecycle boundaries. Before serving requests, it must +Blaze has three related lifecycle boundaries. Before serving requests, it must reconstruct a complete persisted sandbox inventory without exposing a partial result. While serving requests, it exposes lifecycle and guest operations only through the sandbox namespace and rejects reserved reusable-capacity operations -before they can change ownership. Retired `Reset`, `Warm`, and -`start_path = "warm"` values remain decodable so startup can clean non-terminal -records that contain them. +before they can change ownership. Checkpoint capture must publish artifacts, +checkpoint history, and lifecycle state in a recoverable order. Retired +`Reset`, `Warm`, and `start_path = "warm"` values remain decodable so startup +can clean non-terminal records that contain them. -This document defines both boundaries. The inventory-publication protocol does -not change the HTTP API, configuration keys, or persisted JSON format. The +This document defines all three boundaries. The inventory-publication protocol +does not change the HTTP API, configuration keys, or persisted JSON format. The management API section defines the sandbox namespace and the reserved -reusable-capacity boundary. +reusable-capacity boundary. The checkpoint section defines two sandbox routes +and the durable operation fields used to recover interrupted capture. ## Terms and owned objects @@ -98,12 +100,47 @@ validated inventory into a partial one. Blaze attempts to persist the recovery state; if that write also fails, reconciliation reports the additional error and the durable record may still contain its previous state. +## Checkpoint lifecycle and recovery + +`POST /v1/sandboxes/{id}/checkpoint` captures a running sandbox, and +`GET /v1/sandboxes/{id}/checkpoints` lists its committed history. Both +operations hold the same per-sandbox operation lock used by lifecycle and +guest requests. Capture requires a `Running` record with no unfinished +operation, a live matching backend owner, and explicit capture support from +both the backend and storage provider. An unsupported combination returns +`501 Not Implemented` before the backend is paused or lifecycle state changes. + +Capture uses this durable order: + +1. Validate the current checkpoint parent and create a private staging + directory. +2. Persist checkpoint intent, including the generated checkpoint ID, before + pausing the backend. +3. Pause the backend, record that durable phase, capture backend state and the + provider-owned writable root, then publish an integrity-checked manifest by + a no-replace rename. +4. Persist publication, atomically move the sandbox checkpoint HEAD, and + persist the HEAD-update phase. +5. Resume and revalidate the backend, pass through `Checkpointed` back to + `Running`, record `last_checkpoint`, clear the operation, and persist the + final lifecycle record. + +A failure known to precede publication removes the private stage, resumes the +backend, and clears the operation. If publication, HEAD movement, lifecycle +persistence, or backend resume has an unknown or unsafe outcome, Blaze retains +the durable operation and marks the sandbox `RecoveryRequired`. Startup does +not restore a checkpoint or adopt an interrupted backend; normal reconciliation +cleans the owned runtime and checkpoint transaction artifacts. Committed +checkpoint history is retained until sandbox destruction. Restore, deletion, +and pruning are outside this interface. + ## Management API and reusable-state boundary Lifecycle and guest operations are registered under `/v1/sandboxes`. -Action-style reset, checkpoint, and destroy paths are unregistered and return +Action-style reset and destroy paths are unregistered and return `404 Not Found`. Canonical destruction remains -`DELETE /v1/sandboxes/{id}`. Checkpoint capture is defined separately. +`DELETE /v1/sandboxes/{id}`. Checkpoint capture uses the two routes defined in +the preceding section. The following reserved management routes also return `501 Not Implemented` and do not manage reusable capacity: @@ -158,6 +195,10 @@ Future lifecycle-state changes must preserve these rules: checks have passed; - unregistered sandbox action routes return `404` before reading or changing sandbox state; +- checkpoint capture keeps the per-sandbox operation lock until every + supervised backend, storage, publication, and state task has converged; +- a checkpoint is never exposed as committed history before its artifacts and + manifest are durably published, and HEAD never names an unpublished entry; - pool-management rejections occur before lifecycle, runtime, or storage ownership changes; and - lifecycle operations cannot enter or reactivate `Reset` or `Warm`; legacy diff --git a/src/blaze/docs/design/lifecycle-state-consistency_zh.md b/src/blaze/docs/design/lifecycle-state-consistency_zh.md index 3db5a8dbc..0647ef372 100644 --- a/src/blaze/docs/design/lifecycle-state-consistency_zh.md +++ b/src/blaze/docs/design/lifecycle-state-consistency_zh.md @@ -2,14 +2,16 @@ [English](lifecycle-state-consistency.md) -Blaze 有两个相互关联的生命周期边界。提供请求服务前，它必须完整重建已经持久化 +Blaze 有三个相互关联的生命周期边界。提供请求服务前，它必须完整重建已经持久化 的 sandbox 清单，且不能暴露部分结果。提供请求服务期间，它只通过沙箱命名空间 提供生命周期和客户机操作，并在预留的复用容量操作改变所有权之前将其拒绝。已停用 的 `Reset`、`Warm` 和 `start_path = "warm"` 值继续可解析，以便启动恢复清理 -包含这些值的非终态记录。 +包含这些值的非终态记录。检查点捕获还必须按照可恢复的顺序发布制品、检查点历史和 +生命周期状态。 -本设计定义这两个边界。清单发布流程不改变 HTTP API、配置项或持久化 JSON 格式。 -管理 API 章节定义沙箱命名空间以及预留的复用容量边界。 +本设计定义这三个边界。清单发布流程不改变 HTTP API 或配置项。管理 API 章节定义 +沙箱命名空间以及预留的复用容量边界。检查点章节定义两个沙箱路由，以及恢复中断 +捕获所需的持久化操作字段。清单发布流程不改变持久化 JSON 格式。 ## 概念与持有对象 @@ -80,12 +82,36 @@ Blaze 会保留被拒绝的 UUID 目录及其 `state.json`，供运维人员检 变成部分清单。Blaze 会尝试持久化恢复状态；如果这次写入也失败，启动恢复会报告 附加错误，持久化记录仍可能保留先前的状态。 +## 检查点生命周期与恢复 + +`POST /v1/sandboxes/{id}/checkpoint` 捕获运行中的 sandbox， +`GET /v1/sandboxes/{id}/checkpoints` 查询已经提交的历史。两项操作都持有生命周期 +和客户机请求共用的单 sandbox 操作锁。捕获要求记录处于 `Running`、没有未结束 +操作、后端持有者仍存活且身份一致，并且后端与存储提供者都明确声明支持捕获。 +组合不受支持时，在暂停后端或改变生命周期状态前返回 `501 Not Implemented`。 + +捕获按照以下持久化顺序执行： + +1. 校验当前检查点父项，并创建私有暂存目录。 +2. 暂停后端前，先持久化检查点意图和生成的检查点标识符。 +3. 暂停后端并记录该持久化阶段，捕获后端状态和存储提供者持有的可写根目录，再通过 + 不替换已有目标的重命名发布经过完整性校验的清单。 +4. 持久化发布状态，原子移动 sandbox 的检查点 HEAD，并持久化 HEAD 更新阶段。 +5. 恢复并重新检查后端，使生命周期经过 `Checkpointed` 返回 `Running`，记录 + `last_checkpoint`，清除操作并持久化最终生命周期记录。 + +如果能够确认失败发生在发布前，Blaze 会删除私有暂存内容、恢复后端并清除操作。 +如果发布、HEAD 移动、生命周期持久化或后端恢复的结果未知或不安全，Blaze 会保留 +持久化操作，并把 sandbox 标记为 `RecoveryRequired`。启动过程不会从检查点恢复， +也不会接管中断的后端；常规恢复会清理由记录持有的运行环境和检查点事务制品。 +已经提交的检查点历史会保留到 sandbox 销毁。该接口不提供恢复、删除或清理能力。 + ## 管理 API 与可复用状态边界 -生命周期和客户机操作注册在 `/v1/sandboxes` 下。操作式重置、检查点和销毁 -路径不注册，并返回 `404 Not Found`。规范的销毁入口仍是 -`DELETE /v1/sandboxes/{id}`。 -检查点捕获由独立设计定义。 +生命周期和客户机操作注册在 `/v1/sandboxes` 下。操作式重置和销毁路径不注册， +并返回 `404 Not Found`。 +规范的销毁入口仍是 `DELETE /v1/sandboxes/{id}`。检查点捕获使用上一节定义的 +两个路由。 以下保留的管理路由同样返回 `501 Not Implemented`，并且不会管理复用容量： @@ -128,5 +154,8 @@ lock 的 daemon 进程。advisory lock 不会阻止无关进程直接修改该 - 必须先完成最终 UUID 枚举，再复验保留对象； - 所有清单检查完成前，request handler 不能观察到任何一个启动 map； - 未注册的沙箱操作式路由必须在读取或改变 sandbox 状态前返回 `404`； +- 检查点捕获必须持续持有单 sandbox 操作锁，直到所有受监督的后端、存储、发布和 + 状态任务都已经收敛； +- 检查点制品和清单持久发布前不能出现在已提交历史中，HEAD 也不能指向未发布条目； - 资源池管理请求必须在生命周期、运行环境或存储所有权发生变化前被拒绝； - 生命周期操作不能进入或重新启用 `Reset` 或 `Warm`；旧值只能用于清理。 diff --git a/src/blaze/docs/design/storage-artifact-synchronization.md b/src/blaze/docs/design/storage-artifact-synchronization.md index a6e6275ce..00e356ea4 100644 --- a/src/blaze/docs/design/storage-artifact-synchronization.md +++ b/src/blaze/docs/design/storage-artifact-synchronization.md @@ -59,6 +59,33 @@ The file provider calls `sync_all` for the canonical `rootfs.ext4`, `mem.bin`, Other providers can use a different mechanism while preserving the same ownership-until-completion contract. +## Checkpoint artifact capture and publication + +Checkpoint capture is a separate, explicit provider capability. The default +`StorageProvider` implementation reports no support and returns an error, so a +provider cannot silently opt into partial capture. The file provider +reconstructs the canonical slot from the sandbox ID, retains the opened source +file, and copies the writable root into a private target owned by the +checkpoint transaction. It preserves sparse extents when possible and never +replaces an existing target. + +The checkpoint catalog is derived from the retained state-root directory. Each +sandbox has private staging entries, committed checkpoint directories, and one +HEAD reference. Publication verifies the required backend-state, memory, and +writable-root artifacts, records their sizes and SHA-256 digests, synchronizes +the files and directories, and publishes the checkpoint with a no-replace +rename. HEAD is updated atomically only after that publication is durable. +Listing reopens and validates committed manifests and artifacts before +reporting history and HEAD reachability. + +Blocking file copies, manifest publication, and HEAD updates remain supervised +after request cancellation and retain the sandbox operation lock until their +outcome is known. A known pre-publication failure removes only the private +stage. An uncertain publication never removes a path whose identity cannot be +proven. Sandbox destruction removes transaction artifacts and committed +checkpoint history under the same state-root ownership boundary. Restore, +checkpoint deletion, and pruning are not part of this protocol. + ## Capability boundary Each provider synchronization call persists the already-written artifact bytes diff --git a/src/blaze/docs/design/storage-artifact-synchronization_zh.md b/src/blaze/docs/design/storage-artifact-synchronization_zh.md index 423fcb27e..dd5d84447 100644 --- a/src/blaze/docs/design/storage-artifact-synchronization_zh.md +++ b/src/blaze/docs/design/storage-artifact-synchronization_zh.md @@ -49,6 +49,25 @@ sweep，避免一个正在进行的 sandbox 操作阻止 worker 继续处理其 `sync_all`，然后同步 slot 目录；其他 provider 可以采用不同机制，但必须保持 相同的 ownership-until-completion 合同。 +## 检查点制品捕获与发布 + +检查点捕获是存储提供者的一项独立显式能力。`StorageProvider` 默认声明不支持并 +返回错误，因此现有提供者不会在没有完整实现时被视为支持捕获。文件存储提供者会 +根据 sandbox ID 重建规范 slot，保留已经打开的源文件，并把可写根目录复制到检查点 +事务持有的私有目标中。复制会尽量保留稀疏区间，且绝不替换已有目标。 + +检查点目录从已经保留的 state-root 目录派生。每个 sandbox 都有私有暂存条目、 +已经提交的检查点目录和一个 HEAD 引用。发布过程会校验必需的后端状态、内存和 +可写根制品，记录大小与 SHA-256 摘要，同步文件和目录，再通过不替换已有目标的 +重命名发布检查点。只有发布已经持久化后，才会原子更新 HEAD。查询历史前会重新 +打开并校验已经提交的清单和制品，再报告历史与 HEAD 可达性。 + +请求取消后，阻塞文件复制、清单发布和 HEAD 更新仍由监督任务继续执行，并持续 +持有 sandbox 操作锁，直到结果确定。能够确认的发布前失败只删除私有暂存条目。 +发布结果不确定时，不能删除身份无法确认的路径。销毁 sandbox 时，会在同一个 +state-root 所有权边界内删除事务制品和已经提交的检查点历史。本协议不提供检查点 +恢复、删除或清理。 + ## 能力边界 每次 provider 同步调用会持久化本次调用可见、且已经写入的制品字节与目录元数据。