From 5ef5d6489958234055e64a939866f02044ee9717 Mon Sep 17 00:00:00 2001 From: Weisson Date: Thu, 20 Aug 2026 11:50:16 +0800 Subject: [PATCH] feat(blaze): create sandboxes from templates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The template catalog could publish and inspect runtime artifact sets, but sandbox creation could not turn a published entry into a running sandbox. Add an explicit, policy-gated `template` field to `POST /v1/sandboxes`. Before any lifecycle state is written, the manager confirms the name is allowed by the matched policy, resolves the catalog entry, re-hashes every artifact against its manifest, and validates the recorded image identity, backend, exact backend version, snapshot kind, and — for Firecracker — the guest-transport and VM shape the policy would launch. It then materializes an independent provider-owned slot from the artifacts, restores the backend with a fresh network allocation when the template is networked, waits for guest readiness, and persists the catalog name on the instance. Storage growth: a new `StorageProvider::acquire_template` copies and revalidates the open VM-state, memory, and rootfs objects into writable per-sandbox storage, so every template-backed sandbox owns a complete copy that can be checkpointed, rolled back, and deleted independently without touching the catalog. Firecracker host rebinding: a snapshot records its root drive's host path, and `PUT /snapshot/load` overrides only the network and vsock resources. Each owner therefore binds its own rootfs onto one stable in-namespace path, which the recorded machine configuration names, so a snapshot captured by one sandbox restores against the restoring sandbox's independent copy instead of the capture-time disk. The launch already ran inside a private mount namespace, so this adds the bind step and its shared mount target only, and `sh` and `mount` join the protected PATH helper set the launch now depends on. The fixed target is reserved from template catalog roots, literal and resolved target alike, so startup accounting cannot mistake it for a published entry. Cross-sandbox restore: a template capture records its source sandbox, so `RestoreRequest` now states whether a snapshot came from another sandbox. A rollback still requires an adapter-recorded identity to name the restoring sandbox; a template restore accepts the source's identity, which is what lets a mock-backed template create succeed instead of failing after allocation. Copy, restore, readiness, and final-state failures reuse the existing recoverable create cleanup, retaining residual storage for a later destroy when rollback cannot complete. The daemon adds no second template registry or API namespace; `/v1/templates` remains the single template resource. Signed-off-by: Weisson --- docs/user-guide/en/runtime/blaze.md | 123 +++++- docs/user-guide/zh/runtime/blaze.md | 107 ++++- src/blaze/README.md | 6 +- src/blaze/README_zh.md | 4 +- src/blaze/crates/blaze-core/src/backend.rs | 9 + src/blaze/crates/blaze-core/src/lifecycle.rs | 17 + src/blaze/crates/blaze-core/src/storage.rs | 58 +++ src/blaze/crates/blazed/src/api.rs | 328 ++++++++++++++ src/blaze/crates/blazed/src/file_provider.rs | 313 ++++++++++++- .../crates/blazed/src/sandbox/hibernate.rs | 2 + .../crates/blazed/src/sandbox/manager.rs | 412 ++++++++++++++++-- .../crates/blazed/src/sandbox/restore.rs | 2 + .../crates/blazed/src/sandbox/template.rs | 386 +++++++++++++++- src/blaze/crates/blazed/src/spawner.rs | 145 +++++- .../crates/blazed/src/spawner/firecracker.rs | 298 +++++++++++-- src/blaze/docs/design/template-catalog.md | 104 ++++- src/blaze/docs/design/template-catalog_zh.md | 84 +++- 17 files changed, 2284 insertions(+), 114 deletions(-) diff --git a/docs/user-guide/en/runtime/blaze.md b/docs/user-guide/en/runtime/blaze.md index 7b856f34d7..6947ce0717 100644 --- a/docs/user-guide/en/runtime/blaze.md +++ b/docs/user-guide/en/runtime/blaze.md @@ -466,12 +466,18 @@ completes. Daemon-wide connection draining and runtime cleanup remain separate. Blaze can atomically publish operator-prepared runtime artifacts and expose their metadata through the daemon API. `/v1/templates` is the single -operator-facing template resource. Publishing an entry does not yet make -sandbox creation select or boot it. - -Future sandbox-create support will resolve an optional template name from this -same catalog; there is no separate process-local registry for operators to -configure or monitor. +operator-facing template resource. A `POST /v1/sandboxes` request selects a +published entry through the optional `template` field, and the daemon restores +the new sandbox from that entry. + +Sandbox creation resolves the optional template name from this same catalog; +there is no separate process-local registry for operators to configure or +monitor. The named entry must appear in the matched policy's `select.templates` +allow-list, and its recorded image, backend, version, and (for Firecracker) VM +and guest-transport shape must match what the policy would launch. Each +template-backed sandbox receives an independent copy of the artifacts, so it can +be checkpointed, rolled back, and deleted like any other sandbox without +affecting the catalog or its siblings. ### Configuration @@ -495,7 +501,13 @@ instance, and policy roots, from every executable path configured in file is opened for this startup, from that file's configured pathname, and from the configured `daemon.socket` path and the host network coordination path `/run/lock/blaze-network.lock`. They must also remain disjoint from the -conventional named network namespace trees `/var/run/netns` and `/run/netns`. +conventional named network namespace trees `/var/run/netns` and `/run/netns`, +and from the fixed snapshot-view rootfs path +`/run/blaze-snapshot-view/rootfs.ext4`. Every Firecracker sandbox creates that +file as the bind-mount target for its own root filesystem, so a catalog root +configured at `/run/blaze-snapshot-view` — or reachable through a symbolic link +that resolves there — is rejected at startup rather than allowed to accumulate a +root-level file that catalog accounting would read as a malformed entry. Relative `[backends]` paths are resolved once against the daemon's startup working directory; boundary checks, backend probing, and sandbox launch then reuse that absolute path. When a configured backend path is a symbolic link, @@ -542,6 +554,98 @@ The source contains top-level regular files `vmstate.snap`, `mem.bin`, and directories and files must be owned by the daemon user and not writable by group or other users. Nested directories, links, and special files are rejected. + +An entry that a create request will select must carry complete boot metadata in +`template.json`. Import itself only checks that the file is a JSON object, so an +entry without this metadata publishes successfully and is then rejected with +`409 Conflict` at create time: + +| Field | Meaning | +|-------|---------| +| `format_version` | Must be `1` | +| `name` | Must equal the published catalog name | +| `image_digest` | Image identity the create request must also declare | +| `backend` | Backend that captured the snapshot | +| `backend_version` | Must equal the version the backend's restore adapter reports; `mock-v1` for the built-in Mock backend, and the exact capturing binary version for Firecracker | +| `boot_args` | Firecracker kernel command line captured in the snapshot; it must exactly match the selected policy's effective cold-start command line, including Blaze's fixed `ip=` argument when networking is enabled | +| `snapshot_kind` | Snapshot flavor, currently `full` | +| `expose_guest_socket` | Whether the captured runtime exposed the guest transport | +| `network` | Whether the captured runtime held a host network slot | +| `vcpus` / `memory_mib` | Firecracker VM shape captured in the snapshot; both must be non-zero and exactly match the selected policy | +| `rootfs_size` / `memory_size` | Byte sizes, must match `rootfs.ext4` and `mem.bin` | +| `artifacts` | Exactly three entries for `vmstate.snap`, `mem.bin`, and `rootfs.ext4`, each with `size_bytes` and a lowercase-hex `sha256` | + +Create compares the manifest's `backend`, `backend_version`, and `snapshot_kind` +against what the selected backend's restore adapter reports, and a mismatch is +refused with `501 Not Implemented` even though the entry published successfully. +The status depends on where the problem is caught: a Firecracker manifest that +omits `backend_version` fails the manifest's own bootability rules first and is +refused with `409 Conflict`, while a Mock manifest that omits it satisfies those +rules and is refused with `501` by the adapter comparison. +For example, the built-in Mock adapter reports `mock-v1`; recording `mock-v2` +also returns `501`, which means the manifest value must be corrected rather +than selecting a different backend. + +Firecracker entries additionally require `resource_layout = "portable-v1"`, a +present `boot_args` value, non-zero `vcpus` and `memory_mib`, and a `memory_size` +equal to `memory_mib` expressed in bytes. Those rules are also part of the +manifest's bootability check, so violating them yields `409 Conflict`. The +policy's effective cold-start kernel command line, VM shape, and guest-transport +settings must match these values exactly. When networking is enabled, the +effective command line includes the fixed `ip=` argument that Blaze appends. +Restore uses the command line captured in the snapshot rather than rebuilding it +from the current policy. +A missing or zero `vcpus`/`memory_mib`, or a VM shape that differs from the +policy, returns `409 Conflict` during preflight before lifecycle state or +storage allocation and therefore cannot leave a residual sandbox directory. + +The built-in Mock backend does not restore guest transport or host networking, +so Mock entries must set both `expose_guest_socket` and `network` to `false`. +Requesting either unsupported resource is refused with `501 Not Implemented` +before any sandbox lifecycle state is written. + +Template-backed create uses the same recoverable cleanup as ordinary create: + +- Policy, image, backend, version, VM-shape, and guest-transport refusals occur + before create intent or storage allocation. They return `409 Conflict` for a + request or manifest conflict, or `501 Not Implemented` for an unsupported + storage or restore capability, and leave no sandbox-owned storage. +- Copy, backend restore, guest-readiness, and final-state failures occur after + create intent. Blaze first tries to stop the backend, release storage, and + commit the sandbox as destroyed. If all compensation succeeds, it returns the + original error and retains no sandbox resources. +- Incomplete compensation returns HTTP 500 with an error beginning `operation + requires recovery`; the named sandbox remains in `RecoveryRequired` and may + retain its storage or backend owner. Send + `DELETE /v1/sandboxes/{id}` later to retry cleanup. + +```json +{ + "format_version": 1, + "name": "runtime-base", + "image_digest": "sha256:...", + "backend": "firecracker", + "backend_version": "Firecracker v1.16.0", + "resource_layout": "portable-v1", + "boot_args": "console=ttyS0 reboot=k panic=1 pci=off", + "snapshot_kind": "full", + "expose_guest_socket": false, + "network": false, + "vcpus": 1, + "memory_mib": 256, + "rootfs_size": 536870912, + "memory_size": 268435456, + "artifacts": [ + {"name": "vmstate.snap", "size_bytes": 14174, "sha256": "..."}, + {"name": "mem.bin", "size_bytes": 268435456, "sha256": "..."}, + {"name": "rootfs.ext4", "size_bytes": 536870912, "sha256": "..."} + ] +} +``` + +Every artifact is re-hashed against these values when a create request selects +the entry, so the digests must describe the published files exactly. + Published files must have exactly one hard link, and catalog entries and staging directories must remain on the catalog root's mount. Blaze stops rather than changing or traversing data that violates these boundaries. @@ -579,5 +683,6 @@ import. Graceful shutdown rejects new imports, cancels active copies, and waits for their file handles to close. The API validates artifact structure, not whether a snapshot can boot with a -particular backend. Sandbox create does not yet accept a template name, and the -catalog does not yet expose deletion or reference tracking. +particular backend; boot compatibility is checked only when a create request +selects the entry. The catalog does not yet expose deletion or reference +tracking. diff --git a/docs/user-guide/zh/runtime/blaze.md b/docs/user-guide/zh/runtime/blaze.md index 2a7bc616a6..a82b879901 100644 --- a/docs/user-guide/zh/runtime/blaze.md +++ b/docs/user-guide/zh/runtime/blaze.md @@ -386,11 +386,17 @@ runtime 清理仍属于独立职责。 ## Template Catalog Blaze 可以原子发布运维人员准备的 runtime artifact，并通过 daemon API 提供 -其 metadata。`/v1/templates` 是唯一面向运维人员的 template 资源；发布条目 -目前不会让 sandbox create 自动选择或启动它。 - -后续 sandbox create 支持会从同一个 catalog 解析可选的 template name；运维 -人员不需要配置或监控另一套进程内 registry。 +其 metadata。`/v1/templates` 是唯一面向运维人员的 template 资源； +`POST /v1/sandboxes` 请求通过可选的 `template` 字段选择已发布条目，daemon +会从该条目恢复新的 sandbox。 + +sandbox create 会从同一个 catalog 解析可选的 template name；运维人员不需要 +配置或监控另一套进程内 registry。所指定的条目必须出现在所匹配 policy 的 +`select.templates` 允许列表中，且其记录的镜像、backend、版本，以及（对 +Firecracker）VM 与 guest 通信规格必须与 policy 将要启动的一致。每个 +template-backed sandbox 都会获得 artifact 的独立副本，因此可以像其他 sandbox +一样做 checkpoint、rollback 和 delete，而不会影响 catalog 或同源的 +其他 sandbox。 ### 配置方法 @@ -412,7 +418,11 @@ policy 根目录、`[backends]` 中配置的任一 executable 路径、本次启 打开 daemon 配置文件时捕获的解析位置、该文件的配置路径或配置的 `daemon.socket` 路径以及宿主机网络协调路径 `/run/lock/blaze-network.lock` 重叠，也不能与宿主机上两种常见的命名网络空间 -目录 `/var/run/netns` 和 `/run/netns` 重叠。 +目录 `/var/run/netns` 和 `/run/netns` 重叠，还不能与固定的 snapshot view rootfs +路径 `/run/blaze-snapshot-view/rootfs.ext4` 重叠。每个 Firecracker sandbox 都会把 +该文件作为自身根文件系统的 bind-mount 目标，因此把 catalog 根目录配置在 +`/run/blaze-snapshot-view`（或通过符号链接解析到该位置）会在启动时被拒绝，而不是 +放任其中出现被 catalog 记账当成损坏条目的根级文件。 `[backends]` 中的相对路径会在启动时根据 daemon 的工作目录解析一次；目录边界 检查、backend probe 和 sandbox launch 随后复用该绝对路径。如果配置的 backend 路径 是符号链接，则该链接的配置位置及其解析目标都不能进入 template catalog ownership。 @@ -450,6 +460,87 @@ Content-Type: application/json `vmstate.snap`、`mem.bin` 和 `rootfs.ext4`；可选的 `template.json` 必须是 JSON object。源目录和文件必须属于 daemon 用户，且不能允许 group 或其他用户 写入。嵌套目录、链接和特殊文件都会被拒绝。 + +如果条目要被 create 请求选择，`template.json` 必须包含完整的启动元数据。导入本身 +只校验它是 JSON object，因此缺少这些元数据的条目仍能成功发布，但会在 create 时 +返回 `409 Conflict`： + +| 字段 | 含义 | +|------|------| +| `format_version` | 必须为 `1` | +| `name` | 必须与发布的 catalog 名称一致 | +| `image_digest` | 镜像标识，create 请求必须声明相同值 | +| `backend` | 捕获该快照的 backend | +| `backend_version` | 必须与该 backend restore adapter 报告的版本一致；内置 Mock backend 为 `mock-v1`，Firecracker 为捕获时的精确二进制版本 | +| `boot_args` | Firecracker 快照中捕获的内核启动参数，必须与所选 policy 冷启动时的实际参数完全一致；启用网络时包括 Blaze 自动追加的固定 `ip=` 参数 | +| `snapshot_kind` | 快照类型，当前为 `full` | +| `expose_guest_socket` | 捕获时是否暴露 guest 通信通道 | +| `network` | 捕获时是否持有宿主网络 slot | +| `vcpus` / `memory_mib` | Firecracker 快照中捕获的 VM 规格；两者必须非零，并与所选 policy 完全一致 | +| `rootfs_size` / `memory_size` | 字节大小，必须与 `rootfs.ext4`、`mem.bin` 一致 | +| `artifacts` | 恰好三项，对应 `vmstate.snap`、`mem.bin`、`rootfs.ext4`，每项含 `size_bytes` 和小写十六进制 `sha256` | + +create 会把清单中的 `backend`、`backend_version` 和 `snapshot_kind` 与所选 backend +的 restore adapter 报告值逐项比对，不一致时返回 `501 Not Implemented`，即使该条目 +本身已成功发布。具体状态码取决于问题在哪一步被发现：Firecracker 清单遗漏 +`backend_version` 会先违反清单自身的可启动性规则，返回 `409 Conflict`；而 Mock +清单遗漏该字段能通过这些规则，最终由 adapter 比对返回 `501`。 +例如，内置 Mock adapter 固定报告 `mock-v1`；清单填写 `mock-v2` 同样返回 +`501`，表示应修正清单值，而不是改用其他 backend。 + +Firecracker 条目还必须提供 `resource_layout = "portable-v1"`、`boot_args`、 +非零的 `vcpus` 与 `memory_mib`，且 `memory_size` 必须等于 `memory_mib` +换算成字节的值。这些同样属于清单可启动性校验，因此违反时返回 +`409 Conflict`。policy 冷启动时实际生效的内核启动参数、VM 规格与 guest 通信设置 +必须与这些值完全一致。启用网络时，实际启动参数包括 Blaze 自动追加的固定 `ip=` +参数。恢复使用快照中捕获的启动参数，不会根据当前 policy 重建。 +如果 `vcpus`/`memory_mib` 缺失或为零，或者 VM 规格与 policy 不一致，会在 +写入生命周期状态或分配存储之前返回 `409 Conflict`，因此不会留下残留的 +sandbox 目录。 + +内置 Mock backend 不支持恢复 guest 通信或宿主网络，因此 Mock 条目的 +`expose_guest_socket` 与 `network` 都必须为 `false`。请求任一不支持的资源会在写入 +任何 sandbox 生命周期状态之前返回 `501 Not Implemented`。 + +从 template 创建 sandbox 与普通创建使用相同的可恢复清理机制： + +- policy、镜像、backend、版本、VM 规格或 guest 通信校验失败时，尚未写入 create + intent，也未分配存储。请求或清单冲突返回 `409 Conflict`；存储或恢复能力 + 不支持时返回 `501 Not Implemented`，两者都不会留下 sandbox 独占存储。 +- artifact 复制、backend 恢复、guest 就绪或最终状态持久化失败时，create + intent 已经写入。Blaze 会先尝试停止 backend、释放存储，并把 sandbox 提交为 + `Destroyed`。如果补偿全部成功，返回原始错误，且不保留 sandbox 资源。 +- 补偿未完成时返回 HTTP 500，错误文本以 `operation requires recovery` + 开头；错误中指明的 sandbox 会保持 `RecoveryRequired`，并可能仍持有存储或 + backend owner。后续发送 `DELETE /v1/sandboxes/{id}` 可重试清理。 + +```json +{ + "format_version": 1, + "name": "runtime-base", + "image_digest": "sha256:...", + "backend": "firecracker", + "backend_version": "Firecracker v1.16.0", + "resource_layout": "portable-v1", + "boot_args": "console=ttyS0 reboot=k panic=1 pci=off", + "snapshot_kind": "full", + "expose_guest_socket": false, + "network": false, + "vcpus": 1, + "memory_mib": 256, + "rootfs_size": 536870912, + "memory_size": 268435456, + "artifacts": [ + {"name": "vmstate.snap", "size_bytes": 14174, "sha256": "..."}, + {"name": "mem.bin", "size_bytes": 268435456, "sha256": "..."}, + {"name": "rootfs.ext4", "size_bytes": 536870912, "sha256": "..."} + ] +} +``` + +create 请求选择条目时会按这些值重新校验每个 artifact 的摘要，因此摘要必须与已发布 +文件完全对应。 + 已发布文件只能有一个硬链接，catalog 条目和 staging 目录也必须留在 catalog 根目录所在的挂载点。发现不满足这些边界的数据时，Blaze 会停止处理，不会修改或 继续遍历这些数据。 @@ -480,5 +571,5 @@ catalog 根目录上取得并持续持有独占锁；使用同一 catalog 的第 检查或清理仍在使用的 staging 目录前直接失败。正常关闭时会拒绝新导入、取消 正在复制的任务，并等待相关文件句柄关闭。 -API 只校验 artifact 结构，不证明 snapshot 能在特定 backend 上启动。当前的 -sandbox create 不接受 template name，catalog 也尚未提供删除或引用跟踪。 +API 只校验 artifact 结构，不证明 snapshot 能在特定 backend 上启动；只有当 +create 请求选择该条目时才会核对启动兼容性。catalog 尚未提供删除或引用跟踪。 diff --git a/src/blaze/README.md b/src/blaze/README.md index 295afa3a37..427a0e7a02 100644 --- a/src/blaze/README.md +++ b/src/blaze/README.md @@ -205,9 +205,9 @@ fails, the startup warning includes the additional error and the durable record may still contain its previous state. Reconciliation continues with other accepted records. -The `/v1/templates` routes are the single operator-facing template catalog. -Importing an entry does not yet make sandbox creation select it; future create -support will resolve optional names from this same catalog. See the +The `/v1/templates` routes are the single operator-facing template catalog. A +create request restores from an entry through the optional `template` field on +`POST /v1/sandboxes`, resolved from this same catalog. See the [template catalog user guide](../../docs/user-guide/en/runtime/blaze.md#template-catalog) for configuration, accepted artifacts, limits, and publication rules. diff --git a/src/blaze/README_zh.md b/src/blaze/README_zh.md index e2ec42f70f..ba4909a4b9 100644 --- a/src/blaze/README_zh.md +++ b/src/blaze/README_zh.md @@ -189,8 +189,8 @@ Blaze 仍可读取旧版本写入的 `Reset`、`Warm` 和 `start_path = "warm"` 持久化也失败，启动警告会记录附加错误，磁盘上的记录可能仍是先前状态。其他已通过 校验的记录仍会继续恢复。 -`/v1/templates` 是唯一面向运维人员的 template catalog。导入条目目前不会让 -sandbox create 自动选择它；后续 create 支持会从同一个 catalog 解析可选名称。 +`/v1/templates` 是唯一面向运维人员的 template catalog。create 请求通过 +`POST /v1/sandboxes` 上可选的 `template` 字段从同一个 catalog 解析条目并恢复。 配置方法、接受的 artifact、上限和发布规则参见 [Template catalog 用户指南](../../docs/user-guide/zh/runtime/blaze.md#template-catalog)。 diff --git a/src/blaze/crates/blaze-core/src/backend.rs b/src/blaze/crates/blaze-core/src/backend.rs index b13c8005dd..849eed60fd 100644 --- a/src/blaze/crates/blaze-core/src/backend.rs +++ b/src/blaze/crates/blaze-core/src/backend.rs @@ -141,6 +141,15 @@ pub struct RestoreRequest { /// Carried so a restore does not silently stop recording console output for /// a sandbox whose operator asked for it. pub record_console_log: bool, + /// Whether the snapshot was captured by a different sandbox. + /// + /// A checkpoint restore reloads this sandbox's own capture, so a recorded + /// sandbox identity must still match. A template restore deliberately loads + /// one published capture into many new sandboxes, so its recorded identity + /// belongs to the source and cannot match. Adapters that bind a snapshot to + /// a sandbox identity use this to tell the two apart instead of dropping the + /// check for both. + pub snapshot_from_other_sandbox: bool, } /// Snapshot flavor requested from a backend. diff --git a/src/blaze/crates/blaze-core/src/lifecycle.rs b/src/blaze/crates/blaze-core/src/lifecycle.rs index 9ad1f90dcf..68e175fdbd 100644 --- a/src/blaze/crates/blaze-core/src/lifecycle.rs +++ b/src/blaze/crates/blaze-core/src/lifecycle.rs @@ -276,6 +276,9 @@ pub struct SandboxInstance { pub backend: BackendKind, pub workload_class: WorkloadClass, pub image_digest: String, + /// Catalog entry used to restore this sandbox, if it was template-backed. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub template: Option, pub start_path: StartPath, pub created_at: DateTime, pub updated_at: DateTime, @@ -306,6 +309,7 @@ impl SandboxInstance { backend, workload_class, image_digest, + template: None, start_path: StartPath::Cold, created_at: now, updated_at: now, @@ -850,9 +854,22 @@ mod tests { let loaded: SandboxInstance = serde_json::from_value(value).expect("legacy state"); assert!(loaded.operation.is_none()); assert!(loaded.last_checkpoint.is_none()); + assert!(loaded.template.is_none()); assert_eq!(loaded.backend_ownership, BackendOwnership::Unknown); } + #[test] + fn template_identity_round_trips() { + let tmp = tempfile::tempdir().expect("tmp"); + let mut instance = fresh(); + instance.template = Some("runtime-base".to_string()); + instance.persist(tmp.path()).expect("persist"); + + let loaded = SandboxInstance::load(tmp.path(), instance.id).expect("load"); + + assert_eq!(loaded.template.as_deref(), Some("runtime-base")); + } + #[test] fn legacy_reset_and_warm_states_deserialize() { let inst = fresh(); diff --git a/src/blaze/crates/blaze-core/src/storage.rs b/src/blaze/crates/blaze-core/src/storage.rs index 3aac9b8731..60f1e20ff7 100644 --- a/src/blaze/crates/blaze-core/src/storage.rs +++ b/src/blaze/crates/blaze-core/src/storage.rs @@ -5,6 +5,7 @@ //! (copy-on-write, content-addressable dedup) but present //! a uniform interface to the daemon layer. +use std::fs::File; use std::path::{Path, PathBuf}; use async_trait::async_trait; @@ -66,6 +67,40 @@ pub struct AcquireOpts { pub mem_size: u64, } +/// One already-open template artifact. +/// +/// The open file object binds later materialization to the object the catalog +/// validated, even if its catalog path is replaced afterward. +#[derive(Debug)] +pub struct TemplateArtifact { + /// Stable source object positioned at the beginning of the artifact. + pub file: File, + /// Exact byte length recorded by the template manifest. + pub size_bytes: u64, + /// Lowercase SHA-256 digest recorded by the template manifest. + pub sha256: String, +} + +/// Self-contained artifacts needed to restore one template. +#[derive(Debug)] +pub struct TemplateStorage { + /// Backend VM-state snapshot. + pub vmstate: TemplateArtifact, + /// Guest-memory snapshot. + pub memory: TemplateArtifact, + /// Independent root filesystem snapshot. + pub rootfs: TemplateArtifact, +} + +/// Provider-owned storage produced from one template. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct TemplateStorageSlot { + /// Writable storage owned by the new sandbox. + pub storage: StorageSlot, + /// Provider-owned backend payload ready for the restore adapter. + pub payload_dir: PathBuf, +} + /// Storage allocation failure with an optional residual slot owner. /// /// A provider returns `residual` only when rollback could not remove resources @@ -120,6 +155,29 @@ pub trait StorageProvider: Send + Sync { opts: &AcquireOpts, ) -> std::result::Result; + /// Materialize a self-contained template into a new owned slot. + /// + /// Providers must not retain paths into the catalog. Every artifact used + /// by the restored sandbox must be copied into provider-owned storage. + async fn acquire_template( + &self, + opts: &AcquireOpts, + source: TemplateStorage, + ) -> std::result::Result { + let _ = (opts, source); + Err(StorageAcquireError::clean(BlazeError::StorageError { + msg: "storage provider does not support templates".to_string(), + })) + } + + /// Report whether template materialization is implemented. + /// + /// The default is conservative so existing providers do not advertise a + /// data path they have not implemented. + fn supports_templates(&self) -> bool { + false + } + /// Release a storage slot (cleanup all associated resources). async fn release(&self, slot: StorageSlot) -> Result<()>; diff --git a/src/blaze/crates/blazed/src/api.rs b/src/blaze/crates/blazed/src/api.rs index 6a64eb06e8..4641dee8de 100644 --- a/src/blaze/crates/blazed/src/api.rs +++ b/src/blaze/crates/blazed/src/api.rs @@ -219,6 +219,9 @@ struct CreateInstanceReq { labels: HashMap, #[serde(default)] kernel_version: Option, + /// Optional published template to restore this sandbox from. + #[serde(default)] + template: Option, } #[derive(Debug, Serialize)] @@ -323,6 +326,7 @@ async fn create_sandbox(state: &Arc, body: &[u8]) -> Result, + memory: Vec, + rootfs: Vec, + } + + /// A spawner that refuses cold spawn and records restore inputs, then hands + /// off to the guest-ready mock owner so create reaches its readiness gate. + struct TemplateRestoreSpawner { + observed: Arc>>, + } + + #[async_trait] + impl BackendSpawner for TemplateRestoreSpawner { + async fn spawn( + &self, + _request: BackendSpawnRequest, + ) -> std::result::Result { + Err(SpawnFailure::clean(BlazeError::BackendError { + msg: "template create must use restore".to_string(), + })) + } + + async fn restore_capability( + &self, + _executable: Option<&crate::spawner::PinnedExecutable>, + ) -> blaze_core::Result> { + Ok(Some(blaze_core::backend::RestoreCapability { + backend: BackendKind::Mock, + version: Some("guest-mock-v1".to_string()), + snapshot_kind: blaze_core::backend::SnapshotKind::Full, + })) + } + + async fn restore( + &self, + request: crate::spawner::BackendRestoreRequest, + ) -> crate::spawner::RestoreResult { + let observed = ObservedTemplateRestore { + instance_id: request.instance_id, + preserve_network: request.preserve_network, + snapshot: tokio::fs::read(request.payload_dir.join("vmstate.snap")) + .await + .map_err(SpawnFailure::from)?, + memory: tokio::fs::read(request.payload_dir.join("memory.snap")) + .await + .map_err(SpawnFailure::from)?, + rootfs: tokio::fs::read(&request.storage.rootfs_path) + .await + .map_err(SpawnFailure::from)?, + }; + *self.observed.lock().expect("template observation") = Some(observed); + let spawn = BackendSpawnRequest::new( + blaze_core::backend::SpawnRequest { + instance_id: request.instance_id, + binary_path: request.binary_path.clone(), + storage: request.storage.clone(), + backend: BackendConfigs::default(), + vm: None, + }, + request.run_dir.clone(), + ) + .map_err(SpawnFailure::clean)?; + GuestMockSpawner.spawn(spawn).await + } + + async fn probe(&self, _binary_path: &Path) -> blaze_core::Result { + Ok(true) + } + + async fn cleanup_orphan( + &self, + instance_id: Uuid, + run_dir: &OwnedRunDir, + ) -> blaze_core::Result<()> { + GuestMockSpawner.cleanup_orphan(instance_id, run_dir).await + } + } + + /// Build a Mock-backend server state with one imported `runtime-base` + /// template. `allowed` controls whether the policy lists it as selectable. + async fn template_test_state( + temp: &tempfile::TempDir, + allowed: bool, + expose_guest_socket: bool, + ) -> ( + Arc, + Arc>>, + DaemonConfig, + ) { + let mut config = test_config(temp); + // The catalog refuses symlink components in its root. Resolve the + // temporary directory first so these tests also run where the system + // temporary path itself is a symlink, as on macOS. + let resolved = std::fs::canonicalize(temp.path()).expect("resolve temp root"); + config.daemon.state_dir = resolved.join("state"); + config.storage.images_dir = resolved.join("images"); + config.storage.instances_dir = resolved.join("instances"); + config.template.dir = resolved.join("templates"); + let import_root = resolved.join("imports"); + write_template_source(&import_root.join("source"), expose_guest_socket); + config.template.import_root = Some(import_root); + let binary = resolved.join("test-backend"); + std::fs::write(&binary, b"test backend").expect("backend fixture"); + // Preflight pins the configured executable, which requires the file to + // actually be executable. + { + use std::os::unix::fs::PermissionsExt; + std::fs::set_permissions(&binary, std::fs::Permissions::from_mode(0o755)) + .expect("backend fixture permissions"); + } + config.backends.insert("mock".to_string(), binary); + let storage: Arc = Arc::new(FileStorageProvider::with_images( + config.storage.images_dir.clone(), + config.storage.instances_dir.clone(), + )); + let observed = Arc::new(std::sync::Mutex::new(None)); + let mut policy = test_policy(BackendKind::Mock); + if allowed { + policy.select.templates = + vec!["runtime-base".to_string(), "missing-template".to_string()]; + } + let state = build_test_state( + config.clone(), + policy, + spawners( + BackendKind::Mock, + Arc::new(TemplateRestoreSpawner { + observed: observed.clone(), + }), + ), + BackendKind::Mock, + storage, + ); + state + .manager + .import_template( + "runtime-base".to_string(), + PathBuf::from("source"), + String::new(), + ) + .await + .expect("import template"); + (state, observed, config) + } + + #[tokio::test] + async fn template_create_restores_independent_sandboxes() { + let temp = tempfile::tempdir().expect("temp"); + let (state, observed, config) = template_test_state(&temp, true, false).await; + let request = serde_json::to_vec(&json!({ + "workload_class": "agent-tool", + "image_digest": "sha256:template-image", + "template": "runtime-base" + })) + .expect("create request"); + + let first = created_json(&state, &request).await; + let first_id = + Uuid::parse_str(first["instance"]["id"].as_str().expect("instance id")).expect("uuid"); + let first_restore = observed + .lock() + .expect("observation") + .take() + .expect("first restore"); + // Mutating one sandbox's private rootfs must not affect the next. + let first_rootfs = config + .storage + .instances_dir + .join(first_id.to_string()) + .join("rootfs.ext4"); + std::fs::write(&first_rootfs, b"cloned").expect("mutate first rootfs"); + + let second = created_json(&state, &request).await; + let second_id = + Uuid::parse_str(second["instance"]["id"].as_str().expect("instance id")).expect("uuid"); + let second_restore = observed + .lock() + .expect("observation") + .take() + .expect("second restore"); + let catalog_rootfs = config.template.dir.join("runtime-base/rootfs.ext4"); + + assert_ne!(first_id, second_id); + assert_eq!(first["instance"]["template"], "runtime-base"); + assert_eq!(second["instance"]["template"], "runtime-base"); + assert_eq!(first_restore.instance_id, first_id); + assert_eq!(second_restore.instance_id, second_id); + // A new sandbox never inherits the source network slot. + assert!(!first_restore.preserve_network); + // Each restore observed the published artifacts, byte for byte. + assert_eq!(first_restore.snapshot, b"snapshot"); + assert_eq!(second_restore.rootfs, b"rootfs"); + assert_eq!(first_restore.memory.len(), 1024 * 1024); + // The catalog copy is untouched by a per-sandbox mutation. + assert_eq!( + std::fs::read(&catalog_rootfs).expect("catalog rootfs"), + b"rootfs" + ); + assert_eq!( + std::fs::read(&first_rootfs).expect("first rootfs"), + b"cloned" + ); + } + + #[tokio::test] + async fn template_create_is_rejected_when_policy_disallows_it() { + let temp = tempfile::tempdir().expect("temp"); + let (state, observed, config) = template_test_state(&temp, false, false).await; + let instances_dir = config.storage.instances_dir.clone(); + + let error = create_sandbox( + &state, + &serde_json::to_vec(&json!({ + "workload_class": "agent-tool", + "image_digest": "sha256:template-image", + "template": "runtime-base" + })) + .expect("create request"), + ) + .await + .expect_err("policy must allow the template"); + + assert!(matches!(error, BlazeDaemonError::Conflict(_))); + assert!(observed.lock().expect("observation").is_none()); + assert!(state.manager.list().expect("instances").is_empty()); + assert_eq!( + std::fs::read_dir(instances_dir).expect("instances").count(), + 0 + ); + } + + #[tokio::test] + async fn template_create_rejects_mismatched_image_without_lifecycle_state() { + let temp = tempfile::tempdir().expect("temp"); + let (state, observed, config) = template_test_state(&temp, true, false).await; + let instances_dir = config.storage.instances_dir.clone(); + + let error = create_sandbox( + &state, + &serde_json::to_vec(&json!({ + "workload_class": "agent-tool", + "image_digest": "sha256:different-image", + "template": "runtime-base" + })) + .expect("create request"), + ) + .await + .expect_err("image identity must match the template"); + + assert!(matches!(error, BlazeDaemonError::Conflict(_))); + assert!(observed.lock().expect("observation").is_none()); + assert!(state.manager.list().expect("instances").is_empty()); + assert_eq!( + std::fs::read_dir(instances_dir).expect("instances").count(), + 0 + ); + } + + #[tokio::test] + async fn template_create_rejects_unsupported_mock_guest_socket_without_lifecycle_state() { + let temp = tempfile::tempdir().expect("temp"); + let (state, observed, config) = template_test_state(&temp, true, true).await; + let instances_dir = config.storage.instances_dir.clone(); + + let error = create_sandbox( + &state, + &serde_json::to_vec(&json!({ + "workload_class": "agent-tool", + "image_digest": "sha256:template-image", + "template": "runtime-base" + })) + .expect("create request"), + ) + .await + .expect_err("Mock cannot restore a guest transport"); + + assert!(matches!(error, BlazeDaemonError::UnsupportedOperation(_))); + assert!(observed.lock().expect("observation").is_none()); + assert!(state.manager.list().expect("instances").is_empty()); + assert_eq!( + std::fs::read_dir(instances_dir).expect("instances").count(), + 0 + ); + } } diff --git a/src/blaze/crates/blazed/src/file_provider.rs b/src/blaze/crates/blazed/src/file_provider.rs index 8111653b8b..aab0592fad 100644 --- a/src/blaze/crates/blazed/src/file_provider.rs +++ b/src/blaze/crates/blazed/src/file_provider.rs @@ -4,6 +4,7 @@ //! instance slots use separate roots. use std::ffi::OsString; +use std::io::{Read, Seek, SeekFrom}; use std::os::fd::OwnedFd; use std::path::{Component, Path, PathBuf}; use std::sync::Arc; @@ -12,12 +13,13 @@ use async_trait::async_trait; use rustix::fs::{ AtFlags, Mode, OFlags, RenameFlags, fstat, fsync, open, openat, renameat_with, statat, unlinkat, }; +use sha2::{Digest, Sha256}; use uuid::Uuid; use blaze_core::error::{BlazeError, Result}; use blaze_core::storage::{ AcquireOpts, PoolStatus, StorageAcquireError, StorageProvider, StorageRestoreTransaction, - StorageSlot, + StorageSlot, TemplateArtifact, TemplateStorage, TemplateStorageSlot, }; mod restore; @@ -388,6 +390,114 @@ impl StorageProvider for FileStorageProvider { Ok(slot) } + async fn acquire_template( + &self, + opts: &AcquireOpts, + source: TemplateStorage, + ) -> std::result::Result { + crate::failpoint::storage("storage-acquire-template")?; + if opts.rootfs_size != source.rootfs.size_bytes || opts.mem_size != source.memory.size_bytes + { + return Err(StorageAcquireError::clean(BlazeError::StorageError { + msg: format!( + "acquire template '{}': requested rootfs {} and memory {} do not match the \ + template artifacts {} and {}", + opts.instance_id, + opts.rootfs_size, + opts.mem_size, + source.rootfs.size_bytes, + source.memory.size_bytes + ), + })); + } + let slot = self.slot_for_id(&opts.instance_id)?; + let instance_dir = slot.instance_dir.clone(); + + match tokio::fs::create_dir(&instance_dir).await { + Ok(()) => {} + Err(error) if error.kind() == std::io::ErrorKind::AlreadyExists => { + return Err(StorageAcquireError::clean(BlazeError::StorageError { + msg: format!( + "acquire template '{}': instance directory already exists", + opts.instance_id + ), + })); + } + Err(error) => { + return Err(StorageAcquireError::clean(BlazeError::StorageError { + msg: format!( + "acquire template '{}': create dir: {error}", + opts.instance_id + ), + })); + } + } + + let payload_dir = instance_dir.join("backend"); + let snapshot_path = payload_dir.join("vmstate.snap"); + let payload_memory_path = payload_dir.join("memory.snap"); + let result = async { + tokio::fs::create_dir(&payload_dir).await?; + copy_template_artifact(source.rootfs, &slot.rootfs_path).await?; + copy_template_artifact(source.memory, &slot.mem_path).await?; + // The storage slot and restore payload refer to the same private + // memory image. A hard link gives the backend its payload name + // without duplicating a potentially large sparse file. + tokio::fs::hard_link(&slot.mem_path, &payload_memory_path).await?; + copy_template_artifact(source.vmstate, &snapshot_path).await?; + create_empty_durable_file(&slot.mem_diff_path).await?; + create_empty_durable_file(&slot.rootfs_diff_path).await?; + crate::failpoint::storage("storage-acquire-template-artifacts")?; + tokio::fs::File::open(&payload_dir) + .await? + .sync_all() + .await?; + tokio::fs::File::open(&instance_dir) + .await? + .sync_all() + .await?; + Ok::<(), BlazeError>(()) + } + .await; + + if let Err(error) = result { + let rollback = match crate::failpoint::storage("storage-acquire-rollback") { + Ok(()) => tokio::fs::remove_dir_all(&instance_dir) + .await + .map_err(BlazeError::from), + Err(cleanup) => Err(cleanup), + }; + return match rollback { + Ok(()) => Err(StorageAcquireError::clean(BlazeError::StorageError { + msg: format!( + "acquire template '{}': artifact setup failed, rolled back: {error}", + opts.instance_id + ), + })), + Err(cleanup) => Err(StorageAcquireError::with_residual( + BlazeError::StorageError { + msg: format!( + "acquire template '{}': artifact setup failed ({error}); rollback \ + failed for {}: {cleanup}", + opts.instance_id, + instance_dir.display() + ), + }, + slot, + )), + }; + } + + Ok(TemplateStorageSlot { + storage: slot, + payload_dir, + }) + } + + fn supports_templates(&self) -> bool { + true + } + async fn release(&self, slot: StorageSlot) -> Result<()> { crate::failpoint::storage("storage-release")?; // Re-derive the canonical path from instances_dir + slot.id. Do not @@ -692,6 +802,97 @@ async fn create_or_copy( Ok(()) } +/// Copy one template artifact into provider-owned storage and revalidate it. +/// +/// The source is an already-open object, so the copy cannot be redirected by +/// replacing a catalog path after validation. Size and digest are checked +/// again against the provider-owned destination after the sparse copy. Hashing +/// the copied object verifies the exact bytes the sandbox will use without +/// expanding holes in rootfs or guest-memory artifacts. +async fn copy_template_artifact(source: TemplateArtifact, target: &Path) -> Result<()> { + let metadata = source + .file + .metadata() + .map_err(|error| BlazeError::StorageError { + msg: format!("inspect template artifact: {error}"), + })?; + if !metadata.is_file() || metadata.len() != source.size_bytes { + return Err(BlazeError::StorageError { + msg: format!( + "template artifact has size {}; expected {}", + metadata.len(), + source.size_bytes + ), + }); + } + + let target = target.to_path_buf(); + let expected_size = source.size_bytes; + let expected_digest = source.sha256; + crate::failpoint::spawn_blocking(move || { + let mut destination = std::fs::OpenOptions::new() + .read(true) + .write(true) + .create_new(true) + .open(&target)?; + copy_sparse_file(&source.file, &destination)?; + + let copied = destination.metadata()?.len(); + if copied != expected_size { + return Err(BlazeError::StorageError { + msg: format!("template artifact has {copied} bytes; expected {expected_size}"), + }); + } + destination.seek(SeekFrom::Start(0))?; + let mut digest = Sha256::new(); + let mut buffer = [0_u8; 1024 * 1024]; + let mut hashed = 0_u64; + loop { + let read = destination.read(&mut buffer)?; + if read == 0 { + break; + } + hashed = hashed + .checked_add(u64::try_from(read).unwrap_or(u64::MAX)) + .ok_or_else(|| BlazeError::StorageError { + msg: "template artifact size overflow".to_string(), + })?; + digest.update(&buffer[..read]); + } + if hashed != expected_size { + return Err(BlazeError::StorageError { + msg: format!("template artifact has {hashed} bytes; expected {expected_size}"), + }); + } + let actual = format!("{:x}", digest.finalize()); + if actual != expected_digest { + return Err(BlazeError::StorageError { + msg: format!( + "template artifact digest mismatch: expected {expected_digest}, got {actual}" + ), + }); + } + destination.sync_all()?; + Ok(()) + }) + .await + .map_err(|error| BlazeError::StorageError { + msg: format!("copy template artifact task failed: {error}"), + })? +} + +/// Create one empty writable-diff file and persist its directory entry. +async fn create_empty_durable_file(path: &Path) -> Result<()> { + tokio::fs::OpenOptions::new() + .write(true) + .create_new(true) + .open(path) + .await? + .sync_all() + .await?; + Ok(()) +} + async fn canonical_plain_path(path: &Path, required_type: RequiredPathType) -> Result { let metadata = tokio::fs::symlink_metadata(path) @@ -1016,6 +1217,23 @@ fn validate_instance_id(instance_id: &str) -> Result<()> { mod tests { use super::*; + #[cfg(target_os = "linux")] + fn sha256_file(path: &Path) -> String { + use std::io::Read; + + let mut file = std::fs::File::open(path).expect("open digest source"); + let mut digest = Sha256::new(); + let mut buffer = [0_u8; 64 * 1024]; + loop { + let read = file.read(&mut buffer).expect("read digest source"); + if read == 0 { + break; + } + digest.update(&buffer[..read]); + } + format!("{:x}", digest.finalize()) + } + async fn checkpoint_fixture( instance_id: &str, ) -> (tempfile::TempDir, FileStorageProvider, StorageSlot, PathBuf) { @@ -1075,6 +1293,99 @@ mod tests { ); } + #[cfg(target_os = "linux")] + #[tokio::test] + async fn template_artifact_copy_preserves_sparse_regions_and_revalidates_digest() { + use std::io::{Read, Seek, Write}; + use std::os::unix::fs::MetadataExt; + + const LOGICAL_LEN: u64 = 64 * 1024 * 1024; + const FIRST_OFFSET: u64 = 8 * 1024; + const LAST_OFFSET: u64 = 48 * 1024 * 1024 + 91; + const FIRST_DATA: &[u8] = b"template-first-extent"; + const LAST_DATA: &[u8] = b"template-last-extent"; + + let temp = tempfile::tempdir().expect("temp"); + let source_path = temp.path().join("source.img"); + let mut source = std::fs::OpenOptions::new() + .read(true) + .write(true) + .create_new(true) + .open(&source_path) + .expect("source"); + source.set_len(LOGICAL_LEN).expect("logical source length"); + source + .seek(std::io::SeekFrom::Start(FIRST_OFFSET)) + .expect("first offset"); + source.write_all(FIRST_DATA).expect("first data"); + source + .seek(std::io::SeekFrom::Start(LAST_OFFSET)) + .expect("last offset"); + source.write_all(LAST_DATA).expect("last data"); + source.sync_all().expect("source sync"); + let source_blocks = source.metadata().expect("source metadata").blocks(); + drop(source); + let expected_digest = sha256_file(&source_path); + + let target_path = temp.path().join("target.img"); + copy_template_artifact( + TemplateArtifact { + file: std::fs::File::open(&source_path).expect("open source"), + size_bytes: LOGICAL_LEN, + sha256: expected_digest.clone(), + }, + &target_path, + ) + .await + .expect("copy sparse template artifact"); + + let metadata = std::fs::metadata(&target_path).expect("target metadata"); + assert_eq!(metadata.len(), LOGICAL_LEN); + assert!( + metadata.blocks().saturating_mul(512) < LOGICAL_LEN / 4, + "template copy allocated {} bytes for a {LOGICAL_LEN}-byte sparse source", + metadata.blocks().saturating_mul(512) + ); + assert!( + metadata.blocks() <= source_blocks.saturating_add(32), + "template copy used {} blocks for a source using {source_blocks} blocks", + metadata.blocks() + ); + assert_eq!(sha256_file(&target_path), expected_digest); + + let mut target = std::fs::File::open(&target_path).expect("target"); + let mut first = vec![0; FIRST_DATA.len()]; + target + .seek(std::io::SeekFrom::Start(FIRST_OFFSET)) + .expect("target first offset"); + target.read_exact(&mut first).expect("target first data"); + assert_eq!(first, FIRST_DATA); + let mut last = vec![0; LAST_DATA.len()]; + target + .seek(std::io::SeekFrom::Start(LAST_OFFSET)) + .expect("target last offset"); + target.read_exact(&mut last).expect("target last data"); + assert_eq!(last, LAST_DATA); + let mut hole = [1_u8; 4096]; + target + .seek(std::io::SeekFrom::Start(24 * 1024 * 1024)) + .expect("target hole offset"); + target.read_exact(&mut hole).expect("target hole"); + assert!(hole.iter().all(|byte| *byte == 0)); + + let mismatch = copy_template_artifact( + TemplateArtifact { + file: std::fs::File::open(&source_path).expect("reopen source"), + size_bytes: LOGICAL_LEN, + sha256: "0".repeat(64), + }, + &temp.path().join("digest-mismatch.img"), + ) + .await + .expect_err("digest mismatch"); + assert!(mismatch.to_string().contains("digest mismatch")); + } + #[tokio::test] async fn release_removes_instance_dir() { let tmp = tempfile::TempDir::new().unwrap(); diff --git a/src/blaze/crates/blazed/src/sandbox/hibernate.rs b/src/blaze/crates/blazed/src/sandbox/hibernate.rs index e723dd9c9d..032c4424a5 100644 --- a/src/blaze/crates/blazed/src/sandbox/hibernate.rs +++ b/src/blaze/crates/blazed/src/sandbox/hibernate.rs @@ -748,6 +748,8 @@ impl SandboxManager { // and a resume may run after a daemon restart. preserve_network: manifest.preserve_network, record_console_log: manifest.record_console_log, + // Resume reloads this sandbox's own hibernation image. + snapshot_from_other_sandbox: false, }, run_dir, executable.clone(), diff --git a/src/blaze/crates/blazed/src/sandbox/manager.rs b/src/blaze/crates/blazed/src/sandbox/manager.rs index c24f61b800..e151709bf9 100644 --- a/src/blaze/crates/blazed/src/sandbox/manager.rs +++ b/src/blaze/crates/blazed/src/sandbox/manager.rs @@ -7,7 +7,7 @@ use std::sync::{Arc, Mutex}; use std::time::Duration; use blaze_core::BlazeError; -use blaze_core::backend::{BackendKind, SpawnRequest}; +use blaze_core::backend::{BackendKind, RestoreRequest, SpawnRequest}; use blaze_core::lifecycle::{BackendOwnership, OperationKind, SandboxInstance, SandboxState}; use blaze_core::policy::RuntimeDecision; use blaze_core::storage::{AcquireOpts, StorageProvider, StorageSlot}; @@ -19,10 +19,10 @@ use crate::checkpoint_store::CheckpointStore; use crate::error::{BlazeDaemonError, Result}; use crate::guest::{GuestClient, GuestExecResult, MAX_GUEST_FILE_BYTES}; use crate::metrics::Metrics; -use crate::sandbox::template::TemplateCatalog; +use crate::sandbox::template::{ResolvedTemplate, TemplateCatalog}; use crate::spawner::{ - BackendSpawnRequest, DynBackendInstance, DynSpawner, SpawnerRegistry, - spawn_with_runtime_directory, + BackendRestoreRequest, BackendSpawnRequest, DynBackendInstance, DynSpawner, PinnedExecutable, + SpawnerRegistry, restore_with_runtime_directory, spawn_with_runtime_directory, }; use crate::state_store::{OwnedRunDir, StateStore}; @@ -39,6 +39,30 @@ pub struct CreateSandbox { pub runtime_backend: BackendKind, /// Executable selected during daemon startup. pub binary_path: PathBuf, + /// Published template to restore from, when the request named one. + pub template: Option, +} + +/// Prepared inputs for one template-backed create, validated before allocation. +struct TemplateCreate { + resolved: ResolvedTemplate, + spawner: DynSpawner, + executable: Option>, + /// Console-recording shape the matched policy would launch. + /// + /// A restore derives its effective backend config from the request, so this + /// must carry the policy's setting instead of silently disabling recording. + record_console_log: bool, +} + +/// Restore inputs derived from a materialized template slot. +struct TemplateRestore { + payload_dir: PathBuf, + expected_version: Option, + snapshot_kind: blaze_core::backend::SnapshotKind, + expose_guest_socket: bool, + preserve_network: bool, + record_console_log: bool, } /// Result of one managed create request. @@ -297,14 +321,156 @@ impl SandboxManager { .map_err(BlazeDaemonError::from) } - /// Create a sandbox from a fresh runtime allocation. + /// Validate a template-backed create before any lifecycle state is written. + /// + /// Returns `None` for an ordinary create. For a template request it checks + /// the policy allow-list, storage support, and catalog metadata, then + /// confirms the published snapshot's image, backend, version, kernel + /// command line, VM shape, and guest transport all match what the current + /// policy would launch. The pinned executable and resolved artifacts are + /// carried forward so the create path restores exactly what was validated. + async fn prepare_template_create( + &self, + request: &CreateSandbox, + ) -> Result> { + let Some(name) = request.template.as_ref() else { + return Ok(None); + }; + if !request + .decision + .templates + .iter() + .any(|allowed| allowed == name) + { + return Err(BlazeDaemonError::Conflict(format!( + "template {name} is not allowed by policy {}", + request.decision.policy_name + ))); + } + if !self.storage.supports_templates() { + return Err(BlazeDaemonError::UnsupportedOperation( + "configured storage does not support templates".to_string(), + )); + } + + let resolved = self.resolve_template_for_create(name.clone()).await?; + if resolved.image_digest != request.image_digest { + return Err(BlazeDaemonError::Conflict(format!( + "template {name} image identity does not match the create request" + ))); + } + if resolved.backend != request.runtime_backend { + return Err(BlazeDaemonError::Conflict(format!( + "template {name} requires backend {}, but the request selected {}", + resolved.backend, request.runtime_backend + ))); + } + + if resolved.backend == BackendKind::Firecracker { + let config = request + .decision + .backend + .firecracker + .as_ref() + .cloned() + .unwrap_or_default(); + if config.enable_vsock != resolved.expose_guest_socket + || config.enable_network != resolved.network + { + return Err(BlazeDaemonError::Conflict(format!( + "template {name} guest transport shape does not match policy {}", + request.decision.policy_name + ))); + } + let effective_boot_args = + crate::spawner::firecracker::effective_boot_args(&config, config.enable_network)?; + validate_template_boot_args( + name, + resolved.boot_args.as_deref(), + &effective_boot_args, + &request.decision.policy_name, + )?; + let (vcpus, memory_mib) = crate::spawner::firecracker::effective_vm_shape( + &config, + request.decision.vm.as_ref(), + )?; + if resolved.vcpus != Some(vcpus) || resolved.memory_mib != Some(memory_mib) { + return Err(BlazeDaemonError::Conflict(format!( + "template {name} VM shape does not match policy {}", + request.decision.policy_name + ))); + } + } else { + if resolved.expose_guest_socket { + return Err(BlazeDaemonError::UnsupportedOperation(format!( + "template {name} requests guest transport for unsupported backend {}", + resolved.backend + ))); + } + if resolved.network { + return Err(BlazeDaemonError::UnsupportedOperation(format!( + "template {name} requests networking for unsupported backend {}", + resolved.backend + ))); + } + } + + let spawner = self.spawner(resolved.backend).ok_or_else(|| { + BlazeDaemonError::UnsupportedOperation(format!( + "template {name} has no restore adapter for {}", + resolved.backend + )) + })?; + // A backend that runs no separate program of its own carries no + // configured path; pin one only when a real executable is configured. + let executable = if request.binary_path.as_os_str().is_empty() { + None + } else { + Some(Arc::new(PinnedExecutable::open(&request.binary_path)?)) + }; + let capability = spawner + .restore_capability(executable.as_deref()) + .await? + .ok_or_else(|| { + BlazeDaemonError::UnsupportedOperation(format!( + "template {name} backend {} does not support restore", + resolved.backend + )) + })?; + if capability.backend != resolved.backend + || capability.version != resolved.backend_version + || capability.snapshot_kind != resolved.snapshot_kind + { + return Err(BlazeDaemonError::UnsupportedOperation(format!( + "template {name} is incompatible with the current restore adapter" + ))); + } + + Ok(Some(TemplateCreate { + resolved, + spawner, + executable, + record_console_log: request + .decision + .backend + .firecracker + .as_ref() + .is_some_and(|config| config.serial_log), + })) + } + + /// Create a sandbox from a fresh runtime allocation or a published template. pub async fn create(&self, request: CreateSandbox) -> Result { + let template = self.prepare_template_create(&request).await?; let mut instance = SandboxInstance::new( request.runtime_backend, request.decision.workload_class, - request.image_digest, + request.image_digest.clone(), request.decision.policy_name.clone(), ); + instance.template = template + .as_ref() + .map(|template| template.resolved.name.clone()); let operation_lock = self.operation_lock(instance.id); let _operation = operation_lock.lock().await; instance.transition(SandboxState::Creating)?; @@ -340,19 +506,81 @@ impl SandboxManager { ))); } - let storage = match self - .storage - .acquire(&AcquireOpts { - instance_id: instance.id.to_string(), - rootfs_size: self.rootfs_size, - mem_size: self.mem_size, - }) - .await - { - Ok(storage) => storage, - Err(error) => { - let (source, residual) = error.into_parts(); - return Err(self.retain_failed_acquire(&mut instance, residual, source.into())); + let (storage, template_restore, template) = match template { + Some(TemplateCreate { + resolved, + spawner, + executable, + record_console_log, + }) => { + let ResolvedTemplate { + backend_version, + snapshot_kind, + expose_guest_socket, + network, + rootfs_size, + memory_size, + storage: source, + .. + } = resolved; + let materialized = match self + .storage + .acquire_template( + &AcquireOpts { + instance_id: instance.id.to_string(), + rootfs_size, + mem_size: memory_size, + }, + source, + ) + .await + { + Ok(materialized) => materialized, + Err(error) => { + let (source, residual) = error.into_parts(); + return Err(self.retain_failed_acquire( + &mut instance, + residual, + source.into(), + )); + } + }; + ( + materialized.storage, + Some(TemplateRestore { + payload_dir: materialized.payload_dir, + expected_version: backend_version, + snapshot_kind, + expose_guest_socket, + // A new sandbox never inherits the source's network + // slot, so a networked template requests a fresh one. + preserve_network: network, + record_console_log, + }), + Some((spawner, executable)), + ) + } + None => { + let storage = match self + .storage + .acquire(&AcquireOpts { + instance_id: instance.id.to_string(), + rootfs_size: self.rootfs_size, + mem_size: self.mem_size, + }) + .await + { + Ok(storage) => storage, + Err(error) => { + let (source, residual) = error.into_parts(); + return Err(self.retain_failed_acquire( + &mut instance, + residual, + source.into(), + )); + } + }; + (storage, None, None) } }; crate::failpoint::pause("create-after-storage-acquire").await; @@ -365,7 +593,11 @@ impl SandboxManager { .await); } }; - let spawner = match self.spawners.get(self.active_backend) { + let (spawner, template_executable) = match template { + Some((spawner, executable)) => (Some(spawner), executable), + None => (self.spawners.get(self.active_backend), None), + }; + let spawner = match spawner { Some(spawner) => spawner, None => { return Err(self @@ -408,31 +640,84 @@ impl SandboxManager { .await); } - let backend_request = match BackendSpawnRequest::new( - SpawnRequest { - instance_id: instance.id, - binary_path: request.binary_path, - storage: storage.clone(), - backend: request.decision.backend, - vm: request.decision.vm, - }, - work_dir.clone(), - ) { - Ok(request) => request, - Err(error) => { - instance.backend_ownership = BackendOwnership::NotStarted; - return Err(self - .cleanup_failed_create(&mut instance, storage, None, false, error.into()) - .await); + let template_backed = template_restore.is_some(); + let spawn = if let Some(template) = template_restore { + let restore_request = match BackendRestoreRequest::new( + RestoreRequest { + instance_id: instance.id, + binary_path: request.binary_path, + storage: storage.clone(), + payload_dir: template.payload_dir, + checkpoint_backend: instance.backend, + expected_version: template.expected_version, + snapshot_kind: template.snapshot_kind, + expose_guest_socket: template.expose_guest_socket, + preserve_network: template.preserve_network, + record_console_log: template.record_console_log, + // One published capture restores into many new sandboxes. + snapshot_from_other_sandbox: true, + }, + work_dir.clone(), + template_executable, + ) { + Ok(request) => request, + Err(error) => { + instance.backend_ownership = BackendOwnership::NotStarted; + return Err(self + .cleanup_failed_create(&mut instance, storage, None, false, error.into()) + .await); + } + }; + match crate::failpoint::backend("create-spawn") { + Ok(()) => restore_with_runtime_directory(spawner.as_ref(), restore_request).await, + Err(error) => Err(crate::spawner::SpawnFailure::clean(error)), + } + } else { + let backend_request = match BackendSpawnRequest::new( + SpawnRequest { + instance_id: instance.id, + binary_path: request.binary_path, + storage: storage.clone(), + backend: request.decision.backend, + vm: request.decision.vm, + }, + work_dir.clone(), + ) { + Ok(request) => request, + Err(error) => { + instance.backend_ownership = BackendOwnership::NotStarted; + return Err(self + .cleanup_failed_create(&mut instance, storage, None, false, error.into()) + .await); + } + }; + match crate::failpoint::backend("create-spawn") { + Ok(()) => spawn_with_runtime_directory(spawner.as_ref(), backend_request).await, + Err(error) => Err(crate::spawner::SpawnFailure::clean(error)), } - }; - let spawn = match crate::failpoint::backend("create-spawn") { - Ok(()) => spawn_with_runtime_directory(spawner.as_ref(), backend_request).await, - Err(error) => Err(crate::spawner::SpawnFailure::clean(error)), }; let actual_backend = match spawn { Ok(backend_instance) => { instance.backend_ownership = BackendOwnership::Running; + // A restore reloads a captured identity; refuse to adopt a + // backend owner whose identity diverges from durable state. + if template_backed + && (backend_instance.instance_id() != instance.id + || backend_instance.backend() != instance.backend) + { + return Err(self + .cleanup_failed_create( + &mut instance, + storage, + Some(backend_instance), + false, + BlazeDaemonError::Internal( + "restored backend owner identity does not match durable state" + .to_string(), + ), + ) + .await); + } let actual_backend = backend_instance.backend(); if let Err(error) = self .wait_for_guest_ready(&backend_instance, "create-guest-ready") @@ -1107,3 +1392,50 @@ fn requires_automatic_cleanup(instance: &SandboxInstance) -> bool { Some(OperationKind::Hibernate | OperationKind::Resume) ))) } + +/// Require the command line captured in a Firecracker snapshot to equal the +/// command line the matched policy would use for a cold start. +/// +/// Restore loads the captured machine configuration and does not call +/// `write_vm_config`, so accepting a mismatch would silently bypass current +/// policy controls. +fn validate_template_boot_args( + template_name: &str, + captured: Option<&str>, + expected: &str, + policy_name: &str, +) -> Result<()> { + if captured == Some(expected) { + return Ok(()); + } + Err(BlazeDaemonError::Conflict(format!( + "template {template_name} kernel boot arguments do not match policy {policy_name}" + ))) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn firecracker_template_boot_arguments_must_match_the_policy_exactly() { + validate_template_boot_args( + "runtime-base", + Some("console=ttyS0 panic=1"), + "console=ttyS0 panic=1", + "agent-tool", + ) + .expect("identical command lines"); + + for captured in [None, Some("console=ttyS0 panic=2")] { + let error = validate_template_boot_args( + "runtime-base", + captured, + "console=ttyS0 panic=1", + "agent-tool", + ) + .expect_err("missing or different command lines must be rejected"); + assert!(matches!(error, BlazeDaemonError::Conflict(_))); + } + } +} diff --git a/src/blaze/crates/blazed/src/sandbox/restore.rs b/src/blaze/crates/blazed/src/sandbox/restore.rs index 826c053f43..62e20df923 100644 --- a/src/blaze/crates/blazed/src/sandbox/restore.rs +++ b/src/blaze/crates/blazed/src/sandbox/restore.rs @@ -360,6 +360,8 @@ impl SandboxManager { expose_guest_socket, preserve_network, record_console_log, + // A rollback reloads this sandbox's own capture. + snapshot_from_other_sandbox: false, }, run_dir, executable.clone(), diff --git a/src/blaze/crates/blazed/src/sandbox/template.rs b/src/blaze/crates/blazed/src/sandbox/template.rs index 0d96d8bd0d..6fd6b8e624 100644 --- a/src/blaze/crates/blazed/src/sandbox/template.rs +++ b/src/blaze/crates/blazed/src/sandbox/template.rs @@ -6,7 +6,7 @@ use std::ffi::{CStr, CString, OsStr, OsString}; use std::fs::{self, File}; #[cfg(test)] use std::fs::{DirBuilder, OpenOptions}; -use std::io::{self, Read, Write}; +use std::io::{self, Read, Seek, SeekFrom, Write}; use std::mem::MaybeUninit; use std::os::fd::{AsRawFd, FromRawFd, IntoRawFd}; use std::os::unix::ffi::{OsStrExt, OsStringExt}; @@ -22,10 +22,14 @@ use std::sync::{Arc, Mutex}; use std::sync::atomic::{AtomicBool, Ordering}; use blaze_core::BlazeError; +use blaze_core::backend::{BackendKind, SnapshotKind}; use blaze_core::config::{PolicyLoadErrorMode, TemplateSection}; use blaze_core::error::ConfigErrorSource; +use blaze_core::storage::{TemplateArtifact, TemplateStorage}; use hyper::body::Bytes; +use serde::Deserialize; use serde_json::json; +use sha2::{Digest, Sha256}; use tokio::sync::{OwnedSemaphorePermit, Semaphore, watch}; use tokio_util::sync::CancellationToken; use uuid::Uuid; @@ -40,7 +44,9 @@ const CATALOG_FILE_MODE: u32 = 0o600; const COPY_BUFFER_BYTES: usize = 1024 * 1024; const LIST_RESPONSE_CONCURRENCY: usize = 1; const ITEM_RESPONSE_CONCURRENCY: usize = 1; -const HOST_PATH_HELPERS: [&str; 5] = ["ip", "iptables", "kill", "sysctl", "unshare"]; +// `sh` and `mount` join the set because the Firecracker launch now binds the +// sandbox's rootfs onto the portable snapshot-view path before exec. +const HOST_PATH_HELPERS: [&str; 7] = ["ip", "iptables", "kill", "mount", "sh", "sysctl", "unshare"]; #[derive(Clone, Copy)] struct ImportLimits { @@ -84,6 +90,54 @@ pub(crate) struct TemplateCatalog { inner: Arc, } +/// Boot metadata a published `template.json` must carry for create to consume. +#[derive(Debug, Deserialize)] +struct TemplateManifest { + format_version: u32, + name: String, + image_digest: String, + backend: BackendKind, + backend_version: Option, + resource_layout: Option, + boot_args: Option, + snapshot_kind: SnapshotKind, + expose_guest_socket: bool, + network: bool, + vcpus: Option, + memory_mib: Option, + rootfs_size: u64, + memory_size: u64, + artifacts: Vec, +} + +#[derive(Debug, Deserialize)] +struct TemplateArtifactManifest { + name: String, + size_bytes: u64, + sha256: String, +} + +/// Validated launch semantics and stable artifact objects for one create. +/// +/// Each artifact is an already-open file object, so the create path copies the +/// bytes the catalog validated even if a catalog path is replaced afterward. +#[derive(Debug)] +pub(super) struct ResolvedTemplate { + pub(super) name: String, + pub(super) image_digest: String, + pub(super) backend: BackendKind, + pub(super) backend_version: Option, + pub(super) boot_args: Option, + pub(super) snapshot_kind: SnapshotKind, + pub(super) expose_guest_socket: bool, + pub(super) network: bool, + pub(super) vcpus: Option, + pub(super) memory_mib: Option, + pub(super) rootfs_size: u64, + pub(super) memory_size: u64, + pub(super) storage: TemplateStorage, +} + struct CatalogInner { root: File, import_root: Option, @@ -487,6 +541,23 @@ impl TemplateCatalog { Ok(Bytes::from_owner(owner)) } + /// Resolve one published template into validated launch semantics and open + /// artifact objects for a single create request. + async fn resolve_for_create(&self, name: String) -> Result { + validate_name(&name, "runtime template")?; + let catalog = self.clone(); + tokio::task::spawn_blocking(move || { + resolve_published( + &catalog.inner.root, + &name, + catalog.inner.limits, + catalog.inner.boundary, + ) + }) + .await + .map_err(join_error("runtime template resolve"))? + } + async fn import( &self, name: String, @@ -792,6 +863,14 @@ impl SandboxManager { self.template_catalog.get(name).await } + /// Resolve one published template into validated launch inputs for create. + pub(super) async fn resolve_template_for_create( + &self, + name: String, + ) -> Result { + self.template_catalog.resolve_for_create(name).await + } + /// Copy and atomically publish one operator-prepared artifact directory. pub async fn import_template( &self, @@ -2448,6 +2527,16 @@ fn validate_template_root_paths_with_mounts_and_policy_mode( for path in host_named_network_namespace_paths { push_configured_and_resolved_root(&mut roots, "host named network namespace", path)?; } + // Every Firecracker owner creates this fixed mount target before launching, + // and it does so through whatever the path resolves to. Reserve both the + // literal and its resolved target, so neither a catalog root configured at + // this path nor a symlinked parent pointing into one can end up owning the + // file, which startup accounting would read as a malformed published entry. + push_configured_and_resolved_root( + &mut roots, + "snapshot view rootfs", + Path::new(crate::spawner::firecracker::PORTABLE_ROOTFS_PATH), + )?; roots.extend(helper_executable_roots.iter().cloned()); let mut configured_backends = backend_binaries.iter().collect::>(); configured_backends.sort_unstable_by(|(left, _), (right, _)| left.cmp(right)); @@ -3203,6 +3292,254 @@ fn get_published( read_published_directory(&directory, name, limits, boundary) } +/// Resolve one published template into validated launch inputs. +/// +/// Structure and boundary checks match ordinary lookup, then the metadata is +/// parsed as boot metadata, cross-checked for backend consistency, and each +/// artifact is opened and re-hashed so create receives stable file objects. +fn resolve_published( + root: &File, + name: &str, + limits: ImportLimits, + boundary: CatalogBoundary, +) -> Result { + let directory = + match try_open_catalog_directory(root, OsStr::new(name), boundary, Path::new(name))? { + Some(directory) => directory, + None => { + return Err(BlazeDaemonError::NotFound(format!( + "runtime template {name}" + ))); + } + }; + let metadata = read_published_directory(&directory, name, limits, boundary)?; + let manifest = serde_json::from_value::(metadata).map_err(|error| { + BlazeDaemonError::Conflict(format!( + "runtime template {name} does not contain boot metadata: {error}" + )) + })?; + validate_template_manifest(name, &manifest)?; + + let mut artifacts = manifest + .artifacts + .iter() + .map(|artifact| (artifact.name.as_str(), artifact)) + .collect::>(); + let vmstate = open_verified_template_artifact( + &directory, + name, + artifacts + .remove("vmstate.snap") + .expect("validated VM-state manifest"), + boundary, + )?; + let memory = open_verified_template_artifact( + &directory, + name, + artifacts + .remove("mem.bin") + .expect("validated memory manifest"), + boundary, + )?; + let rootfs = open_verified_template_artifact( + &directory, + name, + artifacts + .remove("rootfs.ext4") + .expect("validated rootfs manifest"), + boundary, + )?; + + Ok(ResolvedTemplate { + name: manifest.name, + image_digest: manifest.image_digest, + backend: manifest.backend, + backend_version: manifest.backend_version, + boot_args: manifest.boot_args, + snapshot_kind: manifest.snapshot_kind, + expose_guest_socket: manifest.expose_guest_socket, + network: manifest.network, + vcpus: manifest.vcpus, + memory_mib: manifest.memory_mib, + rootfs_size: manifest.rootfs_size, + memory_size: manifest.memory_size, + storage: TemplateStorage { + vmstate, + memory, + rootfs, + }, + }) +} + +/// Check that manifest metadata describes a self-consistent bootable template. +/// +/// Firecracker entries carry the stricter contract: a pinned backend version, +/// the `portable-v1` resource layout, the captured kernel command line, +/// non-zero VM shape, and a `memory_size` that equals `memory_mib` expressed in +/// bytes. All backends must describe the three known artifacts exactly once +/// with lowercase-hex digests and sizes that agree with the recorded rootfs and +/// memory sizes. +fn validate_template_manifest(expected_name: &str, manifest: &TemplateManifest) -> Result<()> { + let invalid = |reason: &str| { + BlazeDaemonError::Conflict(format!( + "runtime template {expected_name} is not bootable: {reason}" + )) + }; + if manifest.format_version != 1 { + return Err(invalid("format_version must be 1")); + } + if manifest.name != expected_name { + return Err(invalid("metadata name does not match the catalog name")); + } + if manifest.image_digest.trim().is_empty() { + return Err(invalid("image_digest must not be empty")); + } + if manifest.backend == BackendKind::Firecracker + && manifest + .backend_version + .as_deref() + .is_none_or(str::is_empty) + { + return Err(invalid("firecracker templates require backend_version")); + } + if manifest.backend == BackendKind::Firecracker + && manifest.resource_layout.as_deref() != Some("portable-v1") + { + return Err(invalid( + "firecracker templates require resource_layout portable-v1", + )); + } + if manifest.backend == BackendKind::Firecracker && manifest.boot_args.is_none() { + return Err(invalid("firecracker templates require boot_args")); + } + if manifest.backend == BackendKind::Firecracker + && (manifest.vcpus.is_none_or(|vcpus| vcpus == 0) + || manifest.memory_mib.is_none_or(|memory| memory == 0)) + { + return Err(invalid( + "firecracker templates require non-zero vcpus and memory_mib", + )); + } + if manifest.backend == BackendKind::Firecracker { + let expected_memory_size = manifest + .memory_mib + .expect("validated Firecracker memory") + .checked_mul(1024 * 1024) + .ok_or_else(|| invalid("memory_mib exceeds the supported artifact size"))?; + if manifest.memory_size != expected_memory_size { + return Err(invalid( + "memory_size must equal memory_mib expressed in bytes", + )); + } + } + if manifest.rootfs_size == 0 || manifest.memory_size == 0 { + return Err(invalid( + "rootfs_size and memory_size must be greater than zero", + )); + } + if manifest.artifacts.len() != 3 { + return Err(invalid( + "artifacts must describe vmstate.snap, mem.bin, and rootfs.ext4 exactly once", + )); + } + + let mut names = HashSet::new(); + for artifact in &manifest.artifacts { + if !matches!( + artifact.name.as_str(), + "vmstate.snap" | "mem.bin" | "rootfs.ext4" + ) || !names.insert(artifact.name.as_str()) + { + return Err(invalid( + "artifacts must describe vmstate.snap, mem.bin, and rootfs.ext4 exactly once", + )); + } + if artifact.size_bytes == 0 { + return Err(invalid("artifact sizes must be greater than zero")); + } + if artifact.sha256.len() != 64 + || !artifact + .sha256 + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) + { + return Err(invalid("artifact sha256 values must be lowercase hex")); + } + if artifact.name == "rootfs.ext4" && artifact.size_bytes != manifest.rootfs_size { + return Err(invalid("rootfs_size does not match rootfs.ext4")); + } + if artifact.name == "mem.bin" && artifact.size_bytes != manifest.memory_size { + return Err(invalid("memory_size does not match mem.bin")); + } + } + Ok(()) +} + +/// Open one published artifact and re-hash it against the manifest values. +/// +/// The returned open object binds later materialization to the exact bytes the +/// catalog validated, so a catalog path swapped after this check cannot change +/// what is copied into the sandbox. +fn open_verified_template_artifact( + directory: &File, + template: &str, + expected: &TemplateArtifactManifest, + boundary: CatalogBoundary, +) -> Result { + let mut file = open_published_artifact( + directory, + OsStr::new(template), + OsStr::new(&expected.name), + boundary, + )?; + let metadata = file.metadata()?; + validate_published_file(&metadata, template, OsStr::new(&expected.name))?; + if metadata.len() != expected.size_bytes { + return Err(BlazeDaemonError::RecoveryRequired(format!( + "runtime template {template} artifact {} has {} bytes; metadata records {}", + expected.name, + metadata.len(), + expected.size_bytes + ))); + } + + let mut digest = Sha256::new(); + let mut remaining = expected.size_bytes; + let mut buffer = vec![0_u8; COPY_BUFFER_BYTES]; + while remaining > 0 { + let limit = usize::try_from(remaining.min(buffer.len() as u64)).unwrap_or(buffer.len()); + let read = file.read(&mut buffer[..limit])?; + if read == 0 { + return Err(BlazeDaemonError::RecoveryRequired(format!( + "runtime template {template} artifact {} ended before its declared size", + expected.name + ))); + } + digest.update(&buffer[..read]); + remaining -= u64::try_from(read).unwrap_or(remaining); + } + let mut trailing = [0_u8; 1]; + if file.read(&mut trailing)? != 0 { + return Err(BlazeDaemonError::RecoveryRequired(format!( + "runtime template {template} artifact {} exceeds its declared size", + expected.name + ))); + } + let actual = format!("{:x}", digest.finalize()); + if actual != expected.sha256 { + return Err(BlazeDaemonError::RecoveryRequired(format!( + "runtime template {template} artifact {} digest mismatch", + expected.name + ))); + } + file.seek(SeekFrom::Start(0))?; + Ok(TemplateArtifact { + file, + size_bytes: expected.size_bytes, + sha256: expected.sha256.clone(), + }) +} + fn read_published( root: &File, name: &str, @@ -3632,6 +3969,51 @@ fn lock_catalog_state(inner: &CatalogInner) -> std::sync::MutexGuard<'_, Catalog mod tests { use super::*; + fn firecracker_manifest(boot_args: Option<&str>) -> TemplateManifest { + let artifact = |name: &str, size_bytes: u64| TemplateArtifactManifest { + name: name.to_string(), + size_bytes, + sha256: "0".repeat(64), + }; + TemplateManifest { + format_version: 1, + name: "runtime-base".to_string(), + image_digest: "sha256:image".to_string(), + backend: BackendKind::Firecracker, + backend_version: Some("Firecracker v1.16.0".to_string()), + resource_layout: Some("portable-v1".to_string()), + boot_args: boot_args.map(str::to_string), + snapshot_kind: SnapshotKind::Full, + expose_guest_socket: false, + network: false, + vcpus: Some(1), + memory_mib: Some(1), + rootfs_size: 1, + memory_size: 1024 * 1024, + artifacts: vec![ + artifact("vmstate.snap", 1), + artifact("mem.bin", 1024 * 1024), + artifact("rootfs.ext4", 1), + ], + } + } + + #[test] + fn firecracker_template_manifest_requires_captured_boot_arguments() { + validate_template_manifest( + "runtime-base", + &firecracker_manifest(Some("console=ttyS0 panic=1")), + ) + .expect("captured command line"); + + let error = validate_template_manifest("runtime-base", &firecracker_manifest(None)) + .expect_err("missing command line must be rejected"); + assert!(matches!( + error, + BlazeDaemonError::Conflict(message) if message.contains("require boot_args") + )); + } + fn test_config(root: &Path, import_root: &Path) -> TemplateSection { TemplateSection { dir: root.to_path_buf(), diff --git a/src/blaze/crates/blazed/src/spawner.rs b/src/blaze/crates/blazed/src/spawner.rs index c94340e05f..d8ca64f764 100644 --- a/src/blaze/crates/blazed/src/spawner.rs +++ b/src/blaze/crates/blazed/src/spawner.rs @@ -922,6 +922,7 @@ impl BackendSpawner for MockSpawner { checkpoint_backend, expected_version, snapshot_kind, + snapshot_from_other_sandbox, .. } = request.request; if checkpoint_backend != BackendKind::Mock @@ -944,11 +945,23 @@ impl BackendSpawner for MockSpawner { Ok(vmstate) => vmstate, Err(error) => return Err(SpawnFailure::clean(error)), }; + // A template capture belongs to its source sandbox, so it must carry a + // valid, non-nil identity that differs from the new owner. A rollback + // must instead name the sandbox being restored. The cross-sandbox flag + // relaxes equality only; it must not make a missing or malformed + // identity acceptable. + let recorded_identity = vmstate + .get("instance_id") + .and_then(serde_json::Value::as_str) + .and_then(|value| Uuid::parse_str(value).ok()) + .filter(|value| !value.is_nil()); + let identity_matches = match recorded_identity { + Some(recorded) if snapshot_from_other_sandbox => recorded != instance_id, + Some(recorded) => recorded == instance_id, + None => false, + }; if vmstate.get("format").and_then(serde_json::Value::as_str) != Some("blaze-mock-v1") - || vmstate - .get("instance_id") - .and_then(serde_json::Value::as_str) - != Some(instance_id.to_string().as_str()) + || !identity_matches || vmstate.get("kind").and_then(serde_json::Value::as_str) != Some("full") { return Err(SpawnFailure::clean(BlazeError::BackendError { @@ -2335,6 +2348,7 @@ mod tests { expose_guest_socket: true, preserve_network, record_console_log, + snapshot_from_other_sandbox: false, }; // The guest mock runs no pinned executable: it owns an in-process task // rather than a backend binary. @@ -2378,6 +2392,128 @@ mod tests { assert!(instance.snapshot(request).await.is_err()); } + /// A template capture records its source sandbox, so the mock adapter must + /// accept a differing identity for a template restore while still refusing + /// one for a same-sandbox rollback. + #[tokio::test] + async fn mock_restore_accepts_a_foreign_identity_only_for_templates() { + for from_other_sandbox in [false, true] { + let temp = tempfile::tempdir().expect("temp"); + let spawn = request(temp.path()); + let run_dir = spawn.run_dir.clone(); + let payload_dir = temp.path().join("payload"); + std::fs::create_dir(&payload_dir).expect("payload directory"); + let snapshot_path = payload_dir.join("vmstate.snap"); + let mem_path = payload_dir.join("memory.snap"); + // Record a different sandbox, exactly as a published template does. + std::fs::write( + &snapshot_path, + serde_json::to_vec(&serde_json::json!({ + "format": "blaze-mock-v1", + "instance_id": Uuid::new_v4(), + "kind": "full", + })) + .expect("mock vmstate"), + ) + .expect("write vmstate"); + std::fs::write(&mem_path, b"blaze-mock-memory-v1").expect("write memory"); + + let restore = BackendRestoreRequest::new( + RestoreRequest { + instance_id: spawn.instance_id, + binary_path: PathBuf::new(), + storage: spawn.storage.clone(), + payload_dir, + checkpoint_backend: BackendKind::Mock, + expected_version: Some("mock-v1".to_string()), + snapshot_kind: SnapshotKind::Full, + expose_guest_socket: false, + preserve_network: false, + record_console_log: false, + snapshot_from_other_sandbox: from_other_sandbox, + }, + run_dir, + None, + ) + .expect("restore request"); + + let restored = MockSpawner.restore(restore).await; + if from_other_sandbox { + let owner = restored.expect("template restore accepts a foreign identity"); + assert_eq!(owner.instance_id(), spawn.instance_id); + owner.kill().await.expect("release mock owner"); + } else { + let error = restored.err().expect("rollback rejects a foreign identity"); + assert!( + error + .to_string() + .contains("does not match the requested sandbox"), + "{error}" + ); + } + } + } + + #[tokio::test] + async fn mock_template_restore_requires_a_valid_foreign_identity() { + for identity_case in ["missing", "malformed", "nil", "target"] { + let temp = tempfile::tempdir().expect("temp"); + let spawn = request(temp.path()); + let payload_dir = temp.path().join("payload"); + std::fs::create_dir(&payload_dir).expect("payload directory"); + let snapshot_path = payload_dir.join("vmstate.snap"); + let mem_path = payload_dir.join("memory.snap"); + let mut vmstate = serde_json::json!({ + "format": "blaze-mock-v1", + "kind": "full", + }); + match identity_case { + "missing" => {} + "malformed" => vmstate["instance_id"] = serde_json::json!("not-a-uuid"), + "nil" => vmstate["instance_id"] = serde_json::json!(Uuid::nil()), + "target" => vmstate["instance_id"] = serde_json::json!(spawn.instance_id), + _ => unreachable!("covered identity case"), + } + std::fs::write( + &snapshot_path, + serde_json::to_vec(&vmstate).expect("mock vmstate"), + ) + .expect("write vmstate"); + std::fs::write(&mem_path, b"blaze-mock-memory-v1").expect("write memory"); + + let restore = BackendRestoreRequest::new( + RestoreRequest { + instance_id: spawn.instance_id, + binary_path: PathBuf::new(), + storage: spawn.storage.clone(), + payload_dir, + checkpoint_backend: BackendKind::Mock, + expected_version: Some("mock-v1".to_string()), + snapshot_kind: SnapshotKind::Full, + expose_guest_socket: false, + preserve_network: false, + record_console_log: false, + snapshot_from_other_sandbox: true, + }, + spawn.run_dir.clone(), + None, + ) + .expect("restore request"); + + let error = MockSpawner + .restore(restore) + .await + .err() + .expect("invalid template source identity must be rejected"); + assert!( + error + .to_string() + .contains("does not match the requested sandbox"), + "{identity_case}: {error}" + ); + } + } + #[tokio::test] async fn self_freezing_backends_can_bypass_pause_for_capture() { struct SelfFreezing; @@ -2431,6 +2567,7 @@ mod tests { expose_guest_socket: true, preserve_network: false, record_console_log: false, + snapshot_from_other_sandbox: false, }; let executable = Arc::new(PinnedExecutable::open(Path::new("/bin/sh")).expect("pin a real executable")); diff --git a/src/blaze/crates/blazed/src/spawner/firecracker.rs b/src/blaze/crates/blazed/src/spawner/firecracker.rs index f1af725182..fa43601fa8 100644 --- a/src/blaze/crates/blazed/src/spawner/firecracker.rs +++ b/src/blaze/crates/blazed/src/spawner/firecracker.rs @@ -38,6 +38,26 @@ use super::{ const NETWORK_BOOT_IP: &str = "ip=169.254.0.2::169.254.0.1:255.255.255.252::eth0:off"; const MAX_API_RESPONSE_BYTES: usize = 64 * 1024; +const FIRECRACKER_LAUNCH_TOOLS: [&str; 3] = ["unshare", "mount", "sh"]; +/// Stable in-namespace path every Firecracker owner sees as its root drive. +/// +/// A Firecracker snapshot records the block device's host path, and +/// `PUT /snapshot/load` overrides only the network and vsock resources. Binding +/// each sandbox's own rootfs onto one shared path keeps that recorded path valid +/// for any sandbox, which is what lets one published template restore into many +/// independent sandboxes. +pub(crate) const PORTABLE_ROOTFS_PATH: &str = "/run/blaze-snapshot-view/rootfs.ext4"; +#[cfg(target_os = "linux")] +const MOUNT_AND_EXEC: &str = r#"set -eu +rootfs_source=$1 +rootfs_target=$2 +binary=$3 +api_socket=$4 +instance_id=$5 +shift 5 +mount --bind "$rootfs_source" "$rootfs_target" +exec "$binary" --api-sock "$api_socket" --id "$instance_id" "$@" +"#; /// Slowest guest-memory throughput a snapshot deadline still tolerates. /// /// A control request such as `/version` or a pause should answer immediately, so @@ -149,6 +169,7 @@ impl FirecrackerSpawner { if restore.is_none() { validate_regular_file(&self.images_dir.join("vmlinux"), "vmlinux")?; } + prepare_portable_view_target().await?; let api_socket = request.run_dir.path().join("api.sock"); let guest_socket = request.run_dir.path().join("vsock.uds"); let pid_file = request.run_dir.path().join("firecracker.pid"); @@ -282,8 +303,13 @@ impl FirecrackerSpawner { Some(context) => context.executable.program(), None => request.binary_path.clone(), }; - let mut command = - build_launch_command(&program, network.as_ref(), &api_socket, request.instance_id); + let mut command = build_launch_command( + &program, + network.as_ref(), + &api_socket, + request.instance_id, + &request.storage.rootfs_path, + ); request.run_dir.inherit_into(&mut command); if let Some(context) = restore.as_ref() { context.executable.inherit_into(&mut command); @@ -575,6 +601,9 @@ impl BackendSpawner for FirecrackerSpawner { expose_guest_socket, preserve_network, record_console_log, + // Firecracker binds no sandbox identity into its snapshot, so a + // template capture and a rollback capture load the same way. + snapshot_from_other_sandbox: _, } = request.request; let executable = request .executable @@ -633,7 +662,7 @@ impl BackendSpawner for FirecrackerSpawner { } async fn probe(&self, binary_path: &Path) -> Result { - if !binary_path.is_file() || !executable_in_path("unshare") { + if !binary_path.is_file() || !firecracker_launch_tools_available(executable_in_path) { return Ok(false); } if !self.network_probe_ready().await? { @@ -1474,20 +1503,33 @@ impl FirecrackerInstance { } } -fn write_vm_config( - images_dir: &Path, - request: &BackendSpawnRequest, +/// Resolve the effective vCPU and memory-MiB shape for one Firecracker config. +/// +/// Template create uses this to confirm that a published snapshot's recorded VM +/// shape matches the shape the current policy would launch, using the same +/// precedence (backend override, then policy `[vm]`, then code default) as the +/// normal boot path in [`write_vm_config`]. +pub(crate) fn effective_vm_shape( config: &FirecrackerConfig, - guest_socket: &Path, - network: Option<&NetworkSlot>, -) -> Result { - let vcpus = config - .vcpus - .or(request.vm.as_ref().map(|vm| vm.vcpus)) - .unwrap_or(1); - let memory_mib = resolve_memory(config, request.vm.as_ref())?; + vm: Option<&VmConfig>, +) -> Result<(u32, u64)> { + let vcpus = config.vcpus.or(vm.map(|vm| vm.vcpus)).unwrap_or(1); + let memory_mib = resolve_memory(config, vm)?; + Ok((vcpus, memory_mib)) +} + +/// Resolve the kernel command line a cold start would write into Firecracker's +/// machine configuration. +/// +/// Networking uses one fixed guest address. Keep that derived argument in one +/// place so a template restore can compare its captured command line with the +/// exact command line a cold start under the same policy would use. +pub(crate) fn effective_boot_args( + config: &FirecrackerConfig, + network_enabled: bool, +) -> Result { let mut boot_args = config.boot_args.clone(); - if network.is_some() { + if network_enabled { let network_arguments = boot_args .split_whitespace() .filter(|argument| argument.starts_with("ip=")) @@ -1512,6 +1554,22 @@ fn write_vm_config( } } } + Ok(boot_args) +} + +fn write_vm_config( + images_dir: &Path, + request: &BackendSpawnRequest, + config: &FirecrackerConfig, + guest_socket: &Path, + network: Option<&NetworkSlot>, +) -> Result { + let vcpus = config + .vcpus + .or(request.vm.as_ref().map(|vm| vm.vcpus)) + .unwrap_or(1); + let memory_mib = resolve_memory(config, request.vm.as_ref())?; + let boot_args = effective_boot_args(config, network.is_some())?; let mut value = serde_json::json!({ "boot-source": { "kernel_image_path": path_string(&images_dir.join("vmlinux"), "vmlinux")?, @@ -1519,7 +1577,9 @@ fn write_vm_config( }, "drives": [{ "drive_id": "rootfs", - "path_on_host": path_string(&request.storage.rootfs_path, "rootfs")?, + // Name the stable in-namespace path, not this sandbox's own path, so + // a snapshot captured here stays loadable by another sandbox. + "path_on_host": PORTABLE_ROOTFS_PATH, "is_root_device": true, "is_read_only": false }], @@ -1564,11 +1624,75 @@ fn resolve_memory(config: &FirecrackerConfig, vm: Option<&VmConfig>) -> Result Result<()> { + let target = Path::new(PORTABLE_ROOTFS_PATH); + prepare_portable_view_target_at(target).await +} + +#[cfg(any(target_os = "linux", all(test, unix)))] +async fn prepare_portable_view_target_at(target: &Path) -> Result<()> { + if let Some(parent) = target.parent() { + tokio::fs::create_dir_all(parent) + .await + .map_err(|error| BlazeError::BackendError { + msg: format!( + "cannot create snapshot view directory {}: {error}", + parent.display() + ), + })?; + } + match tokio::fs::OpenOptions::new() + .write(true) + .create_new(true) + .open(target) + .await + { + Ok(_) => Ok(()), + Err(error) if error.kind() == std::io::ErrorKind::AlreadyExists => { + let metadata = tokio::fs::symlink_metadata(target).await.map_err(|error| { + BlazeError::BackendError { + msg: format!( + "cannot inspect snapshot view target {}: {error}", + target.display() + ), + } + })?; + if !metadata.file_type().is_file() { + return Err(BlazeError::BackendError { + msg: format!( + "snapshot view target {} is not a regular file", + target.display() + ), + }); + } + Ok(()) + } + Err(error) => Err(BlazeError::BackendError { + msg: format!( + "cannot create snapshot view target {}: {error}", + target.display() + ), + }), + } +} + +#[cfg(not(target_os = "linux"))] +async fn prepare_portable_view_target() -> Result<()> { + Ok(()) +} + fn build_launch_command( binary: &Path, network: Option<&NetworkSlot>, api_socket: &Path, instance_id: Uuid, + rootfs_source: &Path, ) -> Command { #[cfg(target_os = "linux")] let mut command = if let Some(network) = network { @@ -1581,8 +1705,7 @@ fn build_launch_command( .arg("--mount") .arg("--propagation") .arg("private") - .arg("--") - .arg(binary); + .arg("--"); command } else { let mut command = Command::new("unshare"); @@ -1590,17 +1713,36 @@ fn build_launch_command( .arg("--mount") .arg("--propagation") .arg("private") - .arg("--") - .arg(binary); + .arg("--"); command }; #[cfg(not(target_os = "linux"))] let mut command = { - let _ = network; + let _ = (network, rootfs_source); Command::new(binary) }; - command.arg("--api-sock").arg(api_socket); - command.arg("--id").arg(format!("fc-{instance_id}")); + // Bind this sandbox's own rootfs onto one stable path inside the private + // mount namespace, then exec Firecracker. The machine configuration a + // snapshot records therefore names a path that resolves to whichever + // sandbox is running, so a snapshot captured by one sandbox restores + // against the restoring sandbox's independent copy instead of the + // capture-time path. + #[cfg(target_os = "linux")] + command + .arg("sh") + .arg("-c") + .arg(MOUNT_AND_EXEC) + .arg("blaze-firecracker") + .arg(rootfs_source) + .arg(PORTABLE_ROOTFS_PATH) + .arg(binary) + .arg(api_socket) + .arg(format!("fc-{instance_id}")); + #[cfg(not(target_os = "linux"))] + { + command.arg("--api-sock").arg(api_socket); + command.arg("--id").arg(format!("fc-{instance_id}")); + } command } @@ -1829,6 +1971,10 @@ fn executable_in_path(name: &str) -> bool { std::env::split_paths(&path).any(|directory| is_executable_file(&directory.join(name))) } +fn firecracker_launch_tools_available(is_available: impl FnMut(&str) -> bool) -> bool { + FIRECRACKER_LAUNCH_TOOLS.into_iter().all(is_available) +} + fn is_executable_file(candidate: &Path) -> bool { if !candidate.is_file() { return false; @@ -2307,15 +2453,59 @@ mod tests { None, Path::new("/proc/self/fd/17/api.sock"), instance_id, + Path::new("/var/lib/blaze/instances/owner/rootfs.ext4"), ); - let arguments = command.as_std().get_args().collect::>(); - let id_index = arguments - .iter() - .position(|argument| *argument == "--id") - .expect("--id argument"); + let arguments = command + .as_std() + .get_args() + .map(|argument| argument.to_string_lossy().into_owned()) + .collect::>(); let expected = format!("fc-{instance_id}"); - assert_eq!(arguments.get(id_index + 1), Some(&expected.as_ref())); + assert!( + arguments.contains(&expected), + "backend id missing from {arguments:?}" + ); + } + + /// The launch must bind this sandbox's own rootfs onto the shared path the + /// recorded machine configuration names, or a restored snapshot would read + /// the capture-time disk instead of this sandbox's copy. + #[cfg(target_os = "linux")] + #[test] + fn launch_command_binds_the_owned_rootfs_to_the_portable_path() { + let owned = Path::new("/var/lib/blaze/instances/owner/rootfs.ext4"); + let command = build_launch_command( + Path::new("/usr/bin/firecracker"), + None, + Path::new("/proc/self/fd/17/api.sock"), + Uuid::new_v4(), + owned, + ); + let arguments = command + .as_std() + .get_args() + .map(|argument| argument.to_string_lossy().into_owned()) + .collect::>(); + + assert!( + arguments.iter().any(|argument| argument + == "mount --bind \"$rootfs_source\" \"$rootfs_target\"" + || argument.contains("mount --bind")), + "bind-mount step missing from {arguments:?}" + ); + assert!( + arguments + .iter() + .any(|argument| argument == owned.to_string_lossy().as_ref()), + "owned rootfs source missing from {arguments:?}" + ); + assert!( + arguments + .iter() + .any(|argument| argument == PORTABLE_ROOTFS_PATH), + "portable rootfs target missing from {arguments:?}" + ); } #[test] @@ -2426,6 +2616,24 @@ mod tests { ); } + #[test] + fn effective_boot_arguments_include_the_cold_start_network_argument() { + let config = FirecrackerConfig { + enable_network: true, + ..FirecrackerConfig::default() + }; + + assert_eq!( + effective_boot_args(&config, config.enable_network) + .expect("effective network command line"), + format!("{} {NETWORK_BOOT_IP}", config.boot_args) + ); + assert_eq!( + effective_boot_args(&config, false).expect("non-network command line"), + config.boot_args + ); + } + #[test] fn vm_config_accepts_the_matching_network_boot_argument() { let temp = tempfile::tempdir().expect("temp"); @@ -3385,6 +3593,38 @@ mod tests { assert!(spawner.probe(&invalid).await.expect("replaced probe")); } + #[test] + fn launch_tool_probe_requires_the_shell_used_by_the_mount_wrapper() { + let mut checked = Vec::new(); + + assert!(!firecracker_launch_tools_available(|tool| { + checked.push(tool.to_string()); + tool != "sh" + })); + assert_eq!(checked, FIRECRACKER_LAUNCH_TOOLS); + } + + #[cfg(unix)] + #[tokio::test] + async fn portable_view_target_rejects_existing_non_files() { + let temp = tempfile::tempdir().expect("temp"); + let directory = temp.path().join("directory"); + std::fs::create_dir(&directory).expect("directory target"); + + let directory_error = prepare_portable_view_target_at(&directory) + .await + .expect_err("directory target must be rejected"); + assert!(directory_error.to_string().contains("not a regular file")); + + let dangling = temp.path().join("dangling"); + std::os::unix::fs::symlink(temp.path().join("missing"), &dangling) + .expect("dangling target"); + let symlink_error = prepare_portable_view_target_at(&dangling) + .await + .expect_err("symlink target must be rejected"); + assert!(symlink_error.to_string().contains("not a regular file")); + } + #[cfg(target_os = "linux")] #[tokio::test] async fn instance_reports_version_and_captures_full_snapshot_over_uds() { diff --git a/src/blaze/docs/design/template-catalog.md b/src/blaze/docs/design/template-catalog.md index ee8bdd155b..3812bda9bd 100644 --- a/src/blaze/docs/design/template-catalog.md +++ b/src/blaze/docs/design/template-catalog.md @@ -7,12 +7,91 @@ configured by `template.dir`. Imports are disabled unless an operator also configures `template.import_root`. `/v1/templates` is the daemon's single public template resource. The catalog -provides durable publication and lookup, but sandbox creation does not yet -select catalog entries. +provides durable publication and lookup, and sandbox creation restores from a +catalog entry when a create request names one. + +A create request selects an entry through the optional `template` field on +`POST /v1/sandboxes`, resolved against this catalog. There is no second template +registry or other template API namespace. + +## Creating a sandbox from a catalog entry + +A create request restores from a published entry by setting the optional +`template` field to a catalog name: + +```json +{ + "workload_class": "agent-tool", + "image_digest": "sha256:...", + "template": "runtime-base" +} +``` + +The name must appear in the matched policy's `select.templates` allow-list; +otherwise the request is rejected before any lifecycle state is written. The +daemon then resolves the entry, re-hashes each artifact against the manifest, +and confirms the recorded image identity, backend, exact backend version, +snapshot kind, and — for Firecracker — the guest-transport and VM shape the +policy would launch. Only after these checks pass does it publish create +intent. + +The published `template.json` must therefore describe the entry completely: +`format_version` of `1`, a `name` equal to the catalog name, a non-empty +`image_digest`, the capturing `backend`, `snapshot_kind`, the +`expose_guest_socket` and `network` flags the capture ran with, non-zero +`rootfs_size` and `memory_size`, and exactly three `artifacts` entries for +`vmstate.snap`, `mem.bin`, and `rootfs.ext4`, each carrying `size_bytes` and a +lowercase-hex `sha256`. The recorded rootfs and memory sizes must agree with the +corresponding artifact sizes. Firecracker entries additionally require +`resource_layout` of `portable-v1`, the captured kernel `boot_args`, non-zero +`vcpus` and `memory_mib`, and a `memory_size` equal to `memory_mib` expressed in +bytes. + +`backend`, `backend_version`, and `snapshot_kind` are compared for equality +against what the selected backend's restore adapter reports, for every backend +rather than only for Firecracker. A manifest that omits `backend_version` +therefore publishes but cannot be created from: the built-in Mock adapter +reports `mock-v1`, and Firecracker reports its exact binary version. The two +refusals differ because they are caught at different stages — a violated +manifest rule is a conflict, while an adapter that cannot serve the recorded +identity is an unsupported operation. + +The built-in Mock adapter cannot restore guest transport or host networking, +so Mock entries must record both `expose_guest_socket` and `network` as `false`. +The daemon rejects either unsupported shape before publishing create intent. + +Restore adapters must also distinguish a template restore from a rollback. A +template snapshot records the source sandbox's identity but may create many new +sandboxes, so its restore request marks the snapshot as coming from another +sandbox. A rollback clears that marker because it must restore a checkpoint +captured by the same sandbox. Backends that record sandbox identity may relax +only that identity comparison for template restores; the format, snapshot kind, +backend, and backend-version checks remain unchanged. + +Because a Firecracker snapshot records the host path of its root drive and +`PUT /snapshot/load` overrides only the network and vsock resources, each owner +binds its own rootfs onto one stable in-namespace path that the recorded machine +configuration names. `portable-v1` is the label for that layout, which is what +allows one published entry to restore into many independent sandboxes. +Restore also retains the snapshot's captured kernel command line because it +does not write a new machine configuration. Template preflight therefore +requires `boot_args` to equal the matched policy's effective cold-start value +exactly, including the fixed `ip=` argument Blaze appends when networking is +enabled. +Before launch, the daemon creates the fixed mount target when absent and +refuses an existing directory, symbolic link, or other non-regular object. + +Materialization copies the VM-state, memory, and rootfs into a fresh +provider-owned slot, so every template-backed sandbox owns an independent copy; +mutating or destroying one never changes the catalog or another sandbox created +from the same entry. A networked template receives a fresh network allocation +rather than inheriting the source sandbox's slot. The sandbox is started +through the backend restore path, and its catalog name is persisted so ordinary +checkpoint, rollback, and delete continue to work. Copy, +restore, readiness, and final-state failures use the existing recoverable +create cleanup, retaining residual storage for a later destroy when rollback +cannot complete. -A later sandbox-create change will accept an optional template name and resolve -that name against this catalog. It will not add a second template registry or -another template API namespace. ## Import request @@ -34,7 +113,11 @@ by group or other users. The source must contain top-level regular files named `vmstate.snap`, `mem.bin`, and `rootfs.ext4`. An optional `template.json` must contain a JSON -object. Nested directories, links, and special files are rejected. The daemon +object. Import validates only that shape, so an entry intended for create must +additionally carry the complete boot manifest described in +[Creating a sandbox from a catalog entry](#creating-a-sandbox-from-a-catalog-entry); +without it the entry publishes successfully and is refused at create time. +Nested directories, links, and special files are rejected. The daemon sets `name` from the request, applies a non-empty request description, and fills `rootfs_size` and `memory_size` defaults when either field is absent or is not an unsigned integer. @@ -61,7 +144,10 @@ resolved location captured when the daemon configuration file is opened for this startup, that file's configured pathname, the `daemon.socket` path, or the host network coordination path `/run/lock/blaze-network.lock`. The conventional named network namespace trees `/var/run/netns` and `/run/netns` are protected -as well. +as well, as is the fixed snapshot-view rootfs path +`/run/blaze-snapshot-view/rootfs.ext4` that every Firecracker owner uses as its +bind-mount target. Both the literal and its resolved target are reserved, so a +symlinked parent cannot place that file inside a catalog root. Relative `[backends]` paths are resolved once against the daemon's startup working directory, and that absolute path is reused for boundary validation, probing, and launch. For a configured @@ -145,8 +231,8 @@ another item request receives `503 Service Unavailable` until the retained body is released. Corrupt published metadata is reported instead of silently hidden. These routes manage stored artifacts only. Validation is structural; it does not prove that a snapshot is bootable or compatible with a particular -backend. Sandbox creation does not yet accept a template name, and the catalog -does not yet expose deletion or reference tracking. +backend — that is confirmed only when a create request selects the entry. The +catalog does not yet expose deletion or reference tracking. The catalog limits above apply to imported artifacts and metadata. This change does not add a daemon-wide HTTP request-body limit; that input boundary must be diff --git a/src/blaze/docs/design/template-catalog_zh.md b/src/blaze/docs/design/template-catalog_zh.md index b200f66708..b2c9d82a4b 100644 --- a/src/blaze/docs/design/template-catalog_zh.md +++ b/src/blaze/docs/design/template-catalog_zh.md @@ -7,10 +7,74 @@ daemon 可以将一组可复用的 runtime artifact 发布到 `template.import_root` 后，导入功能才会启用。 `/v1/templates` 是 daemon 唯一公开的 template 资源。该 catalog 提供持久化 -发布和查询，但 sandbox create 目前不会选择其中的条目。 +发布和查询，当 create 请求指定某个条目时，sandbox create 会从该条目恢复。 + +create 请求通过 `POST /v1/sandboxes` 上可选的 `template` 字段选择条目，并从 +这个 catalog 解析该名称；不会再引入第二套 template registry 或另一个 +template API namespace。 + +## 使用 catalog 条目创建 sandbox + +create 请求通过把可选的 `template` 字段设为某个 catalog 名称，从已发布条目 +恢复： + +```json +{ + "workload_class": "agent-tool", + "image_digest": "sha256:...", + "template": "runtime-base" +} +``` + +该名称必须出现在所匹配 policy 的 `select.templates` 允许列表中，否则请求会在 +写入任何生命周期状态之前被拒绝。随后 daemon 解析该条目，对每个 artifact 按 +manifest 重新做摘要校验，并核对记录的镜像标识、backend、精确 backend 版本、 +快照类型，以及（对 Firecracker）policy 将要启动的 guest 通信与 VM 规格。只有 +这些校验全部通过后，才会发布 create intent。 + +因此已发布的 `template.json` 必须完整描述该条目：`format_version` 为 `1`、`name` +与 catalog 名称一致、非空的 `image_digest`、捕获时的 `backend`、`snapshot_kind`、 +捕获时的 `expose_guest_socket` 与 `network` 开关、非零的 `rootfs_size` 与 +`memory_size`，以及恰好三项 `artifacts`（`vmstate.snap`、`mem.bin`、 +`rootfs.ext4`），每项带 `size_bytes` 和小写十六进制 `sha256`。记录的 rootfs 与内存 +大小必须与对应 artifact 的大小一致。Firecracker 条目还必须提供 +`resource_layout` 为 `portable-v1`、捕获时的内核 `boot_args`、非零的 `vcpus` +与 `memory_mib`，且 `memory_size` 等于 `memory_mib` 换算后的字节数。 + +`backend`、`backend_version` 和 `snapshot_kind` 会与所选 backend restore adapter +报告的值做相等比对，这一比对适用于所有 backend，而不只是 Firecracker。因此缺少 +`backend_version` 的清单虽然能发布，却无法用于创建：内置 Mock adapter 报告 +`mock-v1`，Firecracker 报告其精确二进制版本。两种拒绝的语义不同，因为发现阶段不 +同——违反清单规则属于冲突，而 adapter 无法承载所记录的身份属于不支持的操作。 + +内置 Mock adapter 无法恢复 guest 通信或宿主网络，因此 Mock 条目必须把 +`expose_guest_socket` 与 `network` 都记录为 `false`。任一不支持的形态都会在发布 +create intent 之前被拒绝。 + +恢复适配器还必须区分“从模板恢复”和“回滚”。模板快照记录的是源沙箱标识， +但它可以用于创建多个新沙箱，所以恢复请求会标明快照来自其他沙箱。回滚请求 +不设置该标记，因为它只能恢复当前沙箱自己捕获的检查点。如果某个后端在 +快照中记录了沙箱标识，它只能在模板恢复时放宽这一项标识比较；格式、快照 +类型、后端及后端版本检查仍须保持不变。 + +由于 Firecracker 快照会记录其根设备的宿主路径，而 `PUT /snapshot/load` 只覆盖网络 +和 vsock 资源，因此每个 owner 都会把自己的 rootfs 绑定到一个稳定的命名空间内路径， +即记录在机器配置中的那个路径。`portable-v1` 就是这一布局的标记，也正是它让同一个 +已发布条目能够恢复成多个彼此独立的 sandbox。 +恢复同时会沿用快照中捕获的内核启动参数，因为该路径不会重新写入机器配置。 +因此 template 预检要求 `boot_args` 必须与所匹配 policy 冷启动时的实际值完全一致； +启用网络时，该值包括 Blaze 自动追加的固定 `ip=` 参数。 +启动前，daemon 会在固定 mount 目标不存在时创建普通文件；如果目标已经是目录、 +符号链接或其他非普通对象，则拒绝启动。 + +物化会把 VM-state、内存和 rootfs 复制到一个全新的 provider 独占 slot，因此每个 +template-backed sandbox 都拥有独立副本；修改或销毁其中一个，永远不会改变 +catalog 或从同一条目创建的其他 sandbox。带网络的 template 会获得一次新的网络 +分配，而不是继承源 sandbox 的 slot。sandbox 通过 backend restore 路径启动，其 +catalog 名称会被持久化，因此普通的 checkpoint、rollback 和 delete +仍可正常工作。复制、恢复、就绪和终态失败沿用既有的可恢复 create 清理； +当回滚无法完成时，会保留残留存储供后续 destroy 使用。 -后续 sandbox create 改动会接受可选的 template name，并从这个 catalog 解析 -该名称；不会再引入第二套 template registry 或另一个 template API namespace。 ## 导入请求 @@ -30,7 +94,10 @@ Content-Type: application/json 允许 group 或其他用户写入。 源目录必须包含顶层普通文件 `vmstate.snap`、`mem.bin` 和 `rootfs.ext4`。 -可选的 `template.json` 必须是 JSON object。嵌套目录、链接和特殊文件都会 +可选的 `template.json` 必须是 JSON object。导入只校验这一层结构，因此要供 create +选择的条目还必须带上 +[使用 catalog 条目创建 sandbox](#使用-catalog-条目创建-sandbox) 一节描述的完整启动 +清单；缺少清单的条目仍能发布，但会在 create 时被拒绝。嵌套目录、链接和特殊文件都会 被拒绝。daemon 会使用请求中的 `name`，采用请求中非空的 `description`， 并在 `rootfs_size` 或 `memory_size` 缺失或不是无符号整数时填入默认值。目标名称 已存在或同名导入正在执行时返回 `409 Conflict`。 @@ -53,7 +120,10 @@ policy 目录、`[backends]` 中配置的任一 executable 路径、本次 启动打开 daemon 配置文件时捕获的解析位置、该文件的配置路径以及 `daemon.socket` 路径或宿主机网络协调路径 `/run/lock/blaze-network.lock` 重叠。宿主机上两种常见的命名网络空间目录 -`/var/run/netns` 和 `/run/netns` 也属于受保护边界。 +`/var/run/netns` 和 `/run/netns` 也属于受保护边界；每个 Firecracker owner 用作 +bind-mount 目标的固定路径 `/run/blaze-snapshot-view/rootfs.ext4` 同样受保护。该 +路径的字面值与其解析目标都会被保留，因此符号链接父目录也无法把该文件放进 catalog +根目录。 `[backends]` 中的相对路径会根据 daemon 启动时的工作目录解析一次，目录边界 检查、backend probe 和 launch 随后复用该绝对路径。配置 backend 符号链接时，链接 位置和解析目标都不能进入 template catalog ownership。daemon 配置路径为符号链接时， @@ -117,8 +187,8 @@ daemon 持有的全部 runtime 资源属于其他关闭职责。 完整单项查询使用独立的 single-flight permit，同时覆盖 metadata 解析和返回的响应 body；在该 body 释放前，其他单项查询返回 `503 Service Unavailable`。已发布元数据 损坏时会返回错误，而不是静默隐藏条目。这些接口只管理已经保存的 artifact，校验 -范围仅限结构，不证明快照能够启动或与某个 backend 兼容。sandbox create 目前不接受 -template name，catalog 也尚未提供删除或引用跟踪。 +范围仅限结构，不证明快照能够启动或与某个 backend 兼容——只有当 create 请求 +选择该条目时才会核对启动兼容性。catalog 尚未提供删除或引用跟踪。 上述 catalog 上限针对导入的 artifact 和 metadata。本改动没有增加 daemon 级 HTTP request body 上限；该输入边界需要在生产发布前单独补齐。