Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 59 additions & 6 deletions docs/user-guide/en/runtime/blaze.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,12 +104,7 @@ Guest operations are available only while a sandbox is `Running` and its
backend reports a compatible guest endpoint. A cold create that reports such
an endpoint waits for the guest agent before publishing `Running`. Backends
without an endpoint, including production mock fallback, skip that wait and
return HTTP 409 for guest operations. Warm-pool activation validates the
retained backend owner and storage before publishing `Running`, but it does
not repeat the guest readiness probe. `Running` on this path therefore does
not guarantee that the guest endpoint is still responsive: the first guest
request performs the normal bounded connection and can return a guest error.
Callers should apply the retry and outcome rules below to that first request.
return HTTP 409 for guest operations.

Guest operations and lifecycle changes use the same per-sandbox operation
lock. After obtaining the lock, the manager checks `Running` again so a request
Expand Down Expand Up @@ -158,6 +153,64 @@ Leave `listen.http_addr` disabled in production until
Daemon shutdown also does not yet wait for every active HTTP handler or release
all runtime owners, so an in-flight request may observe a closed connection.

## Reset and Reusable-Instance Management

`POST /v1/instances/{id}/reset` does not report success until Blaze can reset
both runtime and storage. A malformed identifier returns HTTP 400, an unknown
instance returns HTTP 404, an instance that is not running returns HTTP 422,
and a running instance returns HTTP 501 without changing its state or owned
resources.

The four `/v1/pools` management routes also return HTTP 501. Blaze rejects
`storage.pool_size`, `storage.prefork`, and every `[pool]` section except the
exact historical package defaults. During an upgrade, it temporarily accepts
and ignores only those defaults from the older daemon configuration and two
default policy files, and logs a warning. This exception prevents an
administrator-modified file retained by RPM `%config(noreplace)` from blocking
the new daemon. It does not enable reusable instances. Merge each `.rpmnew`
file or remove the legacy section; later releases may remove this exception.
Any other policy `[pool]` section fails policy loading. At startup,
`policy.on_load_error = "fail"` stops the daemon, while `"warn"` starts with an
empty policy set. A failed administrative or signal-driven reload keeps the
currently active policies unchanged.

The accepted daemon section is exactly:

```toml
[pool]
default_warm_ttl = "30m"
gc_interval = "5m"
```

An accepted policy section must contain exactly these six fields and belong to
one of the two packaged policy identities:

| Policy name | Workload class | `min` | `target` | `max` |
|---|---|---:|---:|---:|
| `agent-rl-default` | `agent-rl` | 4 | 16 | 64 |
| `agent-tool-default` | `agent-tool` | 2 | 8 | 32 |

Both rows require `enabled = true`, `warm_ttl = "30m"`, and
`reset_mode = "full-recreate"`. A missing or additional field, a changed value
or type, a different policy name or workload class, or any other `[pool]`
section is rejected. Accepted compatibility values are ignored and omitted
when configuration is serialized.

Blaze continues to decode persisted `Reset`, `Warm`, and
`start_path = "warm"` values written by earlier releases. Startup
reconciliation treats non-terminal records containing those values as cleanup
candidates and never reuses them. A failed cleanup retains the in-memory record
as `RecoveryRequired` and attempts to persist that state. If persistence also
fails, the startup warning includes the additional error and the durable record
may still contain its previous state. Reconciliation continues with other
accepted records.
The metrics endpoint no longer publishes `blaze_instances_resets_total`,
`blaze_pool_hits_total`, or `blaze_pool_misses_total`.

The lifecycle invariants behind these compatibility responses are recorded in
the
[lifecycle state consistency and compatibility design](../../../../src/blaze/docs/design/lifecycle-state-consistency.md).

## Storage Artifact Synchronization

Blaze can periodically persist the already-written host artifacts and directory
Expand Down
54 changes: 49 additions & 5 deletions docs/user-guide/zh/runtime/blaze.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,11 +88,7 @@ Blaze 负责配置 sandbox 本地的网络路径。主机以外的路由和 DNS
只有 sandbox 处于 `Running` 且 backend 报告兼容的 guest endpoint 时,
才能执行 guest 操作。冷启动 backend 如果报告了该 endpoint,创建流程会在
发布 `Running` 前等待 guest agent。没有 endpoint 的 backend(包括生产环境
mock fallback)会跳过等待,guest 操作返回 HTTP 409。当前从 warm pool 激活
实例时,manager 会先验证保留的 backend owner 和 storage,再发布 `Running`,
但不会再次执行 guest readiness 探测。因此 warm 路径的 `Running` 不保证 guest
endpoint 仍然可响应;第一次 guest 请求仍会执行有界连接,并可能返回 guest
错误。调用方应对第一次请求采用下文说明的重试和结果判定规则。
mock fallback)会跳过等待,guest 操作返回 HTTP 409。

Guest 操作和 lifecycle 变更使用同一个 sandbox operation lock。取得锁后,
manager 会再次检查 `Running`,避免并发 lifecycle 变更后请求仍访问旧 runtime。
Expand Down Expand Up @@ -134,6 +130,54 @@ read 响应过大时返回 HTTP 502 和
保持 `listen.http_addr` 关闭。Daemon 停止时也不会等待全部 HTTP handler 或
释放所有 runtime owner,因此正在执行的请求可能看到连接关闭。

## 重置与可复用实例管理

在 Blaze 能够同时重置运行环境和存储之前,
`POST /v1/instances/{id}/reset` 不会返回成功。实例编号格式错误时返回
HTTP 400,实例不存在时返回 HTTP 404,实例不处于运行状态时返回 HTTP 422,
运行中的实例返回 HTTP 501,且不会改变其状态或已占用资源。

四个 `/v1/pools` 管理接口同样返回 HTTP 501。`storage.pool_size` 和
`storage.prefork` 始终会被拒绝;除历史软件包的精确默认值外,任何 `[pool]`
配置段也会失败。软件包升级时,只会临时接受并忽略旧版守护进程配置和两份默认策略
原样附带的 `[pool]` 默认值,同时记录警告。这项例外用于避免 RPM 通过
`%config(noreplace)` 保留的管理员自定义文件阻止新版服务启动,并不会启用
可复用实例。管理员应合并每个 `.rpmnew` 文件,或删除旧配置段;后续版本可能
取消这项兼容。其他策略 `[pool]` 配置会导致策略加载失败。启动时,
`policy.on_load_error = "fail"` 会让守护进程停止,`"warn"` 则会使用空策略集
继续启动。通过管理接口或信号重新加载策略失败时,当前生效的策略保持不变。

可以接受的 daemon `[pool]` 配置段必须恰好包含以下两个键值:

```toml
[pool]
default_warm_ttl = "30m"
gc_interval = "5m"
```

可以接受的策略配置必须恰好包含六个字段,并且属于以下两个软件包内置策略之一:

| 策略名称 | 工作负载类型 | `min` | `target` | `max` |
|---|---|---:|---:|---:|
| `agent-rl-default` | `agent-rl` | 4 | 16 | 64 |
| `agent-tool-default` | `agent-tool` | 2 | 8 | 32 |

两行都要求 `enabled = true`、`warm_ttl = "30m"` 和
`reset_mode = "full-recreate"`。缺少或增加字段、改变值或类型、策略名称或工作
负载类型不同,或者出现任何其他 `[pool]` 配置,都会被拒绝。接受的兼容值会被
忽略,序列化配置时也会省略。

Blaze 仍可读取旧版本写入的 `Reset`、`Warm` 和 `start_path = "warm"` 持久化
值。启动恢复会把包含这些值的未终止记录作为清理对象,且不会复用这些记录。
清理失败时,内存记录会保留为 `RecoveryRequired`,并尝试持久化该状态。如果
持久化也失败,启动警告会记录附加错误,磁盘上的记录可能仍是先前状态。其他已通过
校验的记录仍会继续恢复。监控接口不再输出 `blaze_instances_resets_total`、
`blaze_pool_hits_total` 和 `blaze_pool_misses_total`。

这些兼容响应背后的生命周期约束记录在
[生命周期状态一致性与兼容性设计](../../../../src/blaze/docs/design/lifecycle-state-consistency_zh.md)
中。

## 存储制品同步

Blaze 可以定期持久化 running sandbox 中已经写入的宿主机制品和目录元数据。
Expand Down
8 changes: 4 additions & 4 deletions src/blaze/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ blaze is a **daemon-only** per-host sandbox orchestrator. All sandbox management

Two-crate workspace:

- **blaze-core** (library): policy engine, lifecycle state machine, backend selector, pool manager, kernel hook registry, config schema. Zero I/O beyond local TOML/JSON parsing.
- **blaze-core** (library): policy engine, lifecycle state machine, backend selector, kernel hook registry, config schema. Zero I/O beyond local TOML/JSON parsing.
- **blazed** (binary): daemon HTTP server (UDS + TCP), spawner implementations, metrics endpoint, CLI for daemon lifecycle commands.

Dependency direction: `blazed` → `blaze-core`. No reverse dependency.
Expand All @@ -26,12 +26,12 @@ Platform: Linux (x86_64 + aarch64) for production. macOS builds succeed but spaw

## Key Design Constraints

- **Daemon-only API model**: No CLI client for sandbox operations. All instance/pool/template management is done via HTTP endpoints on UDS (`/run/blaze/api.sock`) or TCP (`:14159`). The CLI subcommands (`daemon start`, `daemon reload`, `daemon doctor`) only manage daemon lifecycle.
- **Daemon-only API model**: No CLI client for sandbox operations. All instance and template management is done via HTTP endpoints on UDS (`/run/blaze/api.sock`) or TCP (`:14159`). The CLI subcommands (`daemon start`, `daemon reload`, `daemon doctor`) only manage daemon lifecycle.
- **BackendSpawner trait**: All backend-specific process management is behind `BackendSpawner`. Adding a new backend means implementing `spawn()`, `wait()`, `kill()`, `probe()` and registering it in `daemon::build_spawner()`.
- **Policy-driven backend selection**: Workload class → policy file → prioritized backend list. The daemon probes backends at startup and selects the first available. Never hardcode backend preference in application logic.
- **Lifecycle state machine**: 9 states. The main branches are Pending →
Creating → Running, Running ↔ Paused → Checkpointed, and Running → Reset →
Warm → Creating. Any non-terminal state can enter Destroyed; incomplete
Creating → Running and Running ↔ Paused → Checkpointed. Any non-terminal
state can enter Destroyed; incomplete
cleanup enters RecoveryRequired. State transitions are enforced by
`blaze_core::lifecycle`. Do not bypass via direct field mutation.
- **MockSpawner fallback**: When the configured backend binary is missing or fails `probe()`, the daemon auto-downgrades to `MockSpawner` with a warning. This keeps API/integration tests functional without a real backend.
Expand Down
92 changes: 71 additions & 21 deletions src/blaze/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,22 +5,20 @@
Per-host sandbox orchestrator daemon for AI Agent workloads.

Blaze manages sandbox instance lifecycles via HTTP API with policy-driven
backend selection. It supports warm-pool pre-allocation, multi-backend
fallback (Firecracker → Bubblewrap → Mock), and Prometheus metrics export.
backend selection. It supports multi-backend fallback
(Firecracker → Bubblewrap → Mock) and Prometheus metrics export.
Designed as the per-host agent for E2B-style orchestrator platforms.

## Features

- **HTTP API** — Unix domain socket (`/run/blaze/api.sock`) + TCP (`:14159`)
- **Policy-driven backend selection** — workload class → backend priority list
- **Lifecycle state machine** — 9 states: Pending, Creating, Running, Paused,
Checkpointed, RecoveryRequired, Reset, Warm, and Destroyed
- **Lifecycle state machine** — durable state with restart recovery
- **Guest operations** — bounded command execution and file transfer for
running backends that expose a guest endpoint
- **Warm pool management** — pre-warmed instances with TTL-based GC
- **Template catalog** — bounded import and atomic publication of reusable artifacts
- **Kernel hook registry** — state tracking for pre/post hooks
- **Prometheus metrics** — request counts, instance gauges, pool sizes
- **Prometheus metrics** — request and instance counters
- **Spawners** — FirecrackerSpawner, BubblewrapSpawner, MockSpawner
- **Optional VM networking** — isolated namespace, tap, veth, and NAT per Firecracker VM

Expand Down Expand Up @@ -108,12 +106,23 @@ provider = "file" # Storage provider selection. Currently supported: "file
# "auto" probes available providers in priority order (currently equivalent to "file").
# Other values will log a warning and fall back to file.
images_dir = "/var/lib/blaze/images"
# pool_size = 0 # [Reserved] Warm pool slots (not yet active)
# prefork = false # [Reserved] Pre-start VMs in pool (not yet active)
sync_interval = "disabled" # Set a positive duration to persist already-written slot artifacts.
sync_timeout = "30s" # Maximum scheduler wait for reconstruction plus artifact sync.
```

Reusable-instance settings are not supported. Blaze rejects
`storage.pool_size`, `storage.prefork`, and every `[pool]` section except the
exact historical package defaults. Blaze temporarily accepts and ignores those
defaults from older `config.toml`,
`agent-rl.toml`, and `agent-tool.toml` files, and logs a warning. This lets an
administrator-modified file retained by RPM `%config(noreplace)` reach the new
daemon without enabling an incomplete feature. Merge the corresponding
`.rpmnew` file or remove the old `[pool]` section; later releases may remove
this exception. Any other policy `[pool]` section fails policy loading. At
startup, `policy.on_load_error = "fail"` stops the daemon, while `"warn"` starts
with an empty policy set. A failed administrative or signal-driven reload
keeps the currently active policies unchanged.

The `file` provider uses standard filesystem operations for sandbox storage. The `auto` provider probes available backends in priority order (currently equivalent to `file`). Unrecognized values will log a warning and fall back to `file`.
When periodic synchronization is enabled, a completed provider failure is
isolated from later sandboxes. If a provider cannot stop its filesystem work at
Expand Down Expand Up @@ -146,11 +155,11 @@ for configuration, selection, retry, and worker shutdown behavior.
| POST | `/v1/instances/{id}/read` | Compatible guest file read action |
| POST | `/v1/instances/{id}/write` | Compatible guest file write action |
| POST | `/v1/instances/{id}/checkpoint` | Record checkpoint state |
| POST | `/v1/instances/{id}/reset` | Record reset and return to the warm pool |
| GET | `/v1/pools` | List warm pools |
| GET | `/v1/pools/{backend}/{class}` | Get pool status |
| POST | `/v1/pools/{backend}/{class}/drain` | Drain a pool |
| PUT | `/v1/pools/{backend}/{class}/sizing` | Resize a pool |
| POST | `/v1/instances/{id}/reset` | Reserved for running instances; returns `501` until runtime and storage reset are implemented |
Comment thread
WeissonHan marked this conversation as resolved.
| GET | `/v1/pools` | Reserved; returns `501` |
| GET | `/v1/pools/{backend}/{class}` | Reserved; returns `501` |
| POST | `/v1/pools/{backend}/{class}/drain` | Reserved; returns `501` |
| PUT | `/v1/pools/{backend}/{class}/sizing` | Reserved; returns `501` |
| GET | `/v1/templates` | List published template names |
| GET | `/v1/templates/{name}` | Inspect published template metadata |
| POST | `/v1/templates/import` | Publish a template from the configured import root |
Expand All @@ -159,6 +168,45 @@ for configuration, selection, retry, and worker shutdown behavior.
| GET | `/v1/metrics` | Prometheus metrics |
| POST | `/v1/admin/reload` | Hot-reload policies |

For reset requests, a malformed instance identifier returns `400`, an unknown
instance returns `404`, and an instance that is not running returns `422`. A
running instance returns `501` without changing its lifecycle state or its
runtime and storage resources. Clients that require a fresh sandbox must
successfully destroy the old sandbox and create a new one; `501` does not mean
that reset completed.

Upgrade compatibility accepts and ignores only this exact daemon section:

```toml
[pool]
default_warm_ttl = "30m"
gc_interval = "5m"
```

An accepted policy section must contain exactly these six fields and belong to
one of the two packaged policy identities:

| Policy name | Workload class | `min` | `target` | `max` |
|---|---|---:|---:|---:|
| `agent-rl-default` | `agent-rl` | 4 | 16 | 64 |
| `agent-tool-default` | `agent-tool` | 2 | 8 | 32 |

Both rows require `enabled = true`, `warm_ttl = "30m"`, and
`reset_mode = "full-recreate"`. A missing or additional field, a changed value
or type, a different policy name or workload class, any other `[pool]` section,
and every `storage.pool_size` or `storage.prefork` setting are rejected. The
accepted values do not enable reusable instances and are omitted when the
configuration is serialized.

Blaze continues to decode persisted `Reset`, `Warm`, and
`start_path = "warm"` values written by earlier releases. Startup
reconciliation treats non-terminal records containing those values as cleanup
candidates and never reuses them. A failed cleanup retains the in-memory record
as `RecoveryRequired` and attempts to persist that state. If persistence also
fails, the startup warning includes the additional error and the durable record
may still contain its previous state. Reconciliation continues with other
accepted records.

The `/v1/templates` routes are the single operator-facing template catalog.
Importing an entry does not yet make sandbox creation select it; future create
support will resolve optional names from this same catalog. See the
Expand Down Expand Up @@ -196,15 +244,17 @@ map lock until publication. Direct file changes by a process that bypasses the
state-root lock are unsupported.

See the
[lifecycle state consistency design](docs/design/lifecycle-state-consistency.md)
for the writer-coordination, inventory-publication, and failure boundaries.
[lifecycle state consistency and compatibility design](docs/design/lifecycle-state-consistency.md)
for writer coordination, inventory publication, reset rejection, legacy-state
cleanup, and failure boundaries.

The operation journal records the operation and start time, not completion of
each resource step. An interrupted create is cleaned up rather than resumed,
and an existing backend process is not adopted after restart. Failed recovery
does not run in a background retry loop. The checkpoint and reset endpoints
retain their existing metadata transitions; this recovery flow does not add
backend snapshot or restore operations.
does not run in a background retry loop. The checkpoint endpoint retains its
existing metadata transition. Reset remains unavailable until runtime and
storage can be reset together; this recovery flow does not add backend snapshot
or restore operations.

### Guest operations

Expand All @@ -217,13 +267,13 @@ boundaries.

#### Health Check

`GET /v1/health` returns daemon status including storage pool readiness:
`GET /v1/health` returns daemon status including storage capacity:

```json
{
"status": "ok",
"version": "0.3.0",
"storage_pool": { "ready": 0, "capacity": 0, "pending": 0 }
"storage_pool": { "ready": 0, "capacity": 0, "pending": 0, "quarantined": 0 }
}
```

Expand All @@ -232,7 +282,7 @@ boundaries.
```
src/blaze/
├── crates/
│ ├── blaze-core/ # Library: policy, lifecycle, pool, template, kernel, config
│ ├── blaze-core/ # Library: policy, lifecycle, template, kernel, config
│ └── blazed/ # Binary: daemon, API server, spawners, metrics
├── examples/ # config.toml, policies/
├── dist/ # blazed.service, blaze.spec, tmpfiles
Expand Down
Loading
Loading