feat(blaze): capture Firecracker checkpoints

src/blaze/AGENTS.md

@@ -29,7 +29,13 @@ Platform: Linux (x86_64 + aarch64) for production. macOS builds succeed but spaw
 - **Daemon-only API model**: No CLI client for sandbox operations. All instance/pool/template management is done via HTTP endpoints on UDS (`/run/blaze/api.sock`) or TCP (`:14159`). The CLI subcommands (`daemon start`, `daemon reload`, `daemon doctor`) only manage daemon lifecycle.
 - **BackendSpawner trait**: All backend-specific process management is behind `BackendSpawner`. Adding a new backend means implementing `spawn()`, `wait()`, `kill()`, `probe()` and registering it in `daemon::build_spawner()`.
 - **Policy-driven backend selection**: Workload class → policy file → prioritized backend list. The daemon probes backends at startup and selects the first available. Never hardcode backend preference in application logic.
-- **Lifecycle state machine**: 8 states (Pending → Creating → Running → Paused → Checkpointed → Reset → Warm → Destroyed). State transitions are enforced by `blaze_core::lifecycle`. Do not bypass via direct field mutation.
+- **Lifecycle state machine**: The persisted model contains 9 states. Current
+  managed routes implement create, destroy, checkpoint capture/list, and
+  RecoveryRequired cleanup. Checkpoint capture returns `501` without advancing
+  state when either the backend or storage provider lacks capture support.
+  Reset remains reserved and returns `501` without advancing state.
+  Transitions are enforced by `blaze_core::lifecycle`; do not bypass them
+  through direct field mutation.
 - **MockSpawner fallback**: When the configured backend binary is missing or fails `probe()`, the daemon auto-downgrades to `MockSpawner` with a warning. This keeps API/integration tests functional without a real backend.
 
 ## Adding a New Backend

src/blaze/Cargo.toml

@@ -28,7 +28,7 @@ anyhow = "1.0"
 
 # Async runtime
 tokio = { version = "1", features = ["full"] }
-tokio-util = { version = "0.7", features = ["net"] }
+tokio-util = { version = "0.7", features = ["net", "rt"] }
 hyper = { version = "1", features = ["full"] }
 hyper-util = { version = "0.1", features = ["full"] }
 http-body-util = "0.1"
@@ -42,6 +42,11 @@ tracing-subscriber = { version = "0.3", features = ["env-filter", "json"] }
 uuid = { version = "1", features = ["v4", "serde"] }
 chrono = { version = "0.4", default-features = false, features = ["std", "clock", "serde"] }
 semver = "1.0"
+base64 = "0.22"
+sha2 = "0.10"
+
+# Platform
+libc = "0.2"
 
 # Internal crates
 blaze-core = { path = "crates/blaze-core" }

src/blaze/README.md

@@ -13,7 +13,10 @@ Designed as the per-host agent for E2B-style orchestrator platforms.
 
 - **HTTP API** — Unix domain socket (`/run/blaze/api.sock`) + TCP (`:14159`)
 - **Policy-driven backend selection** — workload class → backend priority list
-- **Lifecycle state machine** — 8 states (Pending → Creating → Running → Paused → Checkpointed → Reset → Warm → Destroyed)
+- **Lifecycle state machine** — 9 states: Pending, Creating, Running, Paused,
+  Checkpointed, RecoveryRequired, Reset, Warm, and Destroyed
+- **Guest operations** — bounded command execution and file transfer for
+  running backends that expose a guest endpoint
 - **Warm pool management** — pre-warmed instances with TTL-based GC
 - **Template registry** — in-memory template tracking with idle eviction
 - **Kernel hook registry** — state tracking for pre/post hooks
@@ -38,11 +41,15 @@ sudo ./target/release/blazed daemon start --config examples/config.toml
 curl --unix-socket /run/blaze/api.sock http://localhost/v1/health
 
 # Create a sandbox
-curl -X POST --unix-socket /run/blaze/api.sock http://localhost/v1/instances \
+curl -X POST --unix-socket /run/blaze/api.sock http://localhost/v1/sandboxes \
   -H 'Content-Type: application/json' \
-  -d '{"workload_class":"agent-rl","image_digest":"sha256:..."}'
+  -d '{"workload_class":"agent-tool","image_digest":"sha256:..."}'
 ```
 
+The quick-start request uses an example policy with Firecracker guest transport
+disabled, so an image without the compatible guest agent does not wait for guest
+readiness. Enable the transport only for images that run that agent.
+
 ## Configuration
 
 The daemon reads a TOML config file (default: `/etc/anolisa/blaze/config.toml`)
@@ -58,6 +65,30 @@ and a policies directory containing per-workload-class policy files.
 
 See `src/blaze/examples/` for annotated sample configurations.
 
+### API Request Limits
+
+The daemon accepts request bodies up to 1 MiB by default. It checks both
+declared `Content-Length` values and streamed body frames, and returns HTTP
+413 when the configured limit is exceeded. Override the limit with a positive
+byte count:
+
+```toml
+[api]
+max_body_bytes = 1048576
+```
+
+Guest files are limited to 16 MiB after base64 decoding. A full-size write is
+larger on the wire because JSON and base64 add overhead, so the default 1 MiB
+request limit intentionally rejects it. Set at least 22 MiB when callers need
+the full decoded limit:
+
+```toml
+[api]
+max_body_bytes = 23068672
+```
+
+The daemon checks both the HTTP request size and the decoded file size.
+
 ### VM Resource Configuration
 
 Blaze resolves vCPU and memory settings using a three-layer fallback chain:
@@ -100,12 +131,26 @@ The `file` provider uses standard filesystem operations for sandbox storage. The
 | Method | Path | Description |
 |--------|------|-------------|
 | GET | `/v1/health` | Health check |
-| GET | `/v1/instances` | List all instances |
-| POST | `/v1/instances` | Create a new sandbox instance |
-| GET | `/v1/instances/{id}` | Get instance details |
-| POST | `/v1/instances/{id}/checkpoint` | Checkpoint an instance |
-| POST | `/v1/instances/{id}/reset` | Reset instance to checkpoint |
-| POST | `/v1/instances/{id}/destroy` | Destroy an instance |
+| GET | `/v1/sandboxes` | List all sandboxes |
+| POST | `/v1/sandboxes` | Create a sandbox |
+| GET | `/v1/sandboxes/{id}` | Get sandbox details |
+| DELETE | `/v1/sandboxes/{id}` | Destroy a sandbox |
+| POST | `/v1/sandboxes/{id}/exec` | Execute a guest command |
+| POST | `/v1/sandboxes/{id}/read` | Read a guest file |
+| POST | `/v1/sandboxes/{id}/write` | Replace a guest file |
+| POST | `/v1/sandboxes/{id}/checkpoint` | Capture a full checkpoint when the backend and storage provider support it |
+| GET | `/v1/sandboxes/{id}/checkpoints` | List committed checkpoints and HEAD reachability |
+| GET | `/v1/instances` | Alias for listing sandboxes |
+| POST | `/v1/instances` | Alias for creating a sandbox |
+| GET | `/v1/instances/{id}` | Alias for sandbox details |
+| DELETE | `/v1/instances/{id}` | Alias for destroying a sandbox |
+| POST | `/v1/instances/{id}/destroy` | Compatible destroy action |
+| POST | `/v1/instances/{id}/exec` | Compatible guest command action |
+| POST | `/v1/instances/{id}/read` | Compatible guest file read action |
+| POST | `/v1/instances/{id}/write` | Compatible guest file write action |
+| POST | `/v1/instances/{id}/checkpoint` | Compatible full-checkpoint action |
+| GET | `/v1/instances/{id}/checkpoints` | Compatible checkpoint-list action |
+| POST | `/v1/instances/{id}/reset` | Reserved; returns `501` until runtime reset is implemented |
 | GET | `/v1/pools` | List warm pools |
 | GET | `/v1/pools/{backend}/{class}` | Get pool status |
 | POST | `/v1/pools/{backend}/{class}/drain` | Drain a pool |
@@ -118,6 +163,99 @@ The `file` provider uses standard filesystem operations for sandbox storage. The
 | GET | `/v1/metrics` | Prometheus metrics |
 | POST | `/v1/admin/reload` | Hot-reload policies |
 
+### Managed lifecycle and recovery
+
+Create and destroy record their operation before changing storage or backend
+resources. A successful create finishes in `Running`; a successful destroy
+finishes in `Destroyed`. If compensation cannot release every owned resource,
+the sandbox remains visible as `RecoveryRequired` so destroy can be retried.
+
+At startup, the daemon reconciles each non-terminal sandbox independently.
+Failure to clean up one sandbox does not prevent the remaining records from
+being processed or the API from starting.
+
+During graceful shutdown, the daemon first stops accepting work and drains
+accepted connections. It then attempts bounded cleanup for every persisted
+record and retained backend owner. One cleanup failure does not skip the
+remaining sandboxes, and all unresolved records are reported.
+
+Create and destroy journals record the operation and start time. Checkpoint
+journals also record the generated checkpoint ID and the latest durable
+boundary the daemon confirmed. Checkpoint listing separately reports which
+catalog entries and HEAD update are actually visible after an interruption.
+An interrupted create is cleaned up rather than resumed, and an existing
+backend process is not adopted after restart. Failed recovery does not run in
+a background retry loop.
+
+Checkpoint capture is available only when both the selected backend and the
+configured storage provider report full-capture support. Otherwise the daemon
+returns `501` before creating a journal, pausing the backend, or changing the
+checkpoint catalog. A supported capture:
+
+1. pauses the backend and captures full VM and memory artifacts;
+2. flushes the live storage slot and copies the full root filesystem;
+3. publishes a verified checkpoint and advances HEAD; and
+4. resumes the backend and confirms guest readiness before returning
+   `Running`.
+
+The file storage provider copies the complete root filesystem into each
+checkpoint. This uses more capacity than a shared-base format, but each
+checkpoint remains independent of later changes to the live slot.
+
+Failures detected before calling the catalog publication step resume the
+backend and discard the incomplete stage. If publication or HEAD has an
+uncertain outcome, or the backend cannot resume, the sandbox becomes
+`RecoveryRequired` while runtime ownership and committed checkpoint data
+remain available for explicit cleanup. Listing uses the same per-sandbox
+operation lock as capture, guest operations, and destroy. Destroy removes
+transaction scratch but preserves committed checkpoint history.
+
+Runtime reset remains reserved and returns `501` without changing runtime or
+persisted state.
+
+### Guest operations
+
+Guest operations are available only while a sandbox is `Running` and its
+backend reports a guest endpoint. A cold create that reports such an endpoint
+waits for the guest agent to answer before publishing `Running`. Backends with
+guest support disabled skip that wait, and later guest-operation requests
+return HTTP 409. Warm-pool activation does not currently repeat the guest
+readiness probe.
+
+Guest operations and lifecycle changes use the same per-sandbox operation
+lock. A request may wait for an earlier lifecycle action. After it obtains the
+lock, the manager checks `Running` again; if destroy or another state change
+won the race, the guest request fails without contacting the old runtime.
+
+The endpoints accept JSON:
+
+```json
+{"cmd":"uname -a","cwd":"/","env":{"LANG":"C"},"timeout":10}
+```
+
+```json
+{"path":"/tmp/input","data_b64":"aGVsbG8="}
+```
+
+`read` takes only `path`; successful file reads and command output use standard
+base64 in the response. Exec timeouts must be from 1 through 20 seconds. Guest
+files are limited to 16 MiB after decoding, and response frames are bounded.
+
+An exec or write failure before request delivery is an ordinary transport
+failure. A bounded wait that expires before delivery uses
+`"code": "guest_timeout"`. If delivery began but the daemon cannot determine
+the result, the API returns HTTP 504 with
+`"code": "guest_outcome_unknown"`; callers must reconcile state instead of
+automatically replaying the operation. Reads do not change guest state and
+remain safe for caller-directed retry. An oversized read response returns
+HTTP 502 with `"code": "guest_response_too_large"`. For exec or write after
+delivery starts, an oversized or otherwise untrusted response instead leaves
+the outcome unknown. An oversized caller request returns HTTP 413.
+
+Each request is fully buffered. The limits bound one request, not the sum of
+concurrent requests, so callers should also bound guest-operation concurrency.
+Streaming files, interactive terminals, and session reuse are not supported.
+
 #### Health Check
 
 `GET /v1/health` returns daemon status including storage pool readiness:
@@ -148,4 +286,3 @@ src/blaze/
 - Linux host with root privileges for sandbox backends
 
 ## License
-