[ragnorc]

Phase A, step 2 ("Failure model") of docs/rfcs/2026-10-26-served-graph-maintenance.md (#1168, revision 49d1f345). Stacked on #1175, so until that merges this diff also carries its three commits; the change here is the last two. A served graph cannot run maintenance safely yet for four reasons the RFC records. A storage, I/O, DataFusion or internal failure inside optimize, cleanup, the index writer or the repair report carried no completion evidence, so a server would read it as an unknown outcome and contain the process (F8). Nothing could stop a run short of killing the process (F18). Optimize compacted up to eight tables at once with Lance's defaults, a 32 MiB scan buffer per I/O thread (2 GiB on S3) and full-text builds of about CPUs/2 workers at 2 GiB each (F16). And Lance does the CPU-heavy part of compaction and index builds on whichever tokio runtime polls it, the runtime a server's lease keeper and request handlers share (F18, lease safety).

Every maintenance run now holds its graph's maintenance lane, a new db/maintenance.rs. The lane lives in the root's write-queue manager, so every handle on a root shares it, and admits one run at a time in arrival order; it is process-local derived state and the publication's exact CAS remains the only fence between processes. A run records how far it has gone, and MaintenanceRun::classify gives every failure the matching evidence: an opaque failure before the first durable effect is BeforeEffect (a new OmniError::before_effect_if_opaque, the pre-effect twin of unpublished), after detached commits, staged index files or cleanup deletions it is Unpublished, the publisher's own failure keeps its evidence so a publication in doubt is still the scoped unknown, and a failure after the publication is PartlyPublished. Typed refusals keep their shape. The repair report marks every opaque failure BeforeEffect. Omnigraph::cancel_maintenance cancels the run holding the lane and every run waiting for it, never one that arrives later; the run stops with a new OmniError::MaintenanceCancelled (503 service_unavailable on the wire) at its next cancel point: before every publication, between tables and between the links of a table's detached chain, and at once for work whose effects stay unreachable (planning, a table's chain, an index build), which is dropped where it stands. A publication in progress, and a cleanup deletion in progress, always finish. MaintenanceBudget carries the RFC's knobs with from_env defaults and a handle builder, with_maintenance_budget: unit concurrency 1 (OMNIGRAPH_MAINTENANCE_CONCURRENCY is kept and clamped to 2), CompactionOptions.io_buffer_size 64 MiB plus an explicit 8,192-row batch where no Blob sizing derives one, and InvertedIndexParams 512 MiB over two workers on every full-text build. Each compaction task and index build runs on a two-worker runtime of its own (off_serving_runtime). The lease keeper also logs every failed beat, every failed claim read and a beat unanswered at the deadline, by class and storage kind and before the fence, the step's last item in the revised RFC. The guard is the evidence assertion in detached_commit_matrix.rs, now required of the four maintenance writers as well: an error a maintenance writer returns before its publication is never uncertain.

The finding on Lance work, from the 13.0.0 sources: compaction's decode runs in tasks lance-encoding spawns with tokio::spawn (decoder.rs, "Real decode work happens inside into_batch"), a full-text build tokenizes in a JoinSet of tasks (lance-index scalar/inverted/builder.rs), and a BTREE's training sort runs in the DataFusion stream the caller polls; only page encoding, posting-list merging and BTREE page building use Lance's CPU pool. So yes, the heavy work ran on the caller's tokio workers. Neither remedy the RFC names can hold it: a task spawned from a spawn_blocking thread still lands on the runtime's workers, and Lance's pool accepts only synchronous closures (spawn_cpu) and forbids blocking inside them. Polling the Lance call on a runtime of its own is what moves the spawned tasks. Tests: failpoints.rs gains a cancel after a table's first detached commit (typed error with Unpublished, head and pins unchanged, the staging reclaimed by cleanup once the head moves) and the lane's turns across two handles (the waiting cleanup refused BeforeEffect, the parked optimize stopped BeforeEffect, later runs unaffected); integration/maintenance.rs gains real storage faults, a read-only data/ for optimize and read-only _indices/ for an index build, each Unpublished with nothing published and the next run publishing; in-source tests cover the budget parsing, the evidence by phase, the lane and cancel semantics and the runtime (its thread, abort on drop); the lease test reads the log lines. Locally, on --features failpoints: the whole engine integration binary (642 passed), failpoints (144 passed) and detached_commit_matrix (29 passed), against a clean baseline of the same three on 9b45173c, and the in-source db::maintenance tests.

For review: the decision to challenge is the maintenance runtime, which the RFC did not settle (it names Lance's CPU pool or spawn_blocking, neither of which can hold Lance's own spawned tasks). It is a static two-worker tokio runtime, used only around the Lance calls so OmniGraph's task-local probes stay on the caller, and bypassed on a DST harness thread (one with the logical clock or seeded ids installed) so simulation keeps its single-thread schedule; the cost is that DST never exercises it. Object-store clients are shared across the two runtimes, which works while both live, and both live for the process; the RustFS and Azurite jobs are the first S3 and Azure evidence for that. Other decisions the RFC left open: cleanup's sweep and optimize's planning keep a fan-out of eight, since they are request-bound and narrowing cleanup's sweep would lengthen its global write barrier, so OMNIGRAPH_MAINTENANCE_CONCURRENCY now governs units only; the lane admits waiting runs in arrival order and leaves joining a run of the same operation to the routes step, which can read maintenance_in_progress; the repair report does not take the lane, since it is read-only; removing stale automatic-cleanup configuration from __manifest, a config commit that moves no pin or lineage, is classified as an unpublished effect; and a cancelled optimize skips the adjacency artifact and still returns what it published. The budget table's memory envelope, spill directory, delete rate, pacing, unit time and fold time belong to the unit and index-unit steps and are not here. The RFC puts the budget names in the server guide, which is at its 350-line cap and runs no maintenance yet, so they are documented in the maintenance guide's new Budgets section; the routes step can move them. The direct CLI's default parallelism falls from eight tables to one, and its exit status for a storage or internal maintenance failure falls from 2 to 1; both have notes. Lance's documented LANCE_FTS_PARTITION_SIZE default (256 MiB) disagrees with its source (2048 MiB); the 512 MiB over two workers here matches the documented figure.