Optimizing a served cluster requires stopping the server, and each direct run keeps its lock

Problem

On a cluster with a running server, optimize (and cleanup, repair, rebuild-full-text-indexes) can only run after the server is stopped. Each completed run then keeps the cluster lock, so the next graph cannot start until an operator force-unlocks it. Compacting every graph of a served cluster therefore needs an outage plus one exact-ID unlock per graph.

The path at 0a62bb7:

The direct commands take the cluster lock. Every direct maintenance command calls admission::ensure_graph before it opens the graph (maintenance/mod.rs#L70, and L120, L145, L218). For a cluster graph, that call acquires the cluster's exclusive admission (admission.rs#L63). Since Build durable offline deployments and exact recovery #849 (0.12), that is the same lock the server holds for its whole lifetime, so the command refuses with state_lock_held while the server runs.
The server cannot run it either. The server has no maintenance route. The maintenance docs say these commands "do not run through the HTTP server" (maintenance.md#L11). server: coordinated optimize and cleanup operations with resumable status #646 tracks a served route.
A completed command keeps its lock. A command that succeeds deliberately keeps its own lock: "command completion alone does not release this lock" (admission.rs#L71). The next direct command, for example the next graph's optimize, refuses until cluster force-unlock <id>.
The exclusion comes from admission, not from optimize. The engine's optimize is safe beside live writers. A write that lands on a table while its compaction is staged fails the run with a read-set conflict (maintenance.md#L51). The same page does not say that on a served cluster this guarantee is unreachable, because the command cannot start beside the server.

Evidence

A local cluster with two graphs on 0.14.0-edge+0a62bb7, starting after cluster apply, which keeps its own lock:

optimize --cluster R --graph a     → [state_lock_held] … lock id 01M58B44SH… (the apply's lock)
cluster force-unlock 01M58B44SH… --cluster R
optimize --cluster R --graph a     → ok; stderr: cluster admission retained: … lock_id=01M58B4AF2…
cluster status --cluster R         → lock_id 01M58B4AF2…
optimize --cluster R --graph b     → [state_lock_held] … lock id 01M58B4AF2…

A production deployment with six graphs on an S3-compatible bucket now runs compaction as a scheduled outage. It stops the server, optimizes each graph, force-unlocks each run's lock by the ID that run reported, redeploys, and waits for readiness. Before 0.12, the same workflow ran every two days beside the server. That deployment's read latency is dominated by small-fragment counts, so it cannot simply skip compaction.

Directions

Run maintenance in the owning server (server: coordinated optimize and cleanup operations with resumable status #646). The server already admits writes, merges and schema apply under its own admission. Optimize and cleanup can be operations of the same process:
- authorized by a policy action;
- with an operation ID and status that survive a lost response, as served deployments do.
Optimize publishes once and fails when a write lands on a staged table, so a served run needs a bounded retry or a per-table batch, or a busy table starves it. Feature request: table scoping for optimize (--table / --exclude-table) #432 asks for table selection.
Release a direct command's lock once its I/O has settled. The server already does this on clean shutdown (lib.rs#L3050-L3075): it closes the admission's storage I/O scope, waits for it to go idle, refuses if any outcome is uncertain, then calls release_after_settlement. A direct maintenance command could:
- run its engine under the admission's I/O scope;
- apply the same proof when it finishes;
- release its own lock.
A command with an uncertain outcome keeps the lock and says so, as now.
Document the boundary until (1) lands. On a cluster-served graph, a direct maintenance command requires the serving owner to be stopped, and each run's lock must be released before the next.

Acceptance

Optimizing every graph of a served cluster needs no server restart and no force-unlock.
Until then, a direct maintenance command that completes with settled storage I/O leaves no lock behind, and one with an uncertain outcome keeps it.
A CLI integration test runs two direct optimize commands in a row against a stopped cluster with no unlock between them.

Summary

On a served cluster, optimize, cleanup, repair, and rebuild-full-text-indexes cannot start while the server is up. Each direct command takes the one cluster lock. A command that finishes keeps that lock. The next graph then stops until an operator runs cluster force-unlock with the exact lock id. The engine operation is safe to plan beside writers. The exclusion comes from admission.

Status of this issue

The issue is still open. The two comments are the reporter's production measurements from 2026-10-18. No pull request implements a fix. This note is a reproduction on current main, plus a design assessment.

Reproduction

Tree: b692610c4aa31066cb6a9fd34830f56977288943 (2026-10-20). The report used 0a62bb7.

Binary: omnigraph 0.14.0, internal schema 16, HTTP contract 0.13, Lance 13.0.0. cargo build -p omnigraph-cli -p omnigraph-server --locked --offline finished with no rebuild, so the binary matches this tree.

Host: Linux, local file:// storage, no cloud credentials. Two empty graphs, a and b, each with one Person node type. Root: /tmp/og1082/data.

Claim 1 — reproduced

The four direct commands call ensure_graph_with_binding before Omnigraph::open (maintenance/mod.rs, also L80, L96, L162). For a cluster graph that acquires ClusterAdmissionPurpose::GraphOperation (admission.rs). The server acquires the same __cluster/lock.json with purpose Serve and holds it for the process lifetime (lib.rs).

Server command:

omnigraph-server --cluster /tmp/og1082/data --bind 127.0.0.1:18082 --unauthenticated

The lock file was 01M5DFJ1FT6YGE7R5KWWRR71FG, operation: serve. Each command below exited 1 with state_lock_held for that id. The lock file did not change.

omnigraph optimize --target /tmp/og1082/data --graph a --json
omnigraph repair --target /tmp/og1082/data --graph a --json
omnigraph cleanup --target /tmp/og1082/data --graph a --keep 1 --confirm --yes --json
omnigraph rebuild-full-text-indexes --target /tmp/og1082/data --graph a --branch main --json
omnigraph optimize --target /tmp/og1082/data/graphs/a.omni --json

Error text:

[state_lock_held] __cluster/lock.json: cluster state lock already exists (lock id 01M5DFJ1FT6YGE7R5KWWRR71FG); run `omnigraph cluster force-unlock 01M5DFJ1FT6YGE7R5KWWRR71FG` only after confirming no cluster operation is active

The graph path does not bypass the lock.

Claim 2 — reproduced

GET http://127.0.0.1:18082/openapi.json returned 200 and 34 paths. No path contains optimize, cleanup, repair, or maintenance. These returned 404 with an empty body:

POST /graphs/a/optimize
POST /graphs/a/cleanup
POST /graphs/a/maintenance
POST /graphs/a/repair

PolicyAction has no optimize or cleanup variant (Read, Export, Change, SchemaApply, branch and commit actions, Admin, GraphList, InvokeQuery, ConfigManage).

omnigraph optimize --target http://127.0.0.1:18082 --graph a --json

exited 2:

target_incompatible: optimize is not available for this target kind and scope

The maintenance guide still says these commands do not run through the HTTP server (maintenance.md). That page does not say that on a served cluster the command cannot start.

Claim 3 — reproduced

cluster apply exited 0 and kept lock 01M5DFHFRB5SQKNN56ZZMD1R43 (operation: deployment) after the process exited.

optimize --graph a exited 1 with state_lock_held for that id.

omnigraph --target file:///tmp/og1082/data cluster force-unlock 01M5DFHFRB5SQKNN56ZZMD1R43 --json

returned {"unlocked": true, "lock_id": "01M5DFHFRB5SQKNN56ZZMD1R43"} and removed lock.json. force-unlock requires an explicit --target. The state_lock_held text omits --target.

optimize --graph a then exited 0. The graph had nothing to compact (committed: false on node:Person and __manifest). Stderr still said:

cluster admission retained: root=file:///tmp/og1082/data lock_id=01M5DFHQNZCDHT5EFF65BR690C. After stopping the owner and establishing graph/control I/O quiescence, use exact-ID cluster force-unlock; command completion alone does not release this lock.

cluster status --target /tmp/og1082/data --json reported that same lock_id. optimize, repair, cleanup --keep 1 --confirm --yes, and rebuild-full-text-indexes for graph b each exited 1 with state_lock_held for 01M5DFHQNZCDHT5EFF65BR690C.

A CLI test on this tree fails the same way: cli_cluster::second_graph_optimize_after_completed_optimize_needs_no_force_unlock_issue_1082. The fixture unlocks the apply lock, optimizes knowledge, then optimizes archive with no force-unlock. The second command panics in output_success with state_lock_held for the first command's lock. The test is not in the tree. This assessment does not change the product.

Claim 4 — reproduced at the admission boundary

The refusal is printed by admission, before the engine opens the graph. A standalone graph does not take a cluster lock:

omnigraph init --schema s.pg /tmp/og1082/standalone/g.omni
omnigraph optimize --target /tmp/og1082/standalone/g.omni --json

That optimize exited 0. Stderr has only the target line. There is no lock.json.

The engine rule is unchanged. Optimize plans from the pinned table versions and publishes those pins in one manifest commit. A pin that another writer moved fails the run as a read-set conflict. The write-path guide states that (writes.md). optimize_serializes_concurrent_delete_across_handles in crates/omnigraph/tests/failpoints.rs covers two handles in one process: the second writer waits on the branch gate, then a strict delete prepared against the old head reports the typed read-set conflict. I did not remove the lock and run a second process beside optimize. Admission makes that start impossible.

What differs from the report at `0a62bb7`

Lance is 13.0.0, not 11.0.0 (build(storage): upgrade lance to 13.0.0 #1072). The lock sequence is the same.
feat(cli)!: replace legacy routing with explicit targets #1090 replaced --cluster with --target. optimize --target http://… exits 2 with target_incompatible. The older sentence was that --server does not apply.
ensure_graph is now ensure_graph_with_binding. The retain sentence is the same.
The live OpenAPI document has 34 paths. The server: coordinated optimize and cleanup operations with resumable status #646 note at 0a62bb7 counted 33. The added surface is access, not maintenance.
A clean server stop releases the lock. SIGTERM removed lock.json within about 2 seconds. SIGKILL left the serve lock in place (01M5DFMNMXFK7MPCDR3YETV767, dead pid). A direct command has no equivalent release. Drop for ClusterAdmission does not call release_after_settlement. The CLI sources have no storage I/O scope.
A no-work optimize still retains the lock. That matches the production note that a graph with nothing to compact still needs an unlock.

The six-graph S3 times in the comments were not measured again.

Root cause

__cluster/lock.json is the writer fence for the whole cluster root. The server takes it before it serves, and releases it only after a clean shutdown closes the storage I/O scope, waits until that scope is idle, refuses when an outcome is uncertain, and then calls release_after_settlement (lib.rs).

A direct maintenance command takes the same object and leaves the guard to Drop. The CLI states why: success of an arbitrary command is not a settlement proof (admission.rs). The control-plane guide says the same thing: a return from writable work, including success, retains the lock (control-plane.md).

The in-process write queue is what serializes optimize with other writers in that process. A second process does not share it. The publication CAS is what rejects a stale pin. Cleanup is stricter: it deletes files, and the invariants keep it inside one writer process. #1094 records the read-side cost of that deletion order. A second process that only shares the cluster lock does not hold those gates.

Design

Run the four operations in the server process that already holds the lock, the write queue, and policy. That is #646. Keep one writer and one settlement proof.

What we give up is a second process that compacts while the server runs. That second process cannot share the write queue. The need in this issue is an available server, not a second writer. The long single-graph optimize (the personal graph, about 577 seconds with nothing to compact) remains a bound on that graph's writers. It stops being downtime for the other graphs.

1. Maintenance in the owning server (#646)

This is the change that removes the outage.

Add a policy action. Enforce it at the HTTP boundary and at the engine entry, so a direct or embedded caller does not bypass it.
Return an operation id that survives a lost response. server: coordinated optimize and cleanup operations with resumable status #646 already requires this.
Other graphs stay up. Their requests do not take this graph's branch gate.
Writers on the graph under maintenance wait on that gate for the run. Readers keep the snapshot from before the one publication.
A moved pin fails the run with the existing read-set conflict. The batch publishes once, or not at all.
Cleanup stays in this process, because of Cleanup leaves a table version's manifest readable after deleting its files #1094. The gates are what keep a writer off a pin that cleanup is deleting.
Bound one run. Feature request: table scoping for optimize (--table / --exclude-table) #432 is table selection. A single call that holds the branch gate for many minutes stalls writers on that graph. A per-table batch can publish on its own and record which tables finished. Say that in the operation status. One publication per batch keeps the manifest rule. It is not one publication for a whole multi-hour pass unless the pass is one batch.
performance: a cold commit list reads every merged branch's history block, one at a time #1068 asks optimize to fold the small history extents that merged review branches leave behind. That is more work inside this same operation. It does not need a second lock. On the direct path, that fold makes the outage longer.

2. Release a direct command after settlement

Do this as well, for the outage window that remains until (1) is in use. It does not let the command start beside a live server.

Copy the server shutdown sequence: run the engine under the admission's storage I/O scope, close the scope, wait until it is idle, keep the lock when any outcome is uncertain, then release_after_settlement. An uncertain command says so and keeps the lock.

The CLI does not enter that scope today. Deleting lock.json on exit 0 is the wrong patch. A killed process, a lost acknowledgement, and a late PUT are why the retain rule exists. SIGKILL of the server left the lock. That is the same Drop behavior.

#1096, which implements part of the #1093 RFC, scopes an unknown storage outcome to its key. After that lands, the release proof uses that scope. On this main, uncertainty is still process-wide.

3. A per-graph lock is the wrong fix

The one root lock matches one server that owns the root. A per-graph lock would let graph b start while graph a is locked, and it would let a second process write. #1017 (accepted 2026-10-20 as a document) does not lift the single-writer boundary. #1097 and #1093 are still changing this one lock into a lease. A second fence beside that work adds stuck records and a new takeover rule. It does not give the second process the write queue, and it does not make cleanup safe.

Open work that touches the same lock

feat(cluster)!: take over a silent server's S3 lock through a lease #1097 (open): on local and S3 roots the lock becomes a lease, so a server can take over a silent owner without force-unlock. A CLI writer or a deployment still refuses a held lock at once. A dead server can be replaced. The next graph's optimize still cannot. This PR does not remove the maintenance outage.
rfc: storage outcomes per operation and restartable ownership #1093 (open RFC) and fix(server): scope an unknown storage outcome to its key and branch #1096 (open, stacked on it): a storage outcome belongs to the operation that sent it, and a failed server can restart. That is the settlement model for release-after-proof. Neither adds a maintenance route.
rfc: graph access and cross-process writes #1017 (open): one cluster-backed server. Cross-process engine writes stay out of scope.
Cleanup leaves a table version's manifest readable after deleting its files #1094 and performance: cleanup opens every table version before deleting any, so a run cut short reclaims nothing #1095: cleanup can leave a manifest whose files are already gone, and cleanup's time tracks how many versions it opens. performance: cleanup opens every table version before deleting any, so a run cut short reclaims nothing #1095 stops the server because of this issue. A faster cleanup still cannot start while the server holds the lock. A lock release does not make a half-swept version safe for a live reader.
performance: a cold commit list reads every merged branch's history block, one at a time #1068: fold history extents inside optimize. Same owner, same operation.
Feature request: table scoping for optimize (--table / --exclude-table) #432: choose tables, so one served run has a bound.
Build durable offline deployments and exact recovery #849 (merged): the server holds the lock for its lifetime. That is why the pre-0.12 workflow, optimize beside the server, stopped.

Liability

Served maintenance adds one operation to an authority that already exists. The new obligations are the policy action, the operation record, and a bound on how long one graph's writers wait. Those are visible and testable.

Release-after-settlement removes the operator step that types an exact lock id between two successful graphs. The hazard is a release before a late write is durable. The server path already refuses that case. Use that path.

A per-graph lock adds a fence that the #1097 lease does not know. Two owners, two stuck records, and cleanup in a process that does not hold the writer's gates. That is more liability than the outage it removes.

force-unlock remains the operator tool for a lock whose owner is dead and whose I/O is not proved. It is not the steady step between two successful optimizes.

Lance

Lance does not cause this, and a Lance release does not fix it.

The pin is unmodified Lance 13.0.0. At 0a62bb7 it was 11.0.0. #1072 is the bump. No open pull request in this repository upgrades Lance. On lancedb/lance, the latest stable release is v13.0.0 (2026-10-07). v14.0.0-beta.21 (2026-10-19) is a pre-release.

The cluster lock is OmniGraph's __cluster/lock.json. Lance has no equivalent object.

Graph-table optimize writes detached commits. Those commits do not move HEAD and do not take Lance's conflict pass (lance.md, Detached commits). The read-set conflict in the maintenance guide is the manifest pin CAS.

Lance's transaction page describes optimistic concurrency for linear commits. A Rewrite, which includes compaction, conflicts with overlapping fragments and can rebase when the operations are compatible: Transactions. The read/write guide treats compaction as a file rewrite, not as a cluster lock: Read and write. Open Lance work on compact-versus-update and compact-versus-reindex rebase is lance#3775 and lance#5143. That work applies to Lance's linear conflict resolver. OmniGraph's detached optimize path does not use it. Shipping that resolver would not let a direct command start beside the server, and it would not release the lock after the command.

This credential cannot search lancedb/lance issues (the search API returned 422). The release list, the transaction page, and the public record of those two issues are the Lance evidence here.

Test plan

Extend crates/omnigraph-cli/tests/integration/cli_cluster.rs. applied_two_graph_cluster already unlocks the apply lock. Run optimize on knowledge, then optimize on archive, with no force-unlock. Both exit 0. Repeat for repair, cleanup --keep 1 --confirm --yes, and rebuild-full-text-indexes. The test named above fails on this main and is the regression for the second command.
Start omnigraph-server on that root. The four direct commands exit 1 with state_lock_held for the serve lock. SIGTERM removes lock.json, and the next optimize exits 0. SIGKILL leaves lock.json. v2_root_admission_blocks_cli_write_and_native_control_doors already covers an in-process Serve admission. It does not cover process exit.
When server: coordinated optimize and cleanup operations with resumable status #646 has a route: an authorized caller optimizes one graph while another graph still answers a read. A lost response looks up the operation id. A writer on the optimized graph waits, or gets the existing read-set conflict, and a retry sees the new pins. Cleanup does not run in a second process. A .gqt case cannot see the lock. Keep the tests in the CLI and server owners.

This issue stays open

#646 is the served-operation issue, and it is still open. Its acceptance does not include "a finished direct command releases a settled lock". #1095, #1021, and #1068 reference this issue. None of them has the same scope. This issue stays open.

Limitations

Local files only. The six-graph S3 times were not measured again. No second process wrote beside optimize with the lock removed. The failing CLI test was run on this checkout and then removed from the working tree.

v0.14.0 turns this from a downtime cost into a hard gap for S3 clusters.

Operator configuration version 1 accepts only local-file direct targets (inline access supports local storage only; configure a server or managed target), and docs/user/operations/maintenance.md says cloud-direct maintenance is not exposed. So optimize and cleanup can no longer address an S3 cluster root from the CLI, and there is still no served route.
The previous workaround (stop the server, run a direct optimize against the root, release that run's lock by ID, restart) is closed too. On S3 a v0.14.0 server's clean release rewrites __cluster/lock.json as a released version-3 record instead of deleting it, and older binaries refuse that record, so a pre-0.14 CLI can't take the lock once a v0.14.0 server has run there.

A self-hosted S3 deployment that upgrades to v0.14.0 therefore has no supported way to compact or reclaim storage, and fragment counts and read latency grow until one exists. A served optimize/cleanup admitted under the serving owner's own lock (as schema apply is) would close it. A narrower interim would be to accept a cloud root as a direct target for the maintenance commands only.

Navigation Menu

Repository navigation

Optimizing a served cluster requires stopping the server, and each direct run keeps its lock #1082

Problem

Evidence

Directions

Acceptance

Summary

Status of this issue

Reproduction

Claim 1 — reproduced

Claim 2 — reproduced

Claim 3 — reproduced

Claim 4 — reproduced at the admission boundary

What differs from the report at `0a62bb7`

Root cause

Design

1. Maintenance in the owning server (#646)

2. Release a direct command after settlement

3. A per-graph lock is the wrong fix

Open work that touches the same lock

Liability

Lance

Test plan

This issue stays open

Limitations

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Optimizing a served cluster requires stopping the server, and each direct run keeps its lock #1082

Description

Problem

Evidence

Directions

Acceptance

Activity

ragnorc commented on Oct 20, 2026

Summary

Status of this issue

Reproduction

Claim 1 — reproduced

Claim 2 — reproduced

Claim 3 — reproduced

Claim 4 — reproduced at the admission boundary

What differs from the report at 0a62bb7

Root cause

Design

1. Maintenance in the owning server (#646)

2. Release a direct command after settlement

3. A per-graph lock is the wrong fix

Open work that touches the same lock

Liability

Lance

Test plan

This issue stays open

Limitations

ragnorc commented on Oct 22, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

What differs from the report at `0a62bb7`