bug: an outstanding deployment on an S3 cluster root cannot be reconciled with released tools #1163

Activity

1. github-actions

   added

   needs-triageAwaiting maintainer triageAwaiting maintainer triage

   on Oct 26, 2026
2. ragnorc commented on Oct 26, 2026

   

   ragnorc

   on Oct 26, 2026 – with Cursor

   ContributorAuthor

   More actions

   Summary

   An S3 cluster can be left with one outstanding deployment. The server then refuses to start. No released command can finish that deployment. Each failed start exits, and a restart-on-failure policy repeats the refusal until the platform stops. The graph stays down.

   Reproduction

   Verdict: reproduced on main da1c46368c33876d61615add0ceb6778f9ce2b58 (omnigraph 0.14.1).

   Host: Linux, rustc 1.97.1. The CLI and server binaries were the debug builds of that commit. This run had no Tigris bucket, no Railway policy, and no 10-retry platform loop. The outstanding ledger is a local v2 fixture. The lease takeover uses the in-memory S3 store already in the lock tests.

   The server refuses the outstanding deployment

   Reproduced. Two boots of the same local ledger both return:

   [cluster_deployment_outstanding] __cluster/state.json: only reconciliation of the exact outstanding deployment is admitted; serving and other writers are refused
   

   After each boot the lock file is absent. That string is the body of the error built at crates/omnigraph-server/src/settings.rs:79. The message does not contain the deployment ID.

   omnigraph-server --cluster s3://bucket/prefix on this host exits 1 at that same line. There is no bucket, so the code is state_read_error, not cluster_deployment_outstanding. The outstanding text is from the local fixture.

   The test admission::tests::issue_1163_boot_reconciles_outstanding_deployment expects boot to reconcile and then serve. It fails with the two messages above. It was not committed.

   The existing test outstanding_allows_only_exact_original_reconciliation passes: serve, graph operation, deployment, and a different deployment ID are refused, and the exact ID is admitted only as Reconcile.

   The restart takes the lock, refuses, releases, and exits

   Partly reproduced.

   • Only an S3 serve lock is version 3. only_a_servers_s3_lock_is_leased_issue_1067 passes.
   • A restarted server takes a silent S3 lock after the lease. a_restarted_server_takes_over_a_silent_owners_lock_issue_1067 passes (in-memory store, 2 second lease).
   • The outstanding check runs after that acquire. The refusal path releases the lock. On S3 the release rewrites the record as version 2. It does not delete the key. The next boot replaces that released record. It logs "took over a silent owner's cluster lock" only when the previous process still held version 3.
   • The first start after an uncertain exit (bug: a throttled content-addressed resource write during a served cluster apply stops the server #1162 keeps the lock) does take the silent lock. Later clean refusals do not. Both exit 1. restartPolicy: OnFailure retries exit 1. A config refusal from this CLI exits 2, which is also a failure status.

   No released command can address the S3 root

   Reproduced with omnigraph 0.14.1. ID below is 01K00000000000000000000000:1:01K00000000000000000000001.

   Command	Exit	Result
   --target s3://bucket/prefix cluster apply --deployment-id ID --writers-stopped --json	1	inline access supports local storage only; configure a server or managed target (scope.rs:131)
   same target, cluster force-unlock	1	same text
   same target, cluster status --deployment-id ID	1	same text
   --target s3://bucket/graph.omni optimize and cleanup --keep 1 --confirm	1	same text
   registered kind: direct, scope: root, uri: s3://bucket/prefix	2	targets.production.uri: direct URI must be an absolute local file:// URI without authority
   server target plus --writers-stopped	2	--writers-stopped requires a direct root target
   server target plus --deployment-id and no --config	2	ID-only apply is refused; status is the observation command
   --target file:///tmp/... cluster status --deployment-id ID	1	the target is accepted and storage is read (state_missing on an empty directory)

   scope::tests::issue_1163_s3_direct_root_is_a_recovery_target fails on the same inline refusal. It was not committed.

   Older binaries and the version-3 lock

   Checked in source. Not run as an old binary.

   StorageProvider::S3 and CapturedStorageBinding::capture still build an S3 binding. Version 1 validation never produces one. The server flag --cluster s3://... does open that scheme.

   Before #1097 (888223b7 parent), StateLockFile::parse returns LockVersion(3) for a version-3 record, and that revision's test expects the error. Tag v0.14.0 contains both #1115 and #1097. A binary old enough to accept s3:// on the CLI is too old to parse the lock a v0.14 server leaves.

   optimize, cleanup, and force-unlock

   The S3 address is refused, as the table shows. #1082 is a separate defect: while a server holds the lock, direct maintenance uses that same lock and then keeps it. That remains true for a local root. The 0.14 upgrade note already says old direct s3:// and az:// routes have no inline --target equivalent, and direct-only maintenance has no remote equivalent (docs/user/operations/upgrade-0.14.md).

   Root cause

   Serving an S3 root and recovering one use different doors.

   admit_serving_snapshot accepts an s3:// root (serve.rs). Serve admission is allowed only when outstanding is empty (admission.rs). The outstanding record holds the deployment ID and input_digest. The captured input is the object __cluster/resources/deployment/{digest}.json. reconcile_deployment reads that object and does not replay schema execution. The boot path never calls it.

   The direct-root CLI is the caller that reconciles (commands/cluster/mod.rs), and only with --writers-stopped. #1115 (f7db38da) rejects every non-file:// direct target. That PR says cloud-direct targets are outside the revision. POST /cluster/deployments with the original ID returns the outstanding lookup. It does not call reconcile_deployment. There is no reconcile route. The process is down, so a route on that process is unreachable.

   release_refused_preflight then drops the admission. The next process repeats the refusal and exits 1.

   Recommended fix

   When a new S3 serve admission finds an outstanding deployment, reconcile that exact ID under the admission just taken, then serve if the result is terminal. Keep the lock when reconciliation cannot finish. Log the deployment ID, the root, and the result. Leave the lock held.

   reconcile_deployment_in_store already does this work. The captured bundle is the input. The lease is the fence that --writers-stopped attests for a CLI writer: only an S3 serve lock is leased, and the successor takes it after the previous owner is silent. A request the old process already sent can still land. Keep the current CAS checks (deployment_base_changed, bundle digest). A changed base stays outstanding.

   Land this with the #1162 fix. install_catalog_payloads writes the content-addressed query or policy object with write_text_if_absent. key_class still marks __cluster/resources/** as Mutable, so the put uses the single-attempt client. A SlowDown during boot reconcile can stop the process again and keep the lock. The #1162 reproduction already shows that put: one 503, uncertain=true, no second send. Treat that put-if-absent object as an immutable key, or re-issue it and read it back, in the same change.

   Keep Azure and local roots on the explicit stopped-writer path. Those locks are not leased. An operator attestation is still required there.

   Alternatives

   1. Admit an S3 direct root for cluster status --deployment-id, force-unlock, and cluster apply --deployment-id --writers-stopped only. RFC 0059 specifies inline cloud roots, including recovery from an isolated OMNIGRAPH_HOME. feat(cli)!: adopt owner-scoped operator configuration #1115 left that out. The S3 binding type is still in the tree. This is a qualification gap (region, endpoint, credential chain, conditional writes), not a missing storage stack. It does not stop the restart loop. The server still exits 1 and can take the lock between the operator commands. It is the right later door for force-unlock and for Optimizing a served cluster requires stopping the server, and each direct run keeps its lock #1082. It is not sufficient alone.

   2. Add a served reconcile route. The current apply route returns the outstanding lookup. The route helps only a server that stayed up.

   3. Exit with a status the platform does not retry. This boot exits 1. OnFailure retries every non-zero status. A special status does not finish the deployment. After the retry limit the record is still outstanding, and no command can address the root.

   Liability

   The contract is one outstanding deployment, finished by the process that holds admission, or left outstanding with the lock held. Releasing the lock and exiting hands the retry to the platform and leaves no tool that can finish the work. Boot reconcile puts completion back on the admission owner. The cost is control-plane I/O on the failed boot, and a late write from the fenced process must remain a conflict. A narrow CLI exception is a smaller diff and does not finish the deployment. It also reopens credential and endpoint capture that #1115 deferred.

   Lance

   Pinned Lance is 13.0.0 (crates.io, published 2026-10-07). No stable release is newer. v14.0.0-rc.2 and v15.0.0-beta.1 are pre-releases from 2026-10-22. Their notes are indexes, encodings, and blob projection. They do not add a cluster reconcile door.

   state.json, lock.json, and __cluster/resources/** are OmniGraph control objects. They go through omnigraph-storage and object_store 0.14.2, not through a Lance dataset commit. A Lance upgrade does not admit an s3:// CLI target and does not make serve admission call reconcile_deployment.

   These upstream items can affect a schema apply inside reconcile. They do not fix this issue:

   • Object store guide: S3-compatible stores must honor create-only If-None-Match: *. Tigris has to do that for the lease CAS and for put-if-absent payloads. The guide does not retry a refused boot.
   • lance#10033, open: conditional multipart for large manifests. Dataset commits only.
   • lance#9988, open: experimental transaction idempotency keys. Not this ledger.
   • lance#8773 and lance#8774, open: default commit handler for unrecognized schemes. Dataset commits only.
   • OmniGraph already re-issues an ambiguous Lance manifest put (docs/dev/lance.md, manifest request resolution) because Lance 13 can treat a missing version as final. That fence matters when reconcile runs a schema apply. It does not create the recovery door.

   Related work

   • bug: a throttled content-addressed resource write during a served cluster apply stops the server #1162 is the throttled payload write that leaves the outstanding record. Its reproduction comment points here for the recovery gap. It does not replace this issue.
   • Optimizing a served cluster requires stopping the server, and each direct run keeps its lock #1082 is maintenance while the server holds the lock. The S3 address refusal is extra. The upgrade note already records the missing remote maintenance path.
   • server: a lease fence (Lapsed) logs no heartbeat failure before it stops the server #1164 is a lease fence with no heartbeat warning. A fence can cause the takeover that then hits this refusal. It is a different defect. Its reproduction comment does not cover recovery.
   • feat(cluster)!: take over a silent server's S3 lock through a lease #1097 (merged) is the S3 lease. feat(cli)!: adopt owner-scoped operator configuration #1115 (merged) is the local-only direct target.
   • No open pull request mentions this issue or adds cloud-direct recovery.
   • RFC 0059 still specifies inline cloud roots and the isolated-home recovery command. Shipped version 1 says cloud-direct targets are unsupported (docs/user/cli/targets.md). The cluster recovery section still shows only file:// (docs/user/clusters/index.md).

   Supersession

   No open issue has this defect with a wider scope. #1162 is the write that creates the record. This issue is the stuck cluster after that write. This issue stays open.

   Limitations

   No Tigris bucket, no Railway retry count, and no pre-#1097 binary were run. The outstanding ledger is the local test fixture. The lease takeover is the existing in-memory S3 test. The two failing tests were removed after the run.
3. ragnorc commented on Oct 26, 2026

   

   ragnorc

   on Oct 26, 2026

   ContributorAuthor

   More actions

   Recovered with a CLI built from 7711e4a plus one change in scope.rs. An inline s3:// selection binds StorageProvider::S3 (region and endpoint from AWS_REGION / AWS_ENDPOINT_URL) instead of hitting the "inline access supports local storage only" refusal. Nothing else changed: the S3 binding, admission and reconciliation code are the shipped ones.

   With the server stopped:

   1. cluster status --target s3://… --deployment-id <ID>: outstanding, graphs: {personal: not_started}, lock_id: null. The last refused boot had released the lease lock.
   2. cluster apply --deployment-id <ID> --writers-stopped --target s3://… --as <actor>: status: complete, graph outcome not_attempted, converged: false, result revision unchanged. Admission lock retained, as documented.
   3. cluster force-unlock <that lock> --target s3://…, then the server starts and serves.
   4. Rerunning the same served cluster apply (a corrective successor from the same declaration) converged.

   So the engine path works on S3 today. Only the CLI's target addressing stands between an operator and the documented recovery.
4. ragnorc

   added a commit that references this issue on Oct 29, 2026

   feat(cli): accept an inline s3:// cluster root for deployment, recove…

   ...

   14815c6