deployment.md

Deployment

OmniGraph can keep a cluster on a local filesystem, S3-compatible object storage, or Azure Blob Storage. The server always boots from one cluster root and exposes its ready applied graphs under /graphs/{graph_id}/…; use --require-all-graphs when any blocked graph must fail startup.

Servers use the v0.14 HTTP contract and open graph storage formats v14 to v16. The cluster ledger has a separate version: explicitly converting it to v2 preserves graph data and history. Start with Operating a cluster to create and apply a deployment bundle.

Binary

OMNIGRAPH_SERVER_BEARER_TOKENS_JSON='{"act-service":"secret"}' \
  omnigraph-server \
    --cluster /srv/omnigraph/company-brain \
    --bind 0.0.0.0:8080

For object storage, pass the applied storage root:

AWS_REGION=us-east-1 \
OMNIGRAPH_SERVER_BEARER_TOKENS_JSON='{"act-service":"secret"}' \
  omnigraph-server \
    --cluster s3://company-data/omnigraph/company-brain \
    --bind 0.0.0.0:8080

Use GET /healthz for process health and GET /readyz for readiness: /readyz reports loading (startup is still pending), serving, degraded (some graphs unavailable), blocked (none ready), or draining. It includes the applied config_digest it booted from (booted_serving_digest), the ledger revision, and served_graph_count, ready_graph_count, loading_graph_count and blocked_graph_count. served_graph_count counts the whole registry. Graphs closed for a transition also contribute to blocked_graph_count. Readiness stays 503 while any graph is loading. Once startup finishes, it returns 200 when at least one graph is ready or the applied inventory is empty, and 503 when all graphs are unavailable or shutdown has begun. The listener opens after configuration and authorization validation, before graphs open. Healthy graphs become available as they finish loading, while readiness holds aggregate traffic until loading finishes. A nonempty cluster with no healthy graph exits with a startup error after its opening attempts finish. With --require-all-graphs, all graphs remain unavailable until every graph opens successfully; any failure stops the server. Wait for readiness before sending data requests: the printed listening address and /healthz confirm only that the listener is live.

Readiness is unauthenticated and names no graph. Authorized GET /graphs returns one graphs list, including blocked entries, with each graph's state (loading, ready, blocked, transitioning or stopping), read_available, write_available and action. Availability describes the runtime, not the caller's permissions. Blocked entries include a sanitized failure: invalid_configuration, invalid_policy, invalid_external_blob_policy, open_failed or invalid_stored_queries. Details remain in server logs. apply_correction_or_restart means deploy corrected queries, policies or provider configuration, or restore damaged storage from a verified backup and restart. It never authorizes adopting a different graph identity. wait_for_restart means shutdown. wait_for_startup means the single startup admission is incomplete; it promises no retry time or additional open. wait_for_transition means that graph's admissions are closed while its owner finishes a transition. Wait for that owner, or restart if the transition was abandoned or expired; there is no promised retry time. Other ready graphs can continue serving and transitioning. Server-owned deployments activate schema and stored-query changes through this transition; a proved pre-effect refusal restores the unchanged views. See live deployment. Ready entries use none. There is no separate quarantined list or automatic startup retry.

GET /cluster/deployments/{deployment_id} reports active only when this process has installed the exact deployment and all its graph bindings are ready. Loading, blocked graphs, partial convergence and shutdown keep it false; a restart must verify its bindings again. Successful apply alone is not activation, which never rewrites the durable achieved result or receipt.

Known loading, blocked or transitioning graphs return 503 (graph_unavailable) to callers authorized to read main or list the management inventory; other callers cannot use this response to discover them. Unknown graphs return 404. A 503 does not authorize replaying a write. omnigraph graphs list --target NAME displays graph ID, state and URI for a registered server or managed target. --json preserves the full inventory fields. Minimal discovery remains IDs/names only.

Admitted writes and reads with fully accepted request bodies retain capacity until execution finishes, even after client disconnect. Writes include optional source deletion after a merge. A lost response leaves the write's outcome uncertain; follow failure outcome before resubmitting. MCP cancellation and its 30-second response deadline stop waiting; the admitted read continues and retains its tool slot until completion.

SIGTERM closes operation admission and stops accepting connections. The server waits for entered graph opens, admitted writes, executing reads, read bodies and stream producers for at most --shutdown-grace-seconds (else OMNIGRAPH_SHUTDOWN_GRACE_SECONDS, else 25), then logs unfinished work and exits 2. A thread enforces the deadline even if the runtime stalls. Signal handling starts after configuration loading, before graph opening. Disconnected work remains in the wait; a graph finishing startup during shutdown cannot become available. A panic, or an uncertain outcome the engine cannot scope to one write's keys, closes admission for every graph and starts the same shutdown without renewing its deadline; a storage failure alone never does (storage failures). Once HTTP connections and other known logical owners finish, it exits 2 immediately; the watchdog is the upper bound. Unresolved reservations stay charged until exit. This contains the process; it does not prove native I/O settled or the engine can be reused. A clean native shutdown exits 0. Set the orchestrator's own termination grace longer than this value; a cutoff is crash-equivalent for the work it interrupts, and the next open recovers it as after any crash. On local storage and S3, the server also closes its storage scope, waits for accepted graph and control writes, and releases its own cluster admission. The next server can start normally with the same --cluster root. An initial --bootstrap-handoff receipt is only for the first process; omit it on later starts. The data, history and applied configuration remain in place.

Uncertain or interrupted work retains admission and exits nonzero; on S3 roots a restarted server takes it over through the lock's lease (restart after a failure). S3 release uses a conditional marker, so a lost release response cannot delete a later owner's lock. A nonclean exit is still a reason to inspect the original outcome, not to replay writes. Azure keeps its existing shutdown behavior: even a clean exit retains admission and requires the independent Azure recovery procedure. If OBJECT_STORE_CLIENT_MAX_RETRIES is set, it must be 0 (retries). A clean shutdown that leaves a reused key unsettled retains admission. Other direct writers still follow the ownership-transfer procedure.

Admission limits

Admission refuses excess operations and input immediately. Set these process-wide environment variables before startup; sizes are integer bytes. Invalid numbers warn and use the default; zero capacity refuses the corresponding lane.

Variable	Default	Resource
OMNIGRAPH_GLOBAL_INFLIGHT_MAX	64	Admitted write operations across actors
OMNIGRAPH_GLOBAL_BYTES_MAX	268435456 (256 MiB)	Estimated retained write-input bytes across actors
OMNIGRAPH_PER_ACTOR_INFLIGHT_MAX	16	Admitted writes per actor
OMNIGRAPH_PER_ACTOR_BYTES_MAX	4294967296 (4 GiB)	Estimated retained write-input bytes per actor
OMNIGRAPH_ACTIVE_ACTORS_MAX	1024	Actors with admitted writes; idle records are removed
OMNIGRAPH_INGRESS_INFLIGHT_MAX	64	Write requests retained by collection, operations or responses
OMNIGRAPH_INGRESS_BYTES_MAX	268435456 (256 MiB)	Reserved/retained write-body bytes
OMNIGRAPH_READ_INGRESS_INFLIGHT_MAX	64	Read/MCP requests with bodies retained by collection, execution or responses
OMNIGRAPH_READ_INGRESS_BYTES_MAX	67108864 (64 MiB)	Reserved/retained read/MCP body bytes
OMNIGRAPH_BODY_TIMEOUT_SECONDS	30	Time allowed to collect one complete request body

The default ingress allowances total 320 MiB across the independent lanes; configure both byte limits when setting an instance's input budget. Body timeouts above 86400 seconds also warn and use the 30-second default. Ingress reserves the route's maximum before reading, then reduces the reservation to the actual body size. Body collection limits remain 1 MiB for ordinary JSON requests, 32 MiB for bulk load and 64 KiB for MCP. Cluster deployment POSTs allow 16 MiB plus 1 KiB for the request envelope; the captured bundle itself is limited to 16 MiB. MCP uses the read-body lane after authentication. Only registered bulk routes receive the larger limit; a stored query named load or ingest keeps the ordinary JSON limit. Stored-query admission uses the registry's typed read/write kind. Disconnect does not release capacity held by executing work; the body timeout ends at complete collection and cannot cancel admitted execution. Input limits do not bound process RSS or all engine memory and I/O. Size instances for graph, workload and concurrency as well as request bytes.

The server allows 128 read-response observers and, independently, 64 write-response observers. Executing reads, pending results, bodies, yielded bytes and their server producers retain the slot until their final owner releases it. Bodyless reads consume only a read observer; slow reads cannot exhaust write-body or write-response capacity. Status routes bypass ordinary admission so they remain callable during saturation.

Export and change-baseline responses reserve 256 KiB each from a fixed 2 MiB process allowance, permitting eight simultaneous reservations. Each response allows three outstanding 64 KiB chunks plus one pending producer chunk; at most two chunks wait in its queue. Yielded chunks, including retained clones and slices, keep their reservation until released. A new response waits at most 250 ms for process capacity before HTTP 413. Once admitted, a producer waits for chunk capacity or disconnect without a per-chunk deadline; ordinary queue backpressure has no additional deadline. An interrupted baseline supplies no usable terminal cursor. Consumers must release each received server buffer to continue; an in-process consumer retaining every buffer can exhaust its lane. An HTTP client may still collect its received response: those client buffers are separate from the server's transport buffers. These limits cover transport payload buffers, not engine row/Arrow encoding, native work or total process RSS.

The engine separately limits retained keyed batches, keyed parse estimates and removed-ID collections across each operation's tables; see mutation limits. These fixed limits return HTTP 413 with structured resource_limit details before the operation's data fragments or publication, and leave the server available for smaller work. They do not undo an implicitly created load branch or earlier command effects, and do not establish a process memory ceiling or automatic retry safety.

Container

The container entrypoint reads OMNIGRAPH_CLUSTER and binds to port 8080:

docker run --rm -p 8080:8080 \
  -e OMNIGRAPH_CLUSTER=s3://company-data/omnigraph/company-brain \
  -e AWS_REGION=us-east-1 \
  -e AWS_ACCESS_KEY_ID \
  -e AWS_SECRET_ACCESS_KEY \
  -e OMNIGRAPH_SERVER_BEARER_TOKENS_JSON \
  ghcr.io/modernrelay/omnigraph-server:<tag>

For a local cluster, mount the complete cluster directory and point OMNIGRAPH_CLUSTER at the mount:

docker run --rm -p 8080:8080 \
  -v /srv/company-brain:/var/lib/omnigraph/cluster \
  -e OMNIGRAPH_CLUSTER=/var/lib/omnigraph/cluster \
  -e OMNIGRAPH_SERVER_BEARER_TOKEN \
  ghcr.io/modernrelay/omnigraph-server:<tag>

Terminate TLS at a load balancer or trusted reverse proxy. Keep bearer tokens and storage credentials in the platform's secret store.

Unix-socket access

omnigraph-server listens on TCP only. For clients with Unix socket support, a gateway can relay a socket to its loopback listener:

socat UNIX-LISTEN:/run/omnigraph/server.sock,fork TCP:127.0.0.1:8080

The version-1 CLI does not accept Unix socket targets; use a registered HTTPS endpoint instead. Relay requests and responses unchanged, including Omnigraph-Http-Api. Restrict socket-directory permissions; bearer authentication and policy still apply.

S3-compatible storage

For AWS S3, configure the standard AWS credential chain and region. For a compatible service, these variables may also be needed:

export AWS_ENDPOINT_URL_S3=https://objects.example.com
export AWS_S3_FORCE_PATH_STYLE=true

Set AWS_ALLOW_HTTP=true only for a trusted local development endpoint. Do not put credentials in cluster.yaml or graph URIs.

The same storage root must be visible to servers and out-of-band cluster or maintenance jobs. Submit schema/query changes and graph additions to the running server with cluster apply --target; the root's applied revision remains the deployment artifact. Direct apply requires ownership transfer before serving.

Azure Blob preview

Native az://<container>/<prefix> roots are implemented and tested with Azurite. A live managed-identity smoke deployment is complete, but Azure remains a qualification preview until the adversarial live-Azure matrix is complete.

Every mutation-capable Azure process must be admitted under the cluster root:

AZURE_STORAGE_ACCOUNT_NAME=companygraph \
AZURE_STORAGE_CLIENT_ID=<managed-identity-client-id> \
OMNIGRAPH_SERVER_BEARER_TOKENS_JSON='{"act-service":"secret"}' \
  omnigraph-azure-admission run \
    --mode server \
    --root az://omnigraph/company-brain \
    -- \
    omnigraph-server \
      --cluster az://omnigraph/company-brain \
      --bind 0.0.0.0:8080

Use the same wrapper with --mode job and the same canonical --root for bootstrap/apply jobs, cluster upgrade-ledger, root-addressed reconciliation, cluster force-unlock, direct graph writers, and maintenance. Root-addressed deployment status is read-only and does not need the lease. A v2 cluster's persisted lock is additional admission; it does not replace the Azure lease or its qualification boundary. The container entrypoint wraps an az:// cluster automatically. Replica-count settings are not a correctness fence: the admission lease is.

The checked-in Azure reference deployment contains the supported Container Apps topology, validation command, and stuck-lease runbook. A stranded Azure lease and a retained v2 cluster lock are separate: complete the lease recovery procedure before starting a wrapped cluster-unlock or reconciliation job. Do not break a lease until the owner has been identified and stopped.

Writer topology

Run one mutation-capable writer process per cluster. This includes servers, direct CLI writes, deployments, branch controls, and maintenance. On a v2 cluster, supported servers and direct CLI writers acquire the same exclusive cluster admission before writable graph work and recheck the applied inventory. CLI queries, exports, schema inspection/planning, lint and stored-query validation check the applied identity without acquiring or changing that lock; they refuse outstanding deployments or schema drift. Native maintenance, including repair preview, uses writer admission. An outstanding deployment admits only reconciliation of its exact original ID. Keep older binaries, raw storage tools and embedded writers outside this cooperating boundary stopped; the lock cannot fence their native storage I/O.

Admission remains held after an ordinary direct write command succeeds or a deployment completes. A clean server shutdown follows the bounded release described above; ending work without that proof retains admission. A completed read-only preflight refusal during new deployment, conversion or admission construction releases its admission. Recovery of accepted work, cancellation and uncertain effects retain it. A finished response, zero active HTTP requests, lock age or a stopped PID alone does not prove that previously accepted storage writes have settled. Before transferring ownership, stop the prior owner and establish that its graph work and control-store I/O are terminal. Exclude new admissions and other unlock attempts until the exact-ID unlock finishes. Then start the next owner. Use the root-addressed status and recovery commands to obtain the lock ID; unlocking never resolves an uncertain deployment.

Use the same current toolchain for restart. Older binaries refuse the released S3 lock record; do not delete that record to bypass the version check.

A legacy v1 ledger refuses ordinary serving and writes. Stop its previous owners and explicitly convert the ledger before using the current admission protocol.

Read replicas and zero-downtime overlapping writer replicas are not currently a supported topology. Prefer stop-then-start replacement for a mutation-capable server.

Authentication and policy

Configure tokens from environment or a mounted secret file:

• OMNIGRAPH_SERVER_BEARER_TOKEN
• OMNIGRAPH_SERVER_BEARER_TOKENS_JSON
• OMNIGRAPH_SERVER_BEARER_TOKENS_FILE
• OMNIGRAPH_SERVER_BEARER_TOKENS_AWS_SECRET in the AWS-enabled build

Policy bundles come from the applied cluster. See HTTP server and Authorization and actors.

Upgrade and backup

Back up the whole cluster root, not selected physical files. Before a release:

1. read the release notes;
2. quiesce writers;
3. verify backups or exports;
4. upgrade the fleet together;
5. restart and run representative reads and writes.

Graph-format upgrades and cluster-ledger conversion are separate operations. Follow the storage upgrade guide for qualified format routes; cluster upgrade-ledger never resets or rewrites graph history.