Add sql transport backed by Postgres by wseaton · Pull Request #452 · llm-d/llm-d-async

wseaton · 2026-09-15T20:45:26Z

Adds an experimental sql transport (opt-in with --transport sql): dispatchers lease hash partitions of each queue and batch dispatch, acks, and submits, and each request's payload has its own column, so dispatch decodes only the envelope. A sql-quota gate enforces redis-quota limits exactly across processors from the same database, so the transport needs no Redis; its concurrency protocol is proved in Veil. Stacked on #458; its first thirteen commits belong to that PR (report).

Release note

Adds an experimental `sql` transport backed by Postgres, opt-in with `--transport sql`; its schema may change between releases without a migration. Dispatchers lease hash partitions of each queue and batch dispatch, acks, and cancellation checks, and each request's envelope and payload are stored in separate columns. A `sql-quota` gate enforces `redis-quota` limits exactly across processors from the same database, so the sql transport needs no Redis.

Breaking: `redis-quota` now validates its params at startup: a `mode` other than `rate-limit` or `concurrency`, a `gating_mode` other than `blocking` or `classifying`, or a non-positive `window` in rate-limit mode fails gate creation. Before, an unknown mode admitted everything, any other gating mode classified, and a zero window never limited.

github-actions · 2026-09-16T13:19:54Z

Unsigned commits detected! Please sign your commits.

For instructions on how to set up GPG/SSH signing and verify your commits, please see GitHub Documentation.

Signed-off-by: Will Eaton <weaton@redhat.com>

Workers checked cancellation with one query per request before dispatch, serialising the whole pipeline on a round trip each. Coalesce the checks into one query per batch: 100k requests drained at 1238 req/s instead of 287, with Postgres at 0.53 cores instead of pinned at 2. Batch size and linger are configurable. Signed-off-by: Will Eaton <weaton@redhat.com>

async_requests keeps the envelope in a text column and the payload in a bytea column, so dispatch decodes only the envelope and attaches the payload bytes as stored. A retry rewrites the envelope and leaves the payload in place. Signed-off-by: Will Eaton <weaton@redhat.com>

Signed-off-by: Will Eaton <weaton@redhat.com>

Poll skipped rows it already tracked, but a tracked row is only pending again after a reset or retry, so the skip left it stamped under the new epoch with no worker and no reset path. Signed-off-by: Will Eaton <weaton@redhat.com>

A stalled query held every worker's cancellation check until its own context gave up. Signed-off-by: Will Eaton <weaton@redhat.com>

Signed-off-by: Will Eaton <weaton@redhat.com>

A fenced outcome from an old attempt uncounted work dispatched under the partition's new epoch, and Abandon by key could return a newer attempt to pending. Both now match the stamp. Signed-off-by: Will Eaton <weaton@redhat.com>

Each queue had its own store timeout in a serial pass, so a slow database stretched the heartbeat past the lease TTL once a flow had three or more queues. Signed-off-by: Will Eaton <weaton@redhat.com>

Retry's store write and its bookkeeping are not atomic with Poll. A poll between them re-tracks the key under the same stamp, the late done then untracks it, and an Abandon of that attempt finds nothing to orphan. Signed-off-by: Will Eaton <weaton@redhat.com>

A dispatch whose reply is lost after a retry reuses the retried stamp, so reconcile sees it as tracked, and the retry's late done then untracks it. Signed-off-by: Will Eaton <weaton@redhat.com>

A stamp of key and epoch named a lease tenure, so bookkeeping from a finished dispatch matched a redispatch under the same epoch. Every dispatch now draws an attempt. Signed-off-by: Will Eaton <weaton@redhat.com>

TestModelTrace drives real Consumers against Postgres with a seeded random walk and writes each operation as model steps plus the state it left. llm-d-async-formal replays them against the DispatchToken model. Skipped unless SQLQUEUE_TRACE_DIR is set. Signed-off-by: Will Eaton <weaton@redhat.com>

Move the attribute and gating logic out of the Redis quota gate into flowcontrol.QuotaGate over a QuotaStore interface. redis-quota keeps its Lua scripts as one QuotaStore; sql-quota counts in Postgres, so the sql transport needs no Redis. Unknown mode or gating_mode values now fail at startup instead of letting requests through. Limits are exact across processors. Each quota function locks its key row first; under READ COMMITTED every later statement in the function takes a fresh snapshot and sees what the previous lock holder committed, which a single statement cannot, since its snapshot predates the wait. Each processor keeps at most one statement in flight per key and sends the requests that arrived meanwhile together, so the lock is taken once per batch rather than per request. Without that, one statement per request used 5-10x the Postgres CPU for a fraction of the throughput. Concurrency slots belong to a heartbeated holder and stop counting when its lease lapses; a lapsed holder cannot renew and registers under a new name. That replaces the counter TTL and the llm-d#335 class of expiry bugs. Rate limits log (time, count) per admitted batch, so the log is bounded by batches in the window rather than by the limit. Signed-off-by: Will Eaton <weaton@redhat.com>

database/sql keeps two idle connections by default and never caps open ones, so every burst of concurrent statements started a new Postgres backend per statement and closed it after. A 10 s quota benchmark forked 12,310 backends; keeping idle connections cut that to 257, doubled batched throughput at 64 keys, and cut p99 from 52 ms to 16-18 ms. Unbounded opens also ran into the server's max_connections. Open now caps open connections and keeps the same number idle, 32 by default. The processor sets it with max_connections in the sql config; producers get the default. Signed-off-by: Will Eaton <weaton@redhat.com>

A quota statement that returns an error may still have committed, and counts are not idempotent. Retrying a release whose reply was lost subtracted it twice and freed a slot a running request still held, so a third request ran under a limit of two. An acquire whose reply was lost left a grant counted that no caller held, for as long as the holder kept heartbeating. The Veil model in llm-d-async-formal (QuotaSlots) found both; TestQuotaStoreLostReleaseReplyCannotOverAdmit and TestQuotaStoreLostGrantReplyDoesNotLeak reproduce them against Postgres through a proxy that drops one reply after the statement commits. Now any statement error retires the holder it ran under. A retired holder takes no new grants, is renewed while a statement in flight, a handed-out grant or a queued release still names it, and is deleted once none does; the delete is idempotent and the heartbeat retries it. Whatever the failed statement left counted goes with the holder. QuotaRetire proves exact and no_leak inductive for this protocol, kernel-checked, for any number of processes, holders, keys and slots. The proof assumes a holder is renewed while its grants run. If renewals fail for a whole lease, its slots free under running requests; that is logged. Signed-off-by: Will Eaton <weaton@redhat.com>

Every async_quota_admit call on a key deleted the key's entries older than its own window, so a short-window gate emptied a long-window gate's log: after a one-minute gate filled a key, a call from a ten-second gate on it let the minute gate admit past its limit. The Veil model QuotaRateShared finds this in five steps and TestAdmitQuotaWindowsCountSeparately reproduces it. Rate state now lives in async_quota_windows, keyed by key and window, and admits carry their window, so each window deletes and counts only its own entries. Sharing one log across windows would also count a request that passes a minute gate and an hour gate twice. QuotaRateWindowed proves the limit holds per window, kernel-checked, assuming the database clock never goes back. Existing sql-quota tables must be dropped. Signed-off-by: Will Eaton <weaton@redhat.com>

make test-sql covered producer-sql and pkg/sqlflow but not the flowcontrol quota gate tests, whose Postgres cases skipped in CI. They also truncated the shared tables, so running them in parallel with pkg/sqlflow under make test with TEST_POSTGRES_URL set flaked. They now open the store in a schema they create and drop, and test-sql runs them. Signed-off-by: Will Eaton <weaton@redhat.com>

A drained queue table leaves reltuples = 0, and the planner then scanned the whole table per pop or dispatch. Statements now reach rows by key and the ordered picks run in SQL functions with seq and bitmap scans off. Signed-off-by: Will Eaton <weaton@redhat.com>

The sql producer stores the payload as raw bytes apart from the envelope, so nothing checked it. Mirror the redis producer. Signed-off-by: Will Eaton <weaton@redhat.com>

Backlog and HasRequests counted dispatched rows, so broker_backlog overlapped inflight_requests and a closed gate was recorded with only in-flight work left. redis-sortedset removes a member on claim; match it. Signed-off-by: Will Eaton <weaton@redhat.com>

Signed-off-by: Will Eaton <weaton@redhat.com>

The harness never attached the flow's cancellation checker, so neither the pre-dispatch check nor the in-flight poll reached CancelledKeys. Signed-off-by: Will Eaton <weaton@redhat.com>

Signed-off-by: Will Eaton <weaton@redhat.com>

Close did not wait for the release flush, and left the holder row to lapse, so a rolling restart held slots for up to the lease TTL. Signed-off-by: Will Eaton <weaton@redhat.com>

CONTRIBUTING requires experimental features to be opt-in; the schema is not migrated, so changes to it should not count as breaking. Signed-off-by: Will Eaton <weaton@redhat.com>

wseaton force-pushed the sql-transport-partition-leases branch 8 times, most recently from 913449f to 36c8d6c Compare September 16, 2026 13:17

wseaton changed the title ~~Add sql transport backed by Postgres or SQLite~~ Add sql transport backed by Postgres Sep 16, 2026

wseaton force-pushed the sql-transport-partition-leases branch from 36c8d6c to 1d05a85 Compare September 16, 2026 13:19

wseaton force-pushed the sql-transport-partition-leases branch 4 times, most recently from e1d977f to ef94958 Compare September 16, 2026 16:22

This was referenced Sep 17, 2026

api: carry request payloads as json.RawMessage #455

Closed

api: split a request's payload from its envelope #457

Closed

wseaton added a commit to wseaton/llm-d-async that referenced this pull request Sep 17, 2026

Add release note for llm-d#452

1aeb7c4

Signed-off-by: Will Eaton <weaton@redhat.com>

wseaton force-pushed the sql-transport-partition-leases branch from ef94958 to 1aeb7c4 Compare September 17, 2026 16:24

wseaton added a commit to wseaton/llm-d-async that referenced this pull request Sep 17, 2026

Add release note for llm-d#452

fe5e417

Signed-off-by: Will Eaton <weaton@redhat.com>

wseaton force-pushed the sql-transport-partition-leases branch from 1aeb7c4 to fe5e417 Compare September 17, 2026 17:17

wseaton added a commit to wseaton/llm-d-async that referenced this pull request Sep 17, 2026

Add release note for llm-d#452

e42355d

Signed-off-by: Will Eaton <weaton@redhat.com>

wseaton force-pushed the sql-transport-partition-leases branch from fe5e417 to e42355d Compare September 17, 2026 19:31

This was referenced Sep 22, 2026

redis-sortedset: requests sharing a deadline dequeue in random order #462

Closed

redis: order equal-deadline requests by a Redis sequence #463

Merged

wseaton added a commit to wseaton/llm-d-async that referenced this pull request Sep 22, 2026

Add release note for llm-d#452

78f8d6e

Signed-off-by: Will Eaton <weaton@redhat.com>

wseaton force-pushed the sql-transport-partition-leases branch from e42355d to 624156f Compare September 22, 2026 20:20

wseaton added a commit to wseaton/llm-d-async that referenced this pull request Oct 6, 2026

Add release note for llm-d#452

66e2e6d

Signed-off-by: Will Eaton <weaton@redhat.com>

wseaton force-pushed the sql-transport-partition-leases branch from b9dcfaf to c7e112a Compare October 6, 2026 16:49

wseaton marked this pull request as ready for review October 6, 2026 16:49

wseaton added 28 commits October 6, 2026 13:03

fix(chart): isolate SQL connection secrets

ac529f0

Signed-off-by: Will Eaton <weaton@redhat.com>

Add release note for llm-d#452

020c79d

Signed-off-by: Will Eaton <weaton@redhat.com>

fix(sql): hand out every request a poll dispatches

459cfb2

Poll skipped rows it already tracked, but a tracked row is only pending again after a reset or retry, so the skip left it stamped under the new epoch with no worker and no reset path. Signed-off-by: Will Eaton <weaton@redhat.com>

fix(sql): bound each cancellation batch by the store timeout

08bde93

A stalled query held every worker's cancellation check until its own context gave up. Signed-off-by: Will Eaton <weaton@redhat.com>

sql: count the backlog and its deadline windows in one pass

ebef9e1

Signed-off-by: Will Eaton <weaton@redhat.com>

fix(sql): rebalance queues concurrently

4d30938

Each queue had its own store timeout in a serial pass, so a slow database stretched the heartbeat past the lease TTL once a flow had three or more queues. Signed-off-by: Will Eaton <weaton@redhat.com>

test(sql): show reconcile trusting a finished retry's bookkeeping

5dc2799

A dispatch whose reply is lost after a retry reuses the retried stamp, so reconcile sees it as tracked, and the retry's late done then untracks it. Signed-off-by: Will Eaton <weaton@redhat.com>

fix(sql): fence each dispatch by its own attempt

c822f92

A stamp of key and epoch named a lease tenure, so bookkeeping from a finished dispatch matched a redispatch under the same epoch. Every dispatch now draws an attempt. Signed-off-by: Will Eaton <weaton@redhat.com>

fix(sql): reject a payload that is not JSON at submit

2d09a38

The sql producer stores the payload as raw bytes apart from the envelope, so nothing checked it. Mirror the redis producer. Signed-off-by: Will Eaton <weaton@redhat.com>

docs(sql): note the stricter redis-quota param checks

339ff73

Signed-off-by: Will Eaton <weaton@redhat.com>

test(sql): cancel an in-flight request through the sql checker

ff7d177

The harness never attached the flow's cancellation checker, so neither the pre-dispatch check nor the in-flight poll reached CancelledKeys. Signed-off-by: Will Eaton <weaton@redhat.com>

fix(sql): build the schema functions instead of holding a global

c6a7fee

Signed-off-by: Will Eaton <weaton@redhat.com>

chore(sql): list producer-sql as a module and require api v0.10.0

86e9277

Signed-off-by: Will Eaton <weaton@redhat.com>

docs(sql): mark the redis-quota param checks as breaking

25d447b

Signed-off-by: Will Eaton <weaton@redhat.com>

docs(sql): describe the in-flight cancellation checks

5aa17c3

Signed-off-by: Will Eaton <weaton@redhat.com>

fix(sql): flush quota releases and drop idle holders on close

dfa5540

Close did not wait for the release flush, and left the holder row to lapse, so a rolling restart held slots for up to the lease TTL. Signed-off-by: Will Eaton <weaton@redhat.com>

wseaton force-pushed the sql-transport-partition-leases branch from c7e112a to dfa5540 Compare October 6, 2026 17:06

docs(sql): mark the sql transport and sql-quota experimental

0fb8cc4

CONTRIBUTING requires experimental features to be opt-in; the schema is not migrated, so changes to it should not count as breaking. Signed-off-by: Will Eaton <weaton@redhat.com>

Navigation Menu

Repository navigation

Add sql transport backed by Postgres - #452

Add sql transport backed by Postgres#452
wseaton wants to merge 44 commits into
llm-d:mainfrom
wseaton:sql-transport-partition-leases

wseaton commented Sep 15, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

wseaton commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Release note

Uh oh!

github-actions Bot commented Sep 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

wseaton commented Sep 15, 2026 •

edited

Loading