Skip to content

Commit 8437031

Browse files
committed
docs(workflow): document queue and database ownership by service
## Summary ### Why? Issue #211 (follow-up from PR #205) asks for a single place that records the submitqueue topology at a high level: which service owns each queue topic and database table, for reading and writing. The workflow RFC already covers the cross-queue flow, so ownership belongs alongside it. ### What? Append an "Ownership by service" section to doc/rfc/submitqueue/workflow.md, organized per service to match how the system is operated: - Gateway — RPC entry point and owner of the request log. Publishes the entry topics (start, cancel), consumes log, and is the sole reader and writer of request_log. - Orchestrator — the pipeline. Publishes and consumes the pipeline-internal topics (validate, batch, score, speculate, build, buildsignal, merge, conclude; speculate is the re-entrant hub), publishes log without consuming it, and owns the pipeline state tables (request, batch, batch_dependent, build; with request_batch, speculation_tree, and change_provider defined but not yet wired). - Shared queue — the messagequeue tables both services read and write, backed by a database kept separate from application state. A closing "Request-log ownership invariant" section captures the crux of PR #205: the orchestrator only emits log events, the gateway is the sole consumer and persister. Code is referenced with inline spans (matching the rest of the doc) rather than links. Documentation only; no code, schema, or proto changes. ## Test Plan - ✅ `make lint` (clean tree) - Cross-checked topics against submitqueue/core/consumer/registry.go, database ownership against the compose DSNs (mysql-app vs mysql-queue) and the gateway/orchestrator store usage. ## Issue Closes #211
1 parent 43670be commit 8437031

1 file changed

Lines changed: 28 additions & 0 deletions

File tree

doc/rfc/submitqueue/workflow.md

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,3 +80,31 @@ The pipeline has two cycles: `speculate → build → buildsignal → speculate`
8080
| **merge** | BatchID | conclude, speculate | Merge the batch and advance the queue |
8181
| **conclude** | BatchID || Map terminal batch state to request state |
8282
| **log** | RequestLog || Gateway-owned sink: persists request log events to storage |
83+
84+
## Ownership by service
85+
86+
Each topic and database table is owned by exactly one service. The gateway and orchestrator never touch each other's application data; the only thing they share is the queue.
87+
88+
### Gateway
89+
90+
The gateway is the RPC entry point and the owner of the request log.
91+
92+
- **Queues** — publishes the entry topics `start` (from `Land`) and `cancel` (from `Cancel`); consumes the `log` topic. It is the only writer of `start`/`cancel` and the only consumer of `log`.
93+
- **Database** — owns the request log (`request_log`): the only service that reads or writes it. The gateway persists every `log` event it consumes, and also writes `request_log` synchronously from its RPC handlers (`accepted` on `Land`, `cancelling` on `Cancel`) so the entry is visible the moment the RPC returns.
94+
95+
### Orchestrator
96+
97+
The orchestrator is the pipeline: it advances a request from `start` through to a terminal state.
98+
99+
- **Queues** — publishes and consumes the pipeline-internal topics `validate`, `batch`, `score`, `speculate`, `build`, `buildsignal`, `merge`, and `conclude` as a request moves stage to stage. `speculate` is the re-entrant hub: `build → buildsignal → speculate` (CI feedback) and `merge → speculate` (advance the next batch) both publish back to it. The orchestrator also publishes to `log` but never consumes it.
100+
- **Database** — owns the pipeline state: `request`, `batch`, `batch_dependent`, and `build` (plus `request_batch`, `speculation_tree`, and `change_provider`, whose stores are defined but not yet wired). The orchestrator is the only writer of these tables.
101+
102+
### Shared: the queue
103+
104+
The messaging queue is shared infrastructure that both services read and write — the gateway and orchestrator hand work to each other only by publishing and consuming topics. It is backed by its own database (the `queue_messages`, `queue_offsets`, `queue_delivery_state`, `queue_partition_leases`, and `queue_subscriber_heartbeats` tables from the `messagequeue` extension), kept separate from application state (`mysql-queue` in the example deployment). A topic with no registered subscription on a service is publish-only there.
105+
106+
## Request-log ownership invariant
107+
108+
The request log has exactly one owner: the **gateway**. The orchestrator only *emits* request-log events by publishing to the `log` topic via `submitqueue/core/request.PublishLog`; it never persists them. The gateway is the sole consumer of `log` (consumer group `gateway-log`) and the only writer of `request_log`.
109+
110+
This keeps all request-log writes in one service: the orchestrator stays a pure pipeline that emits events, and the gateway owns the request log end to end. See PR #205 for the move of the `log` consumer from the orchestrator to the gateway.

0 commit comments

Comments
 (0)