Skip to content

Commit 56d22b8

Browse files
committed
Adjust queue config and rename base uri
1 parent 3c2a759 commit 56d22b8

1 file changed

Lines changed: 24 additions & 14 deletions

File tree

doc/rfc/stovepipe/steps/process.md

Lines changed: 24 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -25,12 +25,12 @@ For a delivery carrying request id `R`:
2525
4. R.State is accepted. Load (or create) the Queue row Q.
2626
5. Coalesce: if R.Sequence < Q.latest_request_seq:
2727
- a newer head exists -> mark R superseded, ack, return. (No slot consumed.)
28-
6. R is the latest head. Gate: if Q.in_flight_count >= Q.max_concurrent:
28+
6. R is the latest head. Gate: if Q.in_flight_count >= max_concurrent (from queue config; see below):
2929
- defer (Option 1 or Option 2 below) -> re-check until the slot frees (admit) or a newer head supersedes it. See [Waiting for a slot](#waiting-for-a-slot).
3030
7. Admit R:
3131
a. Derive build strategy + baseline (see "Build-strategy decision").
3232
b. CAS the Queue row: in_flight_count += 1.
33-
c. CAS the Request: accepted -> processing, persist build_strategy + baseline_uri.
33+
c. CAS the Request: accepted -> processing, persist build_strategy + base_uri.
3434
d. Publish R to build.
3535
e. ack.
3636
```
@@ -52,13 +52,15 @@ Strategy and baseline are persisted on the Request and are **immutable**: a rede
5252

5353
## Per-Queue concurrency gate
5454

55-
Validation is expensive and shares a baseline, so heads arriving while an earlier one runs `build → buildsignal → record` must not all start builds at once. Gate state on the Queue row:
55+
Validation is expensive and shares a baseline, so heads arriving while an earlier one runs `build → buildsignal → record` must not all start builds at once.
5656

57-
| Field | Meaning |
58-
|---|---|
59-
| `last_green_uri` | Bookmark `record` advances on whole-repo green; empty until first green. |
60-
| `in_flight_count` | Requests past `process` and not yet terminal. `process` increments on admit; `record` (or DLQ reconciliation) decrements on terminal. |
61-
| `max_concurrent` | Cap on concurrent in-flight validations. **Default 1.** |
57+
**Runtime state** lives on the Queue row; the **concurrency cap** does not — it is deployment configuration, resolved at gate-check time the same way SubmitQueue separates `storage` from `queueconfig` (see [submitqueue/extension/queueconfig/README.md](../../../../submitqueue/extension/queueconfig/README.md)): pipeline stages read mutable state from the store and read knobs like `max_concurrent` from a config `Store` (or a wiring default for MVP). Config is not written by ingest/process/record and does not need optimistic locking.
58+
59+
| Source | Field | Meaning |
60+
|---|---|---|
61+
| Queue row | `last_green_uri` | Bookmark `record` advances on whole-repo green; empty until first green. |
62+
| Queue row | `in_flight_count` | Requests past `process` and not yet terminal. `process` increments on admit; `record` (or DLQ reconciliation) decrements on terminal. |
63+
| Queue config | `max_concurrent` | Cap on concurrent in-flight validations. **Default 1** (global wiring default for MVP; per-queue override when a Stovepipe `queueconfig` extension lands). |
6264

6365
A slot is held for the **entire** Phase 1 cycle (`process → build → buildsignal → record`), not just while `process` runs. It is released when the Request reaches **any** terminal state and `in_flight_count` is decremented — `record` writing green *or* not-green, or the DLQ reconciler forcing a terminal not-green (see [integrity](#in_flight_count-integrity)). A build *failure* frees the slot just like a success; only a Request that never terminates keeps its slot.
6466

@@ -83,7 +85,7 @@ Correctness rests on four rules, all with MVP primitives already in place:
8385

8486
The only cost is speculation: a build on an older baseline re-tests deltas a concurrent build already greened past — correct but wasteful, growing with how far the baseline lags. Bounding that lag ("drain before adopting a new baseline") is a **cost governor, not a safety gate**. It inherits, but doesn't worsen, the incremental-build soundness assumption already used at N=1.
8587

86-
So per-baseline concurrency isn't an unsolved semantics problem — it's speculative validation with a lag-bounded baseline. It's deferred only for the per-lineage bookkeeping (rules 2–4, derivable from `Request.Sequence` + `Request.BaselineURI`) and a coalesce-latest-N policy, neither of which the MVP forecloses.
88+
So per-baseline concurrency isn't an unsolved semantics problem — it's speculative validation with a lag-bounded baseline. It's deferred only for the per-lineage bookkeeping (rules 2–4, derivable from `Request.Sequence` + `Request.BaseURI`) and a coalesce-latest-N policy, neither of which the MVP forecloses.
8789

8890
## Backlog coalescing
8991

@@ -161,28 +163,31 @@ On a crash between admit and `record`, the Request stays non-terminal; visibilit
161163
- **Re-ingest of a superseded URI.** Ingest dedups on `(Queue, URI)` and returns the existing (now terminal `superseded`) id; `process` acks it as a no-op (step 2). Correct: a URI is only superseded for a *strictly newer* head, so re-validating it is never wanted.
162164
- **Gate closed, no newer head.** The single latest head waits for a slot until the in-flight validation completes — the steady state, not an error.
163165
- **Head equals last-green.** `IsAncestor(lastGreen, R.URI)` with `R.URI == lastGreen` is degenerate; treat as already-green, or (simpler) run an incremental build with an empty delta. Left to `build`.
164-
- **Queue row missing.** First head for a Queue: ingest get-or-creates the row with defaults (`in_flight_count = 0`, empty `last_green_uri`, `max_concurrent` from config). `process` treats a missing row as retryable (ingest write not yet visible).
166+
- **Queue row missing.** First head for a Queue: ingest get-or-creates the row with defaults (`in_flight_count = 0`, empty `last_green_uri`). `process` treats a missing row as retryable (ingest write not yet visible).
165167

166168
## Entity model
167169

168170
### Queue (new persisted entity)
169171

172+
Runtime coordination only — fields the pipeline writes under CAS:
173+
170174
| Field | Role | Written by |
171175
|---|---|---|
172176
| `name` | Stable logical id (`monorepo/main`); the string ingest accepts | ingest (create) |
173177
| `last_green_uri` | Bookmark; empty until first green | record |
174178
| `in_flight_count` | Active Phase 1 validations | process (+1), record/DLQ (−1) |
175-
| `max_concurrent` | Concurrency cap (default 1) | config at create |
176179
| `latest_request_seq` | Highest request sequence ingested | ingest |
177180
| `version` | Optimistic-locking version | all writers |
178181

182+
Per-queue knobs such as `max_concurrent` live outside this row — see [Per-Queue concurrency gate](#per-queue-concurrency-gate).
183+
179184
### Request (additions to the existing entity)
180185

181186
| Field | Role |
182187
|---|---|
183188
| `Sequence` | The per-Queue counter value `n` behind the id; the coalescing order key |
184189
| `BuildStrategy` | `incremental_since_green` \| `full_monorepo`; immutable once set by `process` |
185-
| `BaselineURI` | Last-green URI used as the incremental baseline; empty for full builds |
190+
| `BaseURI` | Last-green URI used as the incremental base; empty for full builds |
186191

187192
**States** (extending today's `accepted`-only machine):
188193

@@ -207,10 +212,15 @@ No "list requests by queue/state" query is introduced; coalescing uses the singl
207212

208213
## Waiting for a slot
209214

210-
When the gate is closed, `process` must defer the latest head without admitting it (no `in_flight_count` increment, no publish to `build`). Two options fit; pick one at implementation time. Both re-run the same **coalesce-then-gate** checks on every wake-up (steps 5 → 6):
215+
When the gate is closed, `process` must defer the latest head without admitting it (no `in_flight_count` increment, no publish to `build`). Two options:
216+
217+
- Park until a build slot opens, and extend visibility
218+
- Use PublishAfter to re-enqeue the current head if no build slot is available
219+
220+
Both re-run the same **coalesce-then-gate** checks on every wake-up (steps 5 → 6):
211221

212222
1. **Stale? (checked first.)** If `R.Sequence < latest_request_seq`, `R` is no longer latest → supersede it (ack). A newer head is admitted by its own delivery when its slot attempt runs.
213-
2. **Slot free?** If `in_flight_count < max_concurrent` and `R` is still latest → admit (step 7).
223+
2. **Slot free?** If `in_flight_count < max_concurrent` (from config) and `R` is still latest → admit (step 7).
214224

215225
Neither option admits to `build` until the gate opens.
216226

0 commit comments

Comments
 (0)