You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: doc/rfc/stovepipe/steps/process.md
+24-14Lines changed: 24 additions & 14 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -25,12 +25,12 @@ For a delivery carrying request id `R`:
25
25
4. R.State is accepted. Load (or create) the Queue row Q.
26
26
5. Coalesce: if R.Sequence < Q.latest_request_seq:
27
27
- a newer head exists -> mark R superseded, ack, return. (No slot consumed.)
28
-
6. R is the latest head. Gate: if Q.in_flight_count >= Q.max_concurrent:
28
+
6. R is the latest head. Gate: if Q.in_flight_count >= max_concurrent (from queue config; see below):
29
29
- defer (Option 1 or Option 2 below) -> re-check until the slot frees (admit) or a newer head supersedes it. See [Waiting for a slot](#waiting-for-a-slot).
30
30
7. Admit R:
31
31
a. Derive build strategy + baseline (see "Build-strategy decision").
32
32
b. CAS the Queue row: in_flight_count += 1.
33
-
c. CAS the Request: accepted -> processing, persist build_strategy + baseline_uri.
33
+
c. CAS the Request: accepted -> processing, persist build_strategy + base_uri.
34
34
d. Publish R to build.
35
35
e. ack.
36
36
```
@@ -52,13 +52,15 @@ Strategy and baseline are persisted on the Request and are **immutable**: a rede
52
52
53
53
## Per-Queue concurrency gate
54
54
55
-
Validation is expensive and shares a baseline, so heads arriving while an earlier one runs `build → buildsignal → record` must not all start builds at once. Gate state on the Queue row:
55
+
Validation is expensive and shares a baseline, so heads arriving while an earlier one runs `build → buildsignal → record` must not all start builds at once.
56
56
57
-
| Field | Meaning |
58
-
|---|---|
59
-
|`last_green_uri`| Bookmark `record` advances on whole-repo green; empty until first green. |
60
-
|`in_flight_count`| Requests past `process` and not yet terminal. `process` increments on admit; `record` (or DLQ reconciliation) decrements on terminal. |
61
-
|`max_concurrent`| Cap on concurrent in-flight validations. **Default 1.**|
57
+
**Runtime state** lives on the Queue row; the **concurrency cap** does not — it is deployment configuration, resolved at gate-check time the same way SubmitQueue separates `storage` from `queueconfig` (see [submitqueue/extension/queueconfig/README.md](../../../../submitqueue/extension/queueconfig/README.md)): pipeline stages read mutable state from the store and read knobs like `max_concurrent` from a config `Store` (or a wiring default for MVP). Config is not written by ingest/process/record and does not need optimistic locking.
58
+
59
+
| Source | Field | Meaning |
60
+
|---|---|---|
61
+
| Queue row |`last_green_uri`| Bookmark `record` advances on whole-repo green; empty until first green. |
62
+
| Queue row |`in_flight_count`| Requests past `process` and not yet terminal. `process` increments on admit; `record` (or DLQ reconciliation) decrements on terminal. |
63
+
| Queue config |`max_concurrent`| Cap on concurrent in-flight validations. **Default 1** (global wiring default for MVP; per-queue override when a Stovepipe `queueconfig` extension lands). |
62
64
63
65
A slot is held for the **entire** Phase 1 cycle (`process → build → buildsignal → record`), not just while `process` runs. It is released when the Request reaches **any** terminal state and `in_flight_count` is decremented — `record` writing green *or* not-green, or the DLQ reconciler forcing a terminal not-green (see [integrity](#in_flight_count-integrity)). A build *failure* frees the slot just like a success; only a Request that never terminates keeps its slot.
64
66
@@ -83,7 +85,7 @@ Correctness rests on four rules, all with MVP primitives already in place:
83
85
84
86
The only cost is speculation: a build on an older baseline re-tests deltas a concurrent build already greened past — correct but wasteful, growing with how far the baseline lags. Bounding that lag ("drain before adopting a new baseline") is a **cost governor, not a safety gate**. It inherits, but doesn't worsen, the incremental-build soundness assumption already used at N=1.
85
87
86
-
So per-baseline concurrency isn't an unsolved semantics problem — it's speculative validation with a lag-bounded baseline. It's deferred only for the per-lineage bookkeeping (rules 2–4, derivable from `Request.Sequence` + `Request.BaselineURI`) and a coalesce-latest-N policy, neither of which the MVP forecloses.
88
+
So per-baseline concurrency isn't an unsolved semantics problem — it's speculative validation with a lag-bounded baseline. It's deferred only for the per-lineage bookkeeping (rules 2–4, derivable from `Request.Sequence` + `Request.BaseURI`) and a coalesce-latest-N policy, neither of which the MVP forecloses.
87
89
88
90
## Backlog coalescing
89
91
@@ -161,28 +163,31 @@ On a crash between admit and `record`, the Request stays non-terminal; visibilit
161
163
-**Re-ingest of a superseded URI.** Ingest dedups on `(Queue, URI)` and returns the existing (now terminal `superseded`) id; `process` acks it as a no-op (step 2). Correct: a URI is only superseded for a *strictly newer* head, so re-validating it is never wanted.
162
164
-**Gate closed, no newer head.** The single latest head waits for a slot until the in-flight validation completes — the steady state, not an error.
163
165
-**Head equals last-green.**`IsAncestor(lastGreen, R.URI)` with `R.URI == lastGreen` is degenerate; treat as already-green, or (simpler) run an incremental build with an empty delta. Left to `build`.
164
-
-**Queue row missing.** First head for a Queue: ingest get-or-creates the row with defaults (`in_flight_count = 0`, empty `last_green_uri`, `max_concurrent` from config). `process` treats a missing row as retryable (ingest write not yet visible).
166
+
-**Queue row missing.** First head for a Queue: ingest get-or-creates the row with defaults (`in_flight_count = 0`, empty `last_green_uri`). `process` treats a missing row as retryable (ingest write not yet visible).
165
167
166
168
## Entity model
167
169
168
170
### Queue (new persisted entity)
169
171
172
+
Runtime coordination only — fields the pipeline writes under CAS:
173
+
170
174
| Field | Role | Written by |
171
175
|---|---|---|
172
176
|`name`| Stable logical id (`monorepo/main`); the string ingest accepts | ingest (create) |
173
177
|`last_green_uri`| Bookmark; empty until first green | record |
174
178
|`in_flight_count`| Active Phase 1 validations | process (+1), record/DLQ (−1) |
175
-
|`max_concurrent`| Concurrency cap (default 1) | config at create |
@@ -207,10 +212,15 @@ No "list requests by queue/state" query is introduced; coalescing uses the singl
207
212
208
213
## Waiting for a slot
209
214
210
-
When the gate is closed, `process` must defer the latest head without admitting it (no `in_flight_count` increment, no publish to `build`). Two options fit; pick one at implementation time. Both re-run the same **coalesce-then-gate** checks on every wake-up (steps 5 → 6):
215
+
When the gate is closed, `process` must defer the latest head without admitting it (no `in_flight_count` increment, no publish to `build`). Two options:
216
+
217
+
- Park until a build slot opens, and extend visibility
218
+
- Use PublishAfter to re-enqeue the current head if no build slot is available
219
+
220
+
Both re-run the same **coalesce-then-gate** checks on every wake-up (steps 5 → 6):
211
221
212
222
1.**Stale? (checked first.)** If `R.Sequence < latest_request_seq`, `R` is no longer latest → supersede it (ack). A newer head is admitted by its own delivery when its slot attempt runs.
213
-
2.**Slot free?** If `in_flight_count < max_concurrent` and `R` is still latest → admit (step 7).
223
+
2.**Slot free?** If `in_flight_count < max_concurrent`(from config) and `R` is still latest → admit (step 7).
214
224
215
225
Neither option admits to `build` until the gate opens.
0 commit comments