Commit 04445d2
committed
feat(demo): one quickstart, three provider modes
## Summary
### Why?
The repository had two divergent local paths, and the quickstart documented the wrong one.
`make local-submitqueue-start` brought up an all-fake stack — fake change provider, fake CI, **noop merger** — and `doc/howto/QUICKSTART.md` (#591) described that. It reached `landed` in seconds with no credentials, but nothing was ever pushed, so it proved the pipeline's choreography and nothing about merging. `make local-provider-start` ran the real thing, but was GitHub-only: it hardcoded the token-requiring overlay, so the credential-free provider configuration that already existed could not be reached by hand at all. `PROVIDER=local` failed at `docker compose up` on a missing `GITHUB_TOKEN`, and would have merged nothing past that, since the overlay it needs bind-mounts a repository the other one has never heard of.
Those two are not variants of one thing. Faking every edge and merging into a real repository answer different questions, and calling both "local" is what made them hard to keep apart.
Three further confusions came out of reading the result back:
- **Two stack families did the same thing.** Once `fake` was a first-class provider, `local-submitqueue-start` and `local-provider-start` were the same run under *different compose project names* — which had already produced a wrong instruction, since `make local-submitqueue-ps` inspects project `submitqueue` and reports "not running" for a stack started as `submitqueue-provider`.
- **One document mixed two audiences.** `PROVIDER-E2E.md` opened with a tier table pairing bazel test targets with hand-run demo commands, as though `make e2e-test` and "open a pull request and watch it land" were rungs of one ladder. Automated suites belong to testing, which `TESTING.md` already documents.
- **`demo-pr` was named for something two of the three modes do not have.** A fake change is a URI and a git change is a branch; neither is a pull request.
### What?
`PROVIDER` now names three modes, and is the only thing that changes between them:
| `PROVIDER` | A change is | Landing it | Needs |
|---|---|---|---|
| `fake` (default) | a URI, and nothing else | reports success without touching a repository | nothing |
| `git` | a branch in a bare repository on disk | a real fetch, cherry-pick and push | nothing |
| `github` | a real pull request | a real push to a real repository | a repository and a token |
**Provider directories.** `demo/provider/local/` becomes `demo/provider/git/` — both old names were "local" — and gains a `demo-queue` entry so the default `QUEUE` works there. It needs its own `checkoutPath`: Runway refuses two queues that share a checkout with differing merger configuration, and `demo-queue` squash-rebases where `e2e-git-queue` rebases. A new `demo/provider/fake/` spells out the all-fake mode rather than leaving it implied by absent configuration. `e2e-git-queue` is untouched, and `make e2e-git-test` is the gate that proves it.
**One stack target.** `local-submitqueue-start` takes `PROVIDER` and layers that mode's overlay; `local-provider-start`, `-stop` and `-clean` are gone. One compose project again, so `ps`, `logs`, `stop` and `clean` all describe whatever is running. Which overlay a mode needs is a Makefile map, since it is not something the two config files can express — `github` requires a credential, `git` requires the sandbox mounted, `fake` neither — and a new `docker-compose.fake.yml` covers the third.
**One demo document.** `PROVIDER-E2E.md` is merged into `QUICKSTART.md`, which walks all three rungs in order and carries what only that document had: the token permission table, the GitHub configuration, why a landed pull request shows as *Merged*, wiring real GitHub Actions, and the GitHub-specific failure modes. Its provider-neutral operational material — `land-list` / `land-watch`, `SQ_TOKEN`, service logs, `QUEUE_LOG_LEVEL` — moves with it, into the document people read first. No test target is mentioned in any of it.
**`make demo-requests`** (was `demo-pr`, in `service/submitqueue/demo/requests`) works without GitHub. `createOne` was the only function that touched the provider; it now describes a change and hands it to a `changeSource`. Three implement it: `fakeSource` (no I/O at all — mints a reproducible `git://` URI), `gitSource` (pushes real branches with the pinned `@git//:git`, serializing its commands because one working tree cannot take concurrent checkouts), and `githubSource` (the REST client, extracted unchanged). `GITHUB_TOKEN` is read only when it is needed.
**Folder overlap, made real.** A change writes all of its files into one folder under `demo/`, and states the paths it touched on its change URI (`sq-files=`, beside the existing `sq-fake=`) for the fake change provider to report back. All three profiles then use `pathoverlap` by directory, so a run shows both halves of the queue's behaviour: changes sharing a folder are batched in order and speculate on each other, changes in different folders go out beside each other.
How many folders there are is `FOLDERS`, defaulting to a number between five and ten picked per run and printed with the run's opening line. It is the dial on what a run demonstrates: `FOLDERS=1` serializes everything, a number well above `COUNT` keeps everything apart. The previous layout could do neither — it hashed each file's own name into two nested levels of 256, which put a twelve-file run's odds of any collision at roughly one in a thousand and scattered a single change across as many directories as it had files. What changed is the bucket count and what gets hashed, not the naming.
**Matching stack lifecycle targets.** `local-submitqueue-stop` did not exist — the only way to stop the stack was `local-stop`, which stops every domain's. It exists now, and `local-submitqueue-clean` removes the git sandbox and the overlay's volumes rather than only the base file's, which is what the documentation already claimed it did.
**`tool/gitsandbox`** provisions the bare repository `PROVIDER=git` merges into, idempotently, so a restart keeps whatever landed. **`platform/gitexec`** locates git and strips the ambient environment, so a developer's hooks or signing key cannot fail a demo.
Three fixes fell out of getting this to work at all:
- Runway's checkout directory moves to a named volume unless a path is given. It is where git clones, cherry-picks and commits, and on macOS a freshly written loose object read back over a bind mount can report `loose object … is corrupt` — which failed the first land against every new stack. The E2E still passes a path and still gets a bind mount.
- The Runway image now creates `/var/runway/checkouts`, so a named volume mounted there starts with a mode the service can write. Without it the service failed at boot with `mkdir: permission denied` whenever it ran as a non-root user.
- Change URIs are storage keys capped at 255 bytes, so the file marker names one path per directory rather than every file — for a directory-keyed analyzer that is the same set of keys — and drops paths that would not fit rather than truncating one into a different directory.
## Test Plan
Every command in the rewritten quickstart was run by hand against a live stack, and the output quoted in it is what it printed.
- ✅ `make local-submitqueue-start` (defaults to `fake`) then `make demo-requests COUNT=6` — all six landed
- ✅ inspected `batch_dependent` across three runs. `FOLDERS=1` produced a full chain — every batch depending on every later one. `FOLDERS=50` produced four batches with no dependents at all. The default picked eight folders, put two of six changes in the same one, and recorded a dependency between exactly those two and nothing else
- ✅ `make local-submitqueue-stop` leaves the sandbox and its data; `make local-submitqueue-clean` removes the sandbox, the volumes and the images, verified by looking for leftovers afterwards
- ✅ `PROVIDER=git make local-submitqueue-start` from a clean slate, then `PROVIDER=git make demo-requests` — real branches and multi-file commits, verified with `git -C /tmp/sq-sandbox/sandbox.git log --oneline main`
- ✅ `PROVIDER=git make demo-requests STACKED=true` — three changes, exactly one more entry in the target's reflog, so the stack landed atomically
- ✅ first land against a brand-new stack, repeatedly, which is the case the named volume fixes
- ✅ the 255-byte limit is a real one this hit first: an eight-file change was rejected with `change URI exceeds 255 bytes` before the marker was budgeted. Covered now by a unit test that builds a URI from `changeFilePath` output and asserts both the length and that it still parses
- ✅ `make test` (105 targets), `make lint`, `make check-gazelle`, `make check-tidy`
- ✅ `make e2e-test` and `make e2e-git-test`, after the rename and again after the compose change
`PROVIDER=github` is unchanged in behaviour but needs a token, so it has not been re-run; the GitHub source is the previous code path moved behind the interface.
## Note for reviewers
`make local-provider-start` is gone — use `make local-submitqueue-start`, which with no arguments now means `fake` where the old provider target meant GitHub. `make local-stop` still stops every domain's stack; `make local-submitqueue-stop` is the new one that stops only this.1 parent c34c87f commit 04445d2
39 files changed
Lines changed: 2709 additions & 974 deletions
File tree
- doc/howto
- platform
- fakemarker
- gitexec
- service
- runway/server
- submitqueue
- demo
- provider
- fake
- github
- git
- local
- pr
- requests
- gateway/server
- submitqueue/extension/changeprovider/fake
- test/e2e/submitqueue
- tool/gitsandbox
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
9 | | - | |
10 | 9 | | |
11 | 10 | | |
12 | 11 | | |
| |||
46 | 45 | | |
47 | 46 | | |
48 | 47 | | |
49 | | - | |
50 | | - | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
51 | 51 | | |
52 | | - | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
53 | 57 | | |
54 | 58 | | |
55 | | - | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
56 | 76 | | |
57 | 77 | | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
58 | 82 | | |
59 | 83 | | |
60 | 84 | | |
| |||
77 | 101 | | |
78 | 102 | | |
79 | 103 | | |
80 | | - | |
| 104 | + | |
81 | 105 | | |
82 | 106 | | |
83 | 107 | | |
| |||
172 | 196 | | |
173 | 197 | | |
174 | 198 | | |
175 | | - | |
176 | | - | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
177 | 202 | | |
| 203 | + | |
178 | 204 | | |
| 205 | + | |
179 | 206 | | |
180 | 207 | | |
181 | 208 | | |
| |||
277 | 304 | | |
278 | 305 | | |
279 | 306 | | |
280 | | - | |
| 307 | + | |
281 | 308 | | |
282 | | - | |
283 | | - | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
284 | 314 | | |
285 | 315 | | |
286 | 316 | | |
| |||
301 | 331 | | |
302 | 332 | | |
303 | 333 | | |
304 | | - | |
305 | | - | |
306 | | - | |
307 | | - | |
308 | | - | |
309 | | - | |
310 | | - | |
311 | | - | |
312 | | - | |
313 | | - | |
314 | | - | |
315 | | - | |
316 | | - | |
317 | | - | |
318 | | - | |
319 | | - | |
320 | | - | |
321 | | - | |
322 | | - | |
323 | | - | |
324 | 334 | | |
325 | 335 | | |
326 | 336 | | |
| |||
429 | 439 | | |
430 | 440 | | |
431 | 441 | | |
432 | | - | |
433 | | - | |
434 | | - | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
435 | 462 | | |
436 | 463 | | |
437 | 464 | | |
438 | | - | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
439 | 471 | | |
440 | | - | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
441 | 482 | | |
442 | | - | |
| 483 | + | |
443 | 484 | | |
444 | 485 | | |
445 | 486 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
18 | | - | |
| 18 | + | |
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| |||
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
28 | | - | |
29 | | - | |
30 | | - | |
31 | | - | |
| 28 | + | |
| 29 | + | |
32 | 30 | | |
33 | 31 | | |
34 | | - | |
| 32 | + | |
35 | 33 | | |
36 | 34 | | |
37 | | - | |
| 35 | + | |
38 | 36 | | |
39 | | - | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
40 | 46 | | |
41 | 47 | | |
42 | 48 | | |
43 | 49 | | |
44 | 50 | | |
45 | | - | |
| 51 | + | |
46 | 52 | | |
47 | 53 | | |
48 | 54 | | |
49 | | - | |
50 | 55 | | |
51 | 56 | | |
52 | 57 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
64 | 64 | | |
65 | 65 | | |
66 | 66 | | |
67 | | - | |
68 | | - | |
69 | | - | |
70 | | - | |
| 67 | + | |
| 68 | + | |
71 | 69 | | |
72 | 70 | | |
73 | 71 | | |
74 | 72 | | |
75 | 73 | | |
76 | | - | |
| 74 | + | |
77 | 75 | | |
78 | 76 | | |
79 | 77 | | |
| |||
0 commit comments