Commit 5bb0876
authored
feat(demo): one quickstart, three provider modes (uber#595)
## Summary
### Why?
The repository had two divergent local paths, and the quickstart
documented the wrong one.
`make local-submitqueue-start` brought up an all-fake stack — fake
change provider, fake CI, **noop merger** — and
`doc/howto/QUICKSTART.md` (uber#591) described that. It reached `landed` in
seconds with no credentials, but nothing was ever pushed, so it proved
the pipeline's choreography and nothing about merging. `make
local-provider-start` ran the real thing, but was GitHub-only: it
hardcoded the token-requiring overlay, so the credential-free provider
configuration that already existed could not be reached by hand at all.
`PROVIDER=local` failed at `docker compose up` on a missing
`GITHUB_TOKEN`, and would have merged nothing past that, since the
overlay it needs bind-mounts a repository the other one has never heard
of.
Those two are not variants of one thing. Faking every edge and merging
into a real repository answer different questions, and calling both
"local" is what made them hard to keep apart.
Three further confusions came out of reading the result back:
- **Two stack families did the same thing.** Once `fake` was a
first-class provider, `local-submitqueue-start` and
`local-provider-start` were the same run under *different compose
project names* — which had already produced a wrong instruction, since
`make local-submitqueue-ps` inspects project `submitqueue` and reports
"not running" for a stack started as `submitqueue-provider`.
- **One document mixed two audiences.** `PROVIDER-E2E.md` opened with a
tier table pairing bazel test targets with hand-run demo commands, as
though `make e2e-test` and "open a pull request and watch it land" were
rungs of one ladder. Automated suites belong to testing, which
`TESTING.md` already documents.
- **`demo-pr` was named for something two of the three modes do not
have.** A fake change is a URI and a git change is a branch; neither is
a pull request.
### What?
`PROVIDER` now names three modes, and is the only thing that changes
between them:
| `PROVIDER` | A change is | Landing it | Needs |
|---|---|---|---|
| `fake` (default) | a URI, and nothing else | reports success without
touching a repository | nothing |
| `git` | a branch in a bare repository on disk | a real fetch,
cherry-pick and push | nothing |
| `github` | a real pull request | a real push to a real repository | a
repository and a token |
**Provider directories.** `demo/provider/local/` becomes
`demo/provider/git/` — both old names were "local" — and gains a
`demo-queue` entry so the default `QUEUE` works there. It needs its own
`checkoutPath`: Runway refuses two queues that share a checkout with
differing merger configuration, and `demo-queue` squash-rebases where
`e2e-git-queue` rebases. A new `demo/provider/fake/` spells out the
all-fake mode rather than leaving it implied by absent configuration.
`e2e-git-queue` is untouched, and `make e2e-git-test` is the gate that
proves it.
**One stack target.** `local-submitqueue-start` takes `PROVIDER` and
layers that mode's overlay; `local-provider-start`, `-stop` and `-clean`
are gone. One compose project again, so `ps`, `logs`, `stop` and `clean`
all describe whatever is running. Which overlay a mode needs is a
Makefile map, since it is not something the two config files can express
— `github` requires a credential, `git` requires the sandbox mounted,
`fake` neither — and a new `docker-compose.fake.yml` covers the third.
**One demo document.** `PROVIDER-E2E.md` is merged into `QUICKSTART.md`,
which walks all three rungs in order and carries what only that document
had: the token permission table, the GitHub configuration, why a landed
pull request shows as *Merged*, wiring real GitHub Actions, and the
GitHub-specific failure modes. Its provider-neutral operational material
— `land-list` / `land-watch`, `SQ_TOKEN`, service logs,
`QUEUE_LOG_LEVEL` — moves with it, into the document people read first.
No test target is mentioned in any of it.
**`make demo-requests`** (was `demo-pr`, in
`service/submitqueue/demo/requests`) works without GitHub. `createOne`
was the only function that touched the provider; it now describes a
change and hands it to a `changeSource`. Three implement it:
`fakeSource` (no I/O at all — mints a reproducible `git://` URI),
`gitSource` (pushes real branches with the pinned `@git//:git`,
serializing its commands because one working tree cannot take concurrent
checkouts), and `githubSource` (the REST client, extracted unchanged).
`GITHUB_TOKEN` is read only when it is needed.
**Folder overlap, made real.** A change writes all of its files into one
folder under `demo/`, and states the paths it touched on its change URI
(`sq-files=`, beside the existing `sq-fake=`) for the fake change
provider to report back. All three profiles then use `pathoverlap` by
directory, so a run shows both halves of the queue's behaviour: changes
sharing a folder are batched in order and speculate on each other,
changes in different folders go out beside each other.
How many folders there are is `FOLDERS`, defaulting to a number between
five and ten picked per run and printed with the run's opening line. It
is the dial on what a run demonstrates: `FOLDERS=1` serializes
everything, a number well above `COUNT` keeps everything apart. The
previous layout could do neither — it hashed each file's own name into
two nested levels of 256, which put a twelve-file run's odds of any
collision at roughly one in a thousand and scattered a single change
across as many directories as it had files. What changed is the bucket
count and what gets hashed, not the naming.
**Matching stack lifecycle targets.** `local-submitqueue-stop` did not
exist — the only way to stop the stack was `local-stop`, which stops
every domain's. It exists now, and `local-submitqueue-clean` removes the
git sandbox and the overlay's volumes rather than only the base file's,
which is what the documentation already claimed it did.
**`tool/gitsandbox`** provisions the bare repository `PROVIDER=git`
merges into, idempotently, so a restart keeps whatever landed.
**`platform/gitexec`** locates git and strips the ambient environment,
so a developer's hooks or signing key cannot fail a demo.
Three fixes fell out of getting this to work at all:
- Runway's checkout directory moves to a named volume unless a path is
given. It is where git clones, cherry-picks and commits, and on macOS a
freshly written loose object read back over a bind mount can report
`loose object … is corrupt` — which failed the first land against every
new stack. The E2E still passes a path and still gets a bind mount.
- The Runway image now creates `/var/runway/checkouts`, so a named
volume mounted there starts with a mode the service can write. Without
it the service failed at boot with `mkdir: permission denied` whenever
it ran as a non-root user.
- Change URIs are storage keys capped at 255 bytes, so the file marker
names one path per directory rather than every file — for a
directory-keyed analyzer that is the same set of keys — and drops paths
that would not fit rather than truncating one into a different
directory.
## Test Plan
Every command in the rewritten quickstart was run by hand against a live
stack, and the output quoted in it is what it printed.
- ✅ `make local-submitqueue-start` (defaults to `fake`) then `make
demo-requests COUNT=6` — all six landed
- ✅ inspected `batch_dependent` across three runs. `FOLDERS=1` produced
a full chain — every batch depending on every later one. `FOLDERS=50`
produced four batches with no dependents at all. The default picked
eight folders, put two of six changes in the same one, and recorded a
dependency between exactly those two and nothing else
- ✅ `make local-submitqueue-stop` leaves the sandbox and its data; `make
local-submitqueue-clean` removes the sandbox, the volumes and the
images, verified by looking for leftovers afterwards
- ✅ `PROVIDER=git make local-submitqueue-start` from a clean slate, then
`PROVIDER=git make demo-requests` — real branches and multi-file
commits, verified with `git -C /tmp/sq-sandbox/sandbox.git log --oneline
main`
- ✅ `PROVIDER=git make demo-requests STACKED=true` — three changes,
exactly one more entry in the target's reflog, so the stack landed
atomically
- ✅ first land against a brand-new stack, repeatedly, which is the case
the named volume fixes
- ✅ the 255-byte limit is a real one this hit first: an eight-file
change was rejected with `change URI exceeds 255 bytes` before the
marker was budgeted. Covered now by a unit test that builds a URI from
`changeFilePath` output and asserts both the length and that it still
parses
- ✅ `make test` (105 targets), `make lint`, `make check-gazelle`, `make
check-tidy`
- ✅ `make e2e-test` and `make e2e-git-test`, after the rename and again
after the compose change
`PROVIDER=github` is unchanged in behaviour but needs a token, so it has
not been re-run; the GitHub source is the previous code path moved
behind the interface.
## Note for reviewers
`make local-provider-start` is gone — use `make
local-submitqueue-start`, which with no arguments now means `fake` where
the old provider target meant GitHub. `make local-stop` still stops
every domain's stack; `make local-submitqueue-stop` is the new one that
stops only this.
## Issues1 parent f1ff437 commit 5bb0876
39 files changed
Lines changed: 2709 additions & 974 deletions
File tree
- doc/howto
- platform
- fakemarker
- gitexec
- service
- runway/server
- submitqueue
- demo
- provider
- fake
- github
- git
- local
- pr
- requests
- gateway/server
- submitqueue/extension/changeprovider/fake
- test/e2e/submitqueue
- tool/gitsandbox
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
9 | | - | |
10 | 9 | | |
11 | 10 | | |
12 | 11 | | |
| |||
46 | 45 | | |
47 | 46 | | |
48 | 47 | | |
49 | | - | |
50 | | - | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
51 | 51 | | |
52 | | - | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
53 | 57 | | |
54 | 58 | | |
55 | | - | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
56 | 76 | | |
57 | 77 | | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
58 | 82 | | |
59 | 83 | | |
60 | 84 | | |
| |||
77 | 101 | | |
78 | 102 | | |
79 | 103 | | |
80 | | - | |
| 104 | + | |
81 | 105 | | |
82 | 106 | | |
83 | 107 | | |
| |||
172 | 196 | | |
173 | 197 | | |
174 | 198 | | |
175 | | - | |
176 | | - | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
177 | 202 | | |
| 203 | + | |
178 | 204 | | |
| 205 | + | |
179 | 206 | | |
180 | 207 | | |
181 | 208 | | |
| |||
277 | 304 | | |
278 | 305 | | |
279 | 306 | | |
280 | | - | |
| 307 | + | |
281 | 308 | | |
282 | | - | |
283 | | - | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
284 | 314 | | |
285 | 315 | | |
286 | 316 | | |
| |||
301 | 331 | | |
302 | 332 | | |
303 | 333 | | |
304 | | - | |
305 | | - | |
306 | | - | |
307 | | - | |
308 | | - | |
309 | | - | |
310 | | - | |
311 | | - | |
312 | | - | |
313 | | - | |
314 | | - | |
315 | | - | |
316 | | - | |
317 | | - | |
318 | | - | |
319 | | - | |
320 | | - | |
321 | | - | |
322 | | - | |
323 | | - | |
324 | 334 | | |
325 | 335 | | |
326 | 336 | | |
| |||
429 | 439 | | |
430 | 440 | | |
431 | 441 | | |
432 | | - | |
433 | | - | |
434 | | - | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
435 | 462 | | |
436 | 463 | | |
437 | 464 | | |
438 | | - | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
439 | 471 | | |
440 | | - | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
441 | 482 | | |
442 | | - | |
| 483 | + | |
443 | 484 | | |
444 | 485 | | |
445 | 486 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
18 | | - | |
| 18 | + | |
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| |||
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
28 | | - | |
29 | | - | |
30 | | - | |
31 | | - | |
| 28 | + | |
| 29 | + | |
32 | 30 | | |
33 | 31 | | |
34 | | - | |
| 32 | + | |
35 | 33 | | |
36 | 34 | | |
37 | | - | |
| 35 | + | |
38 | 36 | | |
39 | | - | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
40 | 46 | | |
41 | 47 | | |
42 | 48 | | |
43 | 49 | | |
44 | 50 | | |
45 | | - | |
| 51 | + | |
46 | 52 | | |
47 | 53 | | |
48 | 54 | | |
49 | | - | |
50 | 55 | | |
51 | 56 | | |
52 | 57 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
64 | 64 | | |
65 | 65 | | |
66 | 66 | | |
67 | | - | |
68 | | - | |
69 | | - | |
70 | | - | |
| 67 | + | |
| 68 | + | |
71 | 69 | | |
72 | 70 | | |
73 | 71 | | |
74 | 72 | | |
75 | 73 | | |
76 | | - | |
| 74 | + | |
77 | 75 | | |
78 | 76 | | |
79 | 77 | | |
| |||
0 commit comments