feat(example): create, enqueue, and watch demo pull requests (#569)

## Summary
### Why?

Exercising the demo meant opening pull requests by hand and pasting
their URLs into a land command — slow enough to discourage running it,
and it made the interesting case impossible to reach in practice. A
queue that only ever holds one request in flight never batches, never
analyzes a conflict against another batch, and never speculates.

### What?

`make demo-pr` creates pull requests in the scratch repository, enqueues
them, and watches them settle.

Each pull request is enqueued the moment it is created, so the queue is
already working on the first while the last is still being opened.
Nothing is awaited until everything is in; the wait happens once,
against all of them. That overlap is what puts requests in flight
against each other, which is the whole point of running more than one.

`STACKED=true` is the exception, and necessarily so: one request carries
the whole chain, so it can only be submitted once every change in it
exists. That is the atomic-stack path, where the set reaches the target
in a single push.

Progress is a table keyed on the gateway's status API, redrawn as
requests move, so it is visible which stage each one is in rather than
only whether it finished. On a terminal it redraws in place; piped to a
file it prints a fresh table whenever something changes, so logs stay
readable.

It lives beside the configuration it drives, under
`service/submitqueue/example/`, and reaches GitHub through the REST API
rather than a clone — so it needs no checkout and no git binary, only
`GITHUB_TOKEN`, the credential the stack already uses.

## Test Plan
✅ `bazel build //service/submitqueue/example/pr`; `make help` lists
`demo-pr`.

Running it end to end needs a scratch repository and a token, so it is
exercised the same way the rest of the provider path is — manually. See
`doc/howto/PROVIDER-E2E.md`.


## Also in this PR

- **feat(demo): live table with the full state trail per request**
- **feat(messagequeue): level the queue's own logs separately**
- **docs(demo): correct the sample trail to states the pipeline emits**
- **feat(demo): spread each change over a sharded file tree**

<sub>Each commit above carries its own rationale and test plan in its
message.</sub>

## Issues

Author: behinddwalls

Makefile

@@ -52,7 +52,13 @@ export REPO_ROOT := $(shell pwd)
 PROVIDER ?= github
 export SQ_PROVIDER_CONFIG_DIR ?= $(REPO_ROOT)/service/submitqueue/demo/provider/$(PROVIDER)
 
-# Defaults for `make land` against the provider demo stack.
+# Defaults for `make land` / `make demo-pr` against the provider demo stack.
+DEMO_REPO ?= behinddwalls/sq-demo
+COUNT ?= 3
+FILES ?= 3
+STACKED ?= false
+LAND ?= true
+WATCH ?= true
 QUEUE ?= demo-queue
 STRATEGY ?= SQUASH_REBASE
 GATEWAY_ADDR ?= localhost:8081
@@ -147,6 +153,17 @@ clean-proto: ## Clean generated proto files
 	@rm -f $(foreach p,$(PROTO_PACKAGES),$(p)/protopb/*.pb.go $(p)/protopb/*.pb.yarpc.go)
 	@echo "Proto clean complete!"
 
+demo-pr: ## Create N PRs in the demo repo, enqueue each as it is created, and watch (COUNT=3 FILES=3; needs GITHUB_TOKEN)
+	@$(BAZEL) run //service/submitqueue/demo/pr -- \
+		-repo $(DEMO_REPO) \
+		-count $(COUNT) \
+		-files $(FILES) \
+		-stacked=$(STACKED) \
+		-gateway $(GATEWAY_ADDR) \
+		-queue $(QUEUE) \
+		-strategy $(STRATEGY) \
+		-land=$(LAND) -watch=$(WATCH)
+
 deps: tidy-go ## Download and tidy Go dependencies
 	@echo "Dependencies installed!"
 

doc/howto/PROVIDER-E2E.md

@@ -25,7 +25,7 @@ For a **fine-grained** token, grant these repository permissions. Each is here b
 | Metadata | Read | mandatory on every fine-grained token; GitHub adds it for you |
 | Contents | Read and write | the git merger — clone, fetch, push to the target branch, and force-move each landed change's head branch |
 | Pull requests | Read | the change provider reads pull request metadata, and `land -pr` reads the head commit |
-| Pull requests | Read **and write** | only for `make demo-prs`, which opens pull requests |
+| Pull requests | Read **and write** | only for `make demo-pr`, which opens pull requests |
 | Actions | Read and write | only if you switch the build runner to GitHub Actions — dispatch a run, poll it, cancel it |
 
 A **classic** PAT needs `repo`, plus `workflow` if you use the GitHub Actions build runner.
@@ -89,6 +89,44 @@ make land PRS="https://github.com/<you>/<repo>/pull/1 \
 
 The order of `PRS` is the stack order. All three land as **one push** to `main` — there is no window where a reader sees the stack half-applied — and all three show as merged. Tier 2 asserts the single-push property mechanically, by counting ref updates in the target's reflog.
 
+## Simulating traffic
+
+Opening pull requests by hand gets old fast. `demo-pr` creates them, enqueues them, and shows you where each one is:
+
+```bash
+make demo-pr                      # 3 independent PRs, each enqueued as it is created
+make demo-pr COUNT=8              # more traffic
+make demo-pr FILES=8              # wider changes, more files per PR
+make demo-pr STACKED=true         # one stack, enqueued as a single request
+make demo-pr LAND=false           # create only, print the land command
+```
+
+Each pull request is enqueued the moment it exists, so the queue is already working on the first while the last is still being opened. That overlap is the point: a queue holding one request at a time never batches, never analyzes a conflict against another batch, and never speculates. Nothing is awaited until every request is in.
+
+The table is there from the start — one row per land request, drawn before the first pull request exists and filled in as the run proceeds. Whatever is happening right now is a single line underneath it, so creating and enqueuing does not scroll the table away:
+
+```
+  REQUEST        CHANGES  ELAPSED  STAGE
+  ─────────────  ───────  ───────  ─────────────────────────────────────────────────
+  demo-queue/12  #31          34s  accepted → started → validated → batched → landed
+  demo-queue/13  #32          31s  accepted → started → validated → batched
+  demo-queue/14  #33          28s  accepted → started
+
+  ▸ 1 of 3 settled
+```
+
+Each row shows the states its request passed through, not just the one it is in. That comes from the gateway's history API rather than from sampling the current status, so a transition between two polls is not missed. `CHANGES` links to the pull request: on a terminal `#31` is clickable, and in a redirected run it is written out as a full URL instead. `ELAPSED` runs from the moment the gateway accepted the request and stops when it settles, so a finished row keeps the time it took rather than counting on.
+
+The trail is only as detailed as what the pipeline reports, which today is `accepted`, `started`, `validated`, `batched` and then a terminal `landed`, `error` or `cancelled`. The finer-grained statuses the API defines — `speculating`, `building`, `landing` and the rest — are never published, so a request sits on `batched` for the whole of its active life even while its batch is speculating and building. Do not read that as the request being stuck.
+
+`STACKED=true` is the exception to the overlap: one request carries the whole chain, so it can only go in once every pull request in it exists. That is the atomic-stack path — the whole set reaches `main` in a single push, and the table shows it as the single row it is.
+
+It talks to GitHub over the REST API with the same `GITHUB_TOKEN`, so it needs no clone and no git binary. Each run tags its branches with a timestamp so repeated runs do not collide, and every file a change writes is at a path no other change uses, so independent changes do not conflict by accident.
+
+A change touches several files rather than one, each committed separately, so it arrives as a multi-file, multi-commit pull request — closer to a real change, and enough to exercise replaying a range of commits. `FILES` sets the floor (default 3); the actual count varies a little above it, derived from the run tag so replaying a tag reproduces the same run. Paths are sharded into two levels of hex buckets under `demo/` (`demo/c2/91/<tag>-<change>-<file>.txt`), which keeps the tree from degenerating into one enormous directory as runs accumulate.
+
+The command exits non-zero if any request settles anywhere other than `landed`, so it works in a script. Piped to a file it prints a fresh table whenever a request moves — and not when only the clock did — instead of redrawing in place.
+
 ## Watching it work
 
 ```bash
@@ -101,6 +139,14 @@ Runway logs each merge and each head-branch move:
 moved change head branch to its landed commit  {"change": "you/repo#1", "branch": "refs/heads/feature-a", ...}
 ```
 
+The message queue logs a line per message published, fetched, leased and acked, which at debug level buries everything else a service says. It is levelled separately from the rest of the service, at info by default. To follow the queue itself — chasing a message that never arrived, or a partition that never got leased — turn it back up for the services you care about:
+
+```bash
+QUEUE_LOG_LEVEL=debug make local-submitqueue-start
+```
+
+`QUEUE_LOG_LEVEL` takes any zap level name. It can only raise the queue's level above the one the service logger was built with, never lower it, so it cannot be used to make a quiet service verbose.
+
 ## When it does not work
 
 **The push is rejected on the first try.** Branch protection on `main` — required status checks, or a linear-history or no-force-push rule — applies to the merger like anyone else. Either relax it on the scratch repo or add the token's identity to the bypass list.

platform/extension/messagequeue/mysql/BUILD.bazel

@@ -26,6 +26,7 @@ go_library(
         "@com_github_uber_go_tally//:go_default_library",
         "@org_uber_go_mock//gomock:go_default_library",
         "@org_uber_go_zap//:go_default_library",
+        "@org_uber_go_zap//zapcore:go_default_library",
     ],
 )
 

platform/extension/messagequeue/mysql/sql.go

@@ -22,6 +22,7 @@ import (
 
 	"github.com/uber-go/tally"
 	"go.uber.org/zap"
+	"go.uber.org/zap/zapcore"
 
 	extqueue "github.com/uber/submitqueue/platform/extension/messagequeue"
 )
@@ -40,6 +41,16 @@ type Params struct {
 	// Logger for debugging and observability (required)
 	Logger *zap.Logger
 
+	// LogLevel is the minimum level for the queue's own logs, as a zap level
+	// name ("debug", "info", ...). Empty selects info.
+	//
+	// The queue logs a line per message published, fetched, leased and acked,
+	// which at debug buries everything else a service says. Levelling it here
+	// rather than at the service logger keeps the rest of that service's debug
+	// output intact. The level can only be raised above the one the supplied
+	// logger was built with, never lowered.
+	LogLevel string
+
 	// MetricsScope for metrics collection (required)
 	MetricsScope tally.Scope
 
@@ -55,8 +66,17 @@ func NewQueue(params Params) (extqueue.Queue, error) {
 		return nil, fmt.Errorf("failed to ping database: %w", err)
 	}
 
-	logger := params.Logger.Sugar().Named("queue_mysql")
-	logger.Infow("created SQL queue")
+	level := zapcore.InfoLevel
+	if params.LogLevel != "" {
+		parsed, err := zapcore.ParseLevel(params.LogLevel)
+		if err != nil {
+			return nil, fmt.Errorf("invalid queue log level %q: %w", params.LogLevel, err)
+		}
+		level = parsed
+	}
+
+	logger := params.Logger.WithOptions(zap.IncreaseLevel(level)).Sugar().Named("queue_mysql")
+	logger.Infow("created SQL queue", "log_level", level.String())
 
 	// Create stores
 	messageStore := newMessageStore(params.DB, logger, params.MetricsScope)

platform/extension/messagequeue/mysql/sql_test.go

@@ -70,6 +70,44 @@ func TestNewQueue(t *testing.T) {
 
 		require.NoError(t, mock.ExpectationsWereMet())
 	})
+	t.Run("accepts a log level", func(t *testing.T) {
+		db, mock, err := sqlmock.New(sqlmock.MonitorPingsOption(true))
+		require.NoError(t, err)
+		defer db.Close()
+
+		mock.ExpectPing()
+
+		q, err := NewQueue(Params{
+			DB:           db,
+			Logger:       zaptest.NewLogger(t),
+			LogLevel:     "debug",
+			MetricsScope: tally.NewTestScope("test", nil),
+		})
+
+		require.NoError(t, err)
+		require.NotNil(t, q)
+		assert.NoError(t, q.Close())
+
+		require.NoError(t, mock.ExpectationsWereMet())
+	})
+
+	t.Run("error when the log level is not a level", func(t *testing.T) {
+		db, mock, err := sqlmock.New(sqlmock.MonitorPingsOption(true))
+		require.NoError(t, err)
+		defer db.Close()
+
+		mock.ExpectPing()
+
+		q, err := NewQueue(Params{
+			DB:           db,
+			Logger:       zaptest.NewLogger(t),
+			LogLevel:     "loud",
+			MetricsScope: tally.NewTestScope("test", nil),
+		})
+
+		require.Error(t, err)
+		assert.Nil(t, q)
+	})
 }
 
 func TestQueue_Publisher(t *testing.T) {

service/runway/server/docker-compose.yml

@@ -49,6 +49,9 @@ services:
       - MERGER=${SQ_RUNWAY_MERGER:-}
       # Queue infrastructure connection
       - QUEUE_MYSQL_DSN=root:root@tcp(mysql-queue:3306)/submitqueue?parseTime=true
+      # Level for the queue's own logs; info by default so its per-message
+      # chatter does not bury the rest of the service at debug.
+      - QUEUE_LOG_LEVEL=${QUEUE_LOG_LEVEL:-}
       - HOSTNAME=runway-dev
     depends_on:
       mysql-queue:

service/runway/server/main.go

@@ -138,6 +138,7 @@ func run() error {
 	mysqlQueue, err := queueMySQL.NewQueue(queueMySQL.Params{
 		DB:           queueDB,
 		Logger:       logger,
+		LogLevel:     os.Getenv("QUEUE_LOG_LEVEL"),
 		MetricsScope: scope.SubScope("queue"),
 	})
 	if err != nil {

service/stovepipe/docker-compose.yml

@@ -67,6 +67,9 @@ services:
       - PORT=:8080
       - STORAGE_MYSQL_DSN=root:root@tcp(mysql-app:3306)/submitqueue?parseTime=true
       - QUEUE_MYSQL_DSN=root:root@tcp(mysql-queue:3306)/submitqueue?parseTime=true
+      # Level for the queue's own logs; info by default so its per-message
+      # chatter does not bury the rest of the service at debug.
+      - QUEUE_LOG_LEVEL=${QUEUE_LOG_LEVEL:-}
       - HOSTNAME=stovepipe-dev
     depends_on:
       mysql-app:

service/stovepipe/server/main.go

@@ -241,6 +241,7 @@ func run() error {
 	mysqlQueue, err := queueMySQL.NewQueue(queueMySQL.Params{
 		DB:           queueDB,
 		Logger:       logger,
+		LogLevel:     os.Getenv("QUEUE_LOG_LEVEL"),
 		MetricsScope: scope.SubScope("queue"),
 	})
 	if err != nil {

service/submitqueue/demo/pr/BUILD.bazel

@@ -0,0 +1,35 @@
+load("@rules_go//go:def.bzl", "go_binary", "go_library", "go_test")
+
+go_library(
+    name = "go_default_library",
+    srcs = ["main.go"],
+    importpath = "github.com/uber/submitqueue/service/submitqueue/demo/pr",
+    visibility = ["//visibility:private"],
+    deps = [
+        "//api/base/change/protopb:go_default_library",
+        "//api/base/mergestrategy/protopb:go_default_library",
+        "//api/submitqueue/gateway/protopb:go_default_library",
+        "//platform/base/change/github:go_default_library",
+        "//submitqueue/entity:go_default_library",
+        "@org_golang_google_grpc//:go_default_library",
+        "@org_golang_google_grpc//credentials/insecure:go_default_library",
+    ],
+)
+
+go_binary(
+    name = "pr",
+    embed = [":go_default_library"],
+    visibility = ["//visibility:public"],
+)
+
+go_test(
+    name = "go_default_test",
+    srcs = ["main_test.go"],
+    embed = [":go_default_library"],
+    deps = [
+        "//api/submitqueue/gateway/protopb:go_default_library",
+        "@com_github_stretchr_testify//assert:go_default_library",
+        "@com_github_stretchr_testify//require:go_default_library",
+        "@org_golang_google_grpc//:go_default_library",
+    ],
+)