feat(demo): spread each change over a sharded file tree

## Summary

### Why?

Every demo change wrote exactly one file, and every one of those files went into a single flat `demo/` directory. Two things follow from that, and neither is what the demo wants to show. The directory grows without bound as runs accumulate, until it is the kind of directory nobody wants in a repository. And a pull request that adds one line to one file is not much of a pull request — it exercises none of the handling a real change needs, and reads as a toy in a walkthrough whose whole purpose is to look like the real thing.

### What?

A change now writes several files, each at a path sharded into two levels of hex buckets under `demo/` — `demo/c2/91/<tag>-<change>-<file>.txt`. The buckets are the leading bytes of the leaf name's SHA-256, so the spread is uniform without any coordination and stable across runs, and no directory becomes a dumping ground however many times the demo is run.

Uniqueness is preserved, and it is load-bearing rather than incidental: the leaf carries the run tag, the change index and the file index, so no two files in a run and no two runs against one repository can name the same path. Independent changes that collided on content would make the run measure conflict handling instead of the throughput it exists to show. Two unrelated changes landing in the same bucket is expected and harmless — the bucket is only a directory, and it is the leaf that has to be distinct.

How many files a change touches varies rather than being fixed, so a run does not produce a row of identically shaped pull requests. `-files` sets the floor (default 3, exposed as `FILES` on `make demo-pr`) and the actual count runs a little above it. The variation is derived from the run tag and the change index rather than from a clock, so replaying a tag reproduces the same run — a demo that cannot be reproduced is hard to talk about once something in it goes wrong.

Each file is committed separately, so a change arrives as a multi-file, multi-commit pull request. That is closer to a real change, and it means the demo exercises replaying a range of commits rather than always handing the merger a single one.

## Test Plan

✅ `bazel test //service/submitqueue/demo/pr:go_default_test` — five new cases over the two pure functions: paths are unique across every change and file of two runs, the layout is the demo root plus hex buckets plus the leaf, 80 files spread over more than 50 distinct buckets (a layout that put everything in one directory would pass the uniqueness test alone), the file count respects its floor and clamps a non-positive one, and the count both varies across changes and reproduces for a given tag.

✅ Cross-checked the generated layout against an independent recomputation of the same hashes in shell, rather than only against the implementation's own output.

Not yet exercised against GitHub: the path and count logic is covered above, but the multi-commit loop against the live API — several sequential commits on one branch, and the last SHA as the head the change URI pins — will first run on the next `make demo-pr`.

Author: behinddwalls

Makefile

@@ -55,6 +55,7 @@ export SQ_PROVIDER_CONFIG_DIR ?= $(REPO_ROOT)/service/submitqueue/demo/provider/
 # Defaults for `make land` / `make demo-pr` against the provider demo stack.
 DEMO_REPO ?= behinddwalls/sq-demo
 COUNT ?= 3
+FILES ?= 3
 STACKED ?= false
 LAND ?= true
 WATCH ?= true
@@ -152,10 +153,11 @@ clean-proto: ## Clean generated proto files
 	@rm -f $(foreach p,$(PROTO_PACKAGES),$(p)/protopb/*.pb.go $(p)/protopb/*.pb.yarpc.go)
 	@echo "Proto clean complete!"
 
-demo-pr: ## Create N PRs in the demo repo, enqueue each as it is created, and watch (COUNT=3; needs GITHUB_TOKEN)
+demo-pr: ## Create N PRs in the demo repo, enqueue each as it is created, and watch (COUNT=3 FILES=3; needs GITHUB_TOKEN)
 	@$(BAZEL) run //service/submitqueue/demo/pr -- \
 		-repo $(DEMO_REPO) \
 		-count $(COUNT) \
+		-files $(FILES) \
 		-stacked=$(STACKED) \
 		-gateway $(GATEWAY_ADDR) \
 		-queue $(QUEUE) \

doc/howto/PROVIDER-E2E.md

@@ -96,6 +96,7 @@ Opening pull requests by hand gets old fast. `demo-pr` creates them, enqueues th
 ```bash
 make demo-pr                      # 3 independent PRs, each enqueued as it is created
 make demo-pr COUNT=8              # more traffic
+make demo-pr FILES=8              # wider changes, more files per PR
 make demo-pr STACKED=true         # one stack, enqueued as a single request
 make demo-pr LAND=false           # create only, print the land command
 ```
@@ -120,7 +121,9 @@ The trail is only as detailed as what the pipeline reports, which today is `acce
 
 `STACKED=true` is the exception to the overlap: one request carries the whole chain, so it can only go in once every pull request in it exists. That is the atomic-stack path — the whole set reaches `main` in a single push, and the table shows it as the single row it is.
 
-It talks to GitHub over the REST API with the same `GITHUB_TOKEN`, so it needs no clone and no git binary. Each run tags its branches with a timestamp so repeated runs do not collide, and each change edits its own file so independent changes do not conflict by accident.
+It talks to GitHub over the REST API with the same `GITHUB_TOKEN`, so it needs no clone and no git binary. Each run tags its branches with a timestamp so repeated runs do not collide, and every file a change writes is at a path no other change uses, so independent changes do not conflict by accident.
+
+A change touches several files rather than one, each committed separately, so it arrives as a multi-file, multi-commit pull request — closer to a real change, and enough to exercise replaying a range of commits. `FILES` sets the floor (default 3); the actual count varies a little above it, derived from the run tag so replaying a tag reproduces the same run. Paths are sharded into two levels of hex buckets under `demo/` (`demo/c2/91/<tag>-<change>-<file>.txt`), which keeps the tree from degenerating into one enormous directory as runs accumulate.
 
 The command exits non-zero if any request settles anywhere other than `landed`, so it works in a script. Piped to a file it prints a fresh table whenever a request moves — and not when only the clock did — instead of redrawing in place.
 

service/submitqueue/demo/pr/main.go

@@ -47,6 +47,7 @@ package main
 import (
 	"bytes"
 	"context"
+	"crypto/sha256"
 	"encoding/base64"
 	"encoding/json"
 	"flag"
@@ -106,6 +107,7 @@ type config struct {
 	repo     string
 	base     string
 	count    int
+	files    int
 	stacked  bool
 	prefix   string
 	land     bool
@@ -123,6 +125,7 @@ func parseFlags() config {
 	flag.StringVar(&c.repo, "repo", "behinddwalls/sq-demo", "scratch repository as owner/name")
 	flag.StringVar(&c.base, "base", "main", "branch the changes target")
 	flag.IntVar(&c.count, "count", 3, "how many pull requests to create")
+	flag.IntVar(&c.files, "files", 3, "fewest files each pull request touches; the actual count varies a little above it")
 	flag.BoolVar(&c.stacked, "stacked", false, "chain the pull requests and enqueue them as one stack")
 	flag.StringVar(&c.prefix, "prefix", "demo", "branch name prefix")
 	flag.BoolVar(&c.land, "land", true, "enqueue each pull request as it is created")
@@ -302,6 +305,51 @@ func (rw *row) stage() string {
 	return absent
 }
 
+// shardDirs is how many nested bucket directories a path carries under the demo
+// root. Two levels of 256 buckets spread a run's files widely enough that no
+// directory becomes a dumping ground, while staying shallow enough to read in a
+// diff.
+const shardDirs = 2
+
+// changeFilePath returns the repository path for one file of one change.
+//
+// The leaf name carries the run tag, the change index and the file index, which
+// is what makes it unique: no two files in a run, and no two runs against the
+// same repository, can ever name the same path. That uniqueness is load-bearing
+// — see createAndEnqueue.
+//
+// The directories are the leading bytes of the leaf's SHA-256, so files land in
+// buckets that are uniform without any coordination and stable across runs. Two
+// unrelated changes sharing a bucket is expected and harmless: the bucket is
+// only a directory, and it is the leaf that has to be distinct.
+func changeFilePath(tag string, change, file int) string {
+	leaf := fmt.Sprintf("%s-%d-%d.txt", tag, change, file)
+	sum := sha256.Sum256([]byte(leaf))
+
+	parts := make([]string, 0, shardDirs+2)
+	parts = append(parts, "demo")
+	for i := 0; i < shardDirs; i++ {
+		parts = append(parts, fmt.Sprintf("%02x", sum[i]))
+	}
+	parts = append(parts, leaf)
+	return strings.Join(parts, "/")
+}
+
+// changeFileCount returns how many files a change touches: at least min, varied
+// a little so a run does not produce a row of identically shaped pull requests.
+//
+// The variation is derived from the run tag and the change index rather than
+// from a clock or a global source of randomness, so replaying a tag reproduces
+// the same run. A demo that cannot be reproduced is hard to talk about when
+// something in it goes wrong.
+func changeFileCount(tag string, change, min int) int {
+	if min < 1 {
+		min = 1
+	}
+	sum := sha256.Sum256([]byte(fmt.Sprintf("%s#%d", tag, change)))
+	return min + int(sum[0]%4)
+}
+
 // createAndEnqueue opens the pull requests and puts them on the queue, filling
 // in the tracker's rows as it goes and reporting each step beneath the table.
 //
@@ -311,10 +359,13 @@ func (rw *row) stage() string {
 // carries the whole chain, so it can only be submitted once the chain is
 // complete.
 //
-// Each change edits its own file. Independent changes would otherwise collide
-// on content and the run would measure conflict handling rather than the
-// throughput it is trying to show; a caller wanting a conflict can make one
-// deliberately.
+// Every file a change writes is its own, at a path no other change uses.
+// Independent changes would otherwise collide on content and the run would
+// measure conflict handling rather than the throughput it is trying to show; a
+// caller wanting a conflict can make one deliberately. Each change spreads
+// several files across the sharded tree, so it arrives as a multi-file, multi-
+// commit pull request rather than a single-line edit — which is both closer to
+// a real change and enough to exercise replaying a range of commits.
 func createAndEnqueue(
 	ctx context.Context,
 	gh *githubClient,
@@ -340,12 +391,22 @@ func createAndEnqueue(
 			return nil, fmt.Errorf("create branch %s: %w", branch, err)
 		}
 
-		path := fmt.Sprintf("demo/%s-%d.txt", tag, i)
-		body := fmt.Sprintf("change %d of run %s\n", i, tag)
-		t.note("committing %s", path)
-		headSHA, err := gh.commitFile(ctx, branch, path, body, fmt.Sprintf("demo change %d (run %s)", i, tag))
-		if err != nil {
-			return nil, fmt.Errorf("commit to %s: %w", branch, err)
+		// Each file is its own commit, so the pull request arrives as a range of
+		// commits rather than a single edit. The last one is the head the change
+		// URI pins.
+		var headSHA string
+		fileCount := changeFileCount(tag, i, cfg.files)
+		for k := 1; k <= fileCount; k++ {
+			path := changeFilePath(tag, i, k)
+			body := fmt.Sprintf("change %d of run %s\nfile %d of %d\n", i, tag, k, fileCount)
+			t.note("committing %s (%d/%d)", path, k, fileCount)
+
+			message := fmt.Sprintf("demo change %d (run %s): file %d of %d", i, tag, k, fileCount)
+			sha, err := gh.commitFile(ctx, branch, path, body, message)
+			if err != nil {
+				return nil, fmt.Errorf("commit %s to %s: %w", path, branch, err)
+			}
+			headSHA = sha
 		}
 
 		t.note("opening pull request for %s", branch)

service/submitqueue/demo/pr/main_test.go

@@ -656,3 +656,84 @@ func head(s string, n int) string {
 	}
 	return s[:n]
 }
+
+func TestChangeFilePath_IsUniquePerFileAcrossChangesAndRuns(t *testing.T) {
+	// Uniqueness is the property the whole layout rests on: two changes writing
+	// the same path would collide on content, and the run would measure conflict
+	// handling instead of throughput.
+	seen := make(map[string]string)
+	for _, tag := range []string{"0810-1203", "0810-1204"} {
+		for change := 1; change <= 20; change++ {
+			for file := 1; file <= 8; file++ {
+				path := changeFilePath(tag, change, file)
+				owner := fmt.Sprintf("%s/%d/%d", tag, change, file)
+				if prev, ok := seen[path]; ok {
+					t.Fatalf("path %s produced for both %s and %s", path, prev, owner)
+				}
+				seen[path] = owner
+			}
+		}
+	}
+}
+
+func TestChangeFilePath_ShardsUnderTheDemoRoot(t *testing.T) {
+	path := changeFilePath("0810-1203", 1, 1)
+
+	parts := strings.Split(path, "/")
+	require.Len(t, parts, shardDirs+2, "demo root, %d bucket dirs, and the leaf", shardDirs)
+	assert.Equal(t, "demo", parts[0])
+	for _, bucket := range parts[1 : len(parts)-1] {
+		assert.Len(t, bucket, 2, "each bucket is one hex byte")
+		assert.Regexp(t, "^[0-9a-f]{2}$", bucket)
+	}
+	assert.Equal(t, "0810-1203-1-1.txt", parts[len(parts)-1])
+}
+
+func TestChangeFilePath_SpreadsAcrossManyBuckets(t *testing.T) {
+	// A layout that puts everything in one directory would satisfy the
+	// uniqueness test above while defeating the point of sharding.
+	buckets := make(map[string]struct{})
+	for change := 1; change <= 20; change++ {
+		for file := 1; file <= 4; file++ {
+			parts := strings.Split(changeFilePath("0810-1203", change, file), "/")
+			buckets[strings.Join(parts[1:len(parts)-1], "/")] = struct{}{}
+		}
+	}
+	assert.Greater(t, len(buckets), 50, "80 files should land in many distinct buckets")
+}
+
+func TestChangeFileCount(t *testing.T) {
+	tests := []struct {
+		name string
+		min  int
+	}{
+		{name: "default minimum", min: 3},
+		{name: "single file floor", min: 1},
+		{name: "non-positive is clamped", min: 0},
+		{name: "negative is clamped", min: -5},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			floor := tt.min
+			if floor < 1 {
+				floor = 1
+			}
+			for change := 1; change <= 50; change++ {
+				got := changeFileCount("0810-1203", change, tt.min)
+				assert.GreaterOrEqual(t, got, floor)
+				assert.LessOrEqual(t, got, floor+3)
+			}
+		})
+	}
+}
+
+func TestChangeFileCount_VariesButIsReproducible(t *testing.T) {
+	counts := make(map[int]struct{})
+	for change := 1; change <= 30; change++ {
+		got := changeFileCount("0810-1203", change, 3)
+		counts[got] = struct{}{}
+		assert.Equal(t, got, changeFileCount("0810-1203", change, 3),
+			"replaying a tag must reproduce the run")
+	}
+	assert.Greater(t, len(counts), 1, "the count should vary across changes, not be constant")
+}