feat: a git change provider, and a git rung that reads its own repository

doc/howto/QUICKSTART.md

@@ -4,15 +4,17 @@ Start the stack, put traffic through it, and watch changes land — beginning wi
 
 The stack always runs the same way. What changes is where the changes come from and what landing them does, chosen with `PROVIDER`:
 
-| `PROVIDER` | A change is | Building it | Landing it | Needs |
-|---|---|---|---|---|
-| **`fake`** (default) | a URI, and nothing else | instant fake pass | reports success without touching a repository | nothing |
-| **`git`** | a branch in a bare repository on disk | instant fake pass | a real fetch, cherry-pick and push | nothing |
-| **`github`** | a real pull request | a real GitHub Actions run per batch | a real push to a real repository | a repository, a token, and CI minutes |
+| `PROVIDER` | A change is | Read from | Building it | Landing it | Needs |
+|---|---|---|---|---|---|
+| **`fake`** (default) | a URI, and nothing else | the URI itself | instant fake pass | reports success without touching a repository | nothing |
+| **`git`** | a branch in a bare repository on disk | the repository | instant fake pass | a real fetch, cherry-pick and push | nothing |
+| **`github`** | a real pull request | GitHub's API | a real GitHub Actions run per batch | a real push to a real repository | a repository, a token, and CI minutes |
 
 They are a ladder, not alternatives: the same commands work on each rung, so you can start with the one that needs nothing and only pay for what you want to see next. Each is a directory of configuration under [`service/submitqueue/demo/provider/`](../../service/submitqueue/demo/provider) — the difference between rungs is two YAML files, not a code path.
 
-The queue's own logic is real on every rung; what changes is how much of the world around it is. Two things are worth knowing before reading a `landed` as more than it is. On `fake` and `git` **the build is faked**, so `landed` means the pipeline ran, not that anything was tested. And on `fake` and `git` the change provider is faked too: it cannot read a repository to see what a change touched, so `make demo-requests` states the paths on the change URI itself (`sq-files=`) for the conflict analyzer to key on. A change submitted by hand on those rungs touches nothing as far as the analyzer can tell, and conflicts with nothing.
+The queue's own logic is real on every rung; what changes is how much of the world around it is. The one thing to keep in mind before reading a `landed` as more than it is: on `fake` and `git` **the build is faked**, so it means the pipeline ran, not that anything was tested.
+
+"Read from" is what the queue knows about a change — which files it touches, how large it is — and it is what conflict analysis and scoring are computed from. Only `fake` invents it: a change there is a URI pointing at nothing, so `make demo-requests` states the paths on the URI itself (`sq-files=`) and the fake reads them back, which means a change submitted by hand on that rung conflicts with nothing. On `git` the orchestrator keeps its own copy of the repository and reads the commits, so a change pushed by anyone is described correctly.
 
 ## Start the stack
 

platform/gitexec/gitexectest/BUILD.bazel

@@ -0,0 +1,9 @@
+load("@rules_go//go:def.bzl", "go_library")
+
+go_library(
+    name = "go_default_library",
+    srcs = ["gitexectest.go"],
+    importpath = "github.com/uber/submitqueue/platform/gitexec/gitexectest",
+    visibility = ["//visibility:public"],
+    deps = ["@com_github_stretchr_testify//require:go_default_library"],
+)

platform/gitexec/gitexectest/gitexectest.go

@@ -0,0 +1,86 @@
+// Copyright (c) 2025 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package gitexectest resolves the Bazel-pinned git a test target supplies.
+//
+// Tests that shell out to git run against the same pinned build the services
+// use, rather than whatever the host happens to have, so a developer's git
+// version cannot change what a test proves. A target opts in by depending on
+// the binary and naming it in the environment:
+//
+//	go_test(
+//	    data = ["@git"],
+//	    env = {"SUBMITQUEUE_TEST_GIT": "$(location @git//:git)"},
+//	)
+//
+// The indirection through an environment variable is Bazel's: rules_go expands
+// $(location) to an execroot-relative path, which for an external output has to
+// be re-rooted under the runfiles tree the test actually runs from.
+package gitexectest
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"github.com/stretchr/testify/require"
+)
+
+// GitEnv is the variable a test target sets to the pinned git binary.
+const GitEnv = "SUBMITQUEUE_TEST_GIT"
+
+// Git returns an absolute path to the pinned git binary, failing the test if
+// the target did not supply one.
+func Git(t *testing.T) string {
+	t.Helper()
+	return Runfile(t, GitEnv)
+}
+
+// Runfile resolves the $(location)-expanded path held by the named environment
+// variable, failing the test if it is unset or does not resolve.
+//
+// A path that is already usable is returned as-is, which is what happens under
+// `bazel run`, where the process already starts inside the runfiles tree.
+func Runfile(t *testing.T, name string) string {
+	t.Helper()
+
+	path := os.Getenv(name)
+	require.NotEmpty(t, path, "%s must be set by the test target", name)
+
+	if absolute, err := filepath.Abs(path); err == nil {
+		if _, err := os.Stat(absolute); err == nil {
+			return absolute
+		}
+	}
+
+	// Both spellings occur: the path is execroot-relative for a target in this
+	// repository, and carries a leading segment for one reached through another.
+	slashed := filepath.ToSlash(path)
+	external := ""
+	if strings.HasPrefix(slashed, "external/") {
+		external = strings.TrimPrefix(slashed, "external/")
+	} else if i := strings.Index(slashed, "/external/"); i >= 0 {
+		external = slashed[i+len("/external/"):]
+	}
+	require.NotEmpty(t, external, "%s=%q is not a runfile", name, path)
+
+	root := os.Getenv("TEST_SRCDIR")
+	require.NotEmpty(t, root, "TEST_SRCDIR must be set when %s is a runfile", name)
+
+	candidate := filepath.Join(root, filepath.FromSlash(external))
+	_, err := os.Stat(candidate)
+	require.NoError(t, err, "%s=%q does not resolve under TEST_SRCDIR", name, path)
+	return candidate
+}

runway/extension/merger/git/BUILD.bazel

@@ -48,6 +48,7 @@ go_test(
         "//api/base/mergestrategy/protopb:go_default_library",
         "//api/runway/messagequeue:go_default_library",
         "//api/runway/messagequeue/protopb:go_default_library",
+        "//platform/gitexec/gitexectest:go_default_library",
         "//runway/extension/merger:go_default_library",
         "@com_github_stretchr_testify//assert:go_default_library",
         "@com_github_stretchr_testify//require:go_default_library",

runway/extension/merger/git/git_merger_test.go

@@ -35,6 +35,7 @@ import (
 	mergestrategypb "github.com/uber/submitqueue/api/base/mergestrategy/protopb"
 	runwaymq "github.com/uber/submitqueue/api/runway/messagequeue"
 	runwaypb "github.com/uber/submitqueue/api/runway/messagequeue/protopb"
+	"github.com/uber/submitqueue/platform/gitexec/gitexectest"
 	"github.com/uber/submitqueue/runway/extension/merger"
 )
 
@@ -1782,54 +1783,15 @@ func mustGitOutput(t *testing.T, dir string, args ...string) []byte {
 
 func testGitRuntime(t *testing.T) GitRuntime {
 	t.Helper()
-	executable := absoluteTestPath(t, "SUBMITQUEUE_TEST_GIT")
-	templateDescription := absoluteTestPath(t, "SUBMITQUEUE_TEST_GIT_TEMPLATE_DESCRIPTION")
+	executable := gitexectest.Git(t)
+	templateDescription := gitexectest.Runfile(t, "SUBMITQUEUE_TEST_GIT_TEMPLATE_DESCRIPTION")
 	return GitRuntime{
 		Executable:  executable,
 		ExecPath:    filepath.Dir(executable),
 		TemplateDir: filepath.Dir(templateDescription),
 	}
 }
 
-func absoluteTestPath(t *testing.T, name string) string {
-	t.Helper()
-	path := os.Getenv(name)
-	require.NotEmpty(t, path)
-	if filepath.IsAbs(path) {
-		_, err := os.Stat(path)
-		require.NoError(t, err)
-		return path
-	}
-	if absolute, err := filepath.Abs(path); err == nil {
-		if _, err := os.Stat(absolute); err == nil {
-			return absolute
-		}
-	}
-
-	// rules_go expands $(location) to an execroot-relative path. Tests run from
-	// their main-repository runfiles directory, so translate an external output
-	// to its canonical repository path under TEST_SRCDIR.
-	const externalMarker = "/external/"
-	slashed := filepath.ToSlash(path)
-	externalPath := ""
-	if strings.HasPrefix(slashed, "external/") {
-		externalPath = strings.TrimPrefix(slashed, "external/")
-	} else if i := strings.Index(slashed, externalMarker); i >= 0 {
-		externalPath = slashed[i+len(externalMarker):]
-	}
-	if externalPath != "" {
-		runfilesRoot := os.Getenv("TEST_SRCDIR")
-		require.NotEmpty(t, runfilesRoot)
-		candidate := filepath.Join(runfilesRoot, filepath.FromSlash(externalPath))
-		_, err := os.Stat(candidate)
-		require.NoError(t, err)
-		return candidate
-	}
-
-	require.FailNowf(t, "resolve test path", "%s=%q is not a runfile", name, path)
-	return ""
-}
-
 func writeFile(path, contents string) error {
 	if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
 		return err

service/runway/server/BUILD.bazel

@@ -96,6 +96,7 @@ go_test(
     },
     deps = [
         "//api/base/mergestrategy/protopb:go_default_library",
+        "//platform/gitexec/gitexectest:go_default_library",
         "//runway/extension/merger/git:go_default_library",
         "@com_github_stretchr_testify//assert:go_default_library",
         "@com_github_stretchr_testify//require:go_default_library",

service/runway/server/checkout_test.go

@@ -26,52 +26,23 @@ import (
 	"github.com/stretchr/testify/require"
 	"go.uber.org/zap/zaptest"
 
+	"github.com/uber/submitqueue/platform/gitexec/gitexectest"
 	gitmerger "github.com/uber/submitqueue/runway/extension/merger/git"
 )
 
 // testRuntime resolves the pinned git the Bazel target supplies. Provisioning
 // runs real git, so these tests use the same runtime the merger will.
 func testRuntime(t *testing.T) gitmerger.GitRuntime {
 	t.Helper()
-	executable := runfilePath(t, "SUBMITQUEUE_TEST_GIT")
-	templateDescription := runfilePath(t, "SUBMITQUEUE_TEST_GIT_TEMPLATE_DESCRIPTION")
+	executable := gitexectest.Git(t)
+	templateDescription := gitexectest.Runfile(t, "SUBMITQUEUE_TEST_GIT_TEMPLATE_DESCRIPTION")
 	return gitmerger.GitRuntime{
 		Executable:  executable,
 		ExecPath:    filepath.Dir(executable),
 		TemplateDir: filepath.Dir(templateDescription),
 	}
 }
 
-// runfilePath resolves a $(location)-expanded path from the test environment.
-// rules_go emits an execroot-relative path, which for an external output has to
-// be re-rooted under the runfiles directory the test actually runs from.
-func runfilePath(t *testing.T, name string) string {
-	t.Helper()
-	path := os.Getenv(name)
-	require.NotEmpty(t, path, "%s must be set by the test target", name)
-	if absolute, err := filepath.Abs(path); err == nil {
-		if _, err := os.Stat(absolute); err == nil {
-			return absolute
-		}
-	}
-
-	slashed := filepath.ToSlash(path)
-	external := ""
-	if strings.HasPrefix(slashed, "external/") {
-		external = strings.TrimPrefix(slashed, "external/")
-	} else if i := strings.Index(slashed, "/external/"); i >= 0 {
-		external = slashed[i+len("/external/"):]
-	}
-	require.NotEmpty(t, external, "%s=%q is not a runfile", name, path)
-
-	root := os.Getenv("TEST_SRCDIR")
-	require.NotEmpty(t, root)
-	candidate := filepath.Join(root, filepath.FromSlash(external))
-	_, err := os.Stat(candidate)
-	require.NoError(t, err)
-	return candidate
-}
-
 // seedBareRepo creates a bare repository holding one commit on branch and
 // returns its path — the shape of the remote a merge target points at.
 func seedBareRepo(t *testing.T, branch string) string {

service/submitqueue/demo/provider/README.md

@@ -14,10 +14,12 @@ Pick one with `make local-submitqueue-start PROVIDER=<name>`, which bind-mounts
 | Directory | What it demonstrates | Needs |
 |---|---|---|
 | [`fake/`](fake) | the queue alone: a change is a URI, nothing merges anywhere — the default, and what the quickstart runs | nothing |
-| [`git/`](git) | a plain git remote with no provider at all: real fetch, cherry-pick and push against a bare repository | nothing |
+| [`git/`](git) | a plain git remote with no provider at all: change metadata read out of the repository, and a real fetch, cherry-pick and push against it | nothing |
 | [`github/`](github) | a live provider: GitHub change metadata, a real repository, pull requests marked merged | a repository and a token |
 
-The three are a ladder, and the rung is the only thing that changes: the same commands land against all of them. `git/` is worth reading first of the two real ones. It is proof that the merge machinery has no provider in it — the same Runway code path lands changes against a bare repository addressed by path, with no credential and no API — and it is what the hermetic git E2E (`make e2e-git-test`) runs against.
+The three are a ladder, and the rung is the only thing that changes: the same commands land against all of them. `git/` is worth reading first of the two real ones. It is proof that neither half needs a provider — change metadata comes from reading the repository and the merge is the same Runway code path, both against a bare repository addressed by path, with no credential and no API — and it is what the hermetic git E2E (`make e2e-git-test`) runs against.
+
+Note that the orchestrator and Runway each keep their **own** copy of a queue's repository and configure their own remote for it: the change provider's copy is in `profiles.yaml`, the merger's checkout in `merge.yaml`, and they are independent even when they name the same remote. That is the same shape a deployment has when both point at a remote host.
 
 ## Adding a provider
 

service/submitqueue/demo/provider/git/profiles.yaml

@@ -17,15 +17,32 @@ queues:
   # The queue `make demo-requests` and `make land` use by default. Serializes
   # batches that touch a shared directory, matching the github mode.
   #
-  # The commits here are real, but the change provider above is not, and it
-  # cannot read a repository to find out what they touched. `make demo-requests`
-  # states the paths it committed on the change URI (`sq-files=`) for the fake
-  # provider to report back. A change submitted by hand carries no such marker
-  # and so conflicts with nothing.
+  # Both halves are real here. The change provider keeps its own copy of the
+  # same bare repository Runway merges into and reads each change out of it, so
+  # the analyzer keys on paths that were actually committed — including for a
+  # change submitted by hand, which nothing else could have described.
   - name: demo-queue
+    changeProvider:
+      type: git
+      git:
+        # The orchestrator's own copy, and its own remote pointing at the
+        # sandbox — mounted read-only, since it only ever fetches. Runway
+        # configures the same repository separately for itself in merge.yaml.
+        remoteUrl: file:///srv/git/sandbox.git
+        target: main
+        repoPath: /var/submitqueue/changerepos/demo
     analyzer: {type: pathoverlap, by: directory}
 
   - name: e2e-git-queue
+    # Its own copy, separate from demo-queue's: the two agree about the
+    # repository but not about everything else, and a shared path would make
+    # whichever was built first decide for both.
+    changeProvider:
+      type: git
+      git:
+        remoteUrl: file:///srv/git/sandbox.git
+        target: main
+        repoPath: /var/submitqueue/changerepos/sandbox
     # Maximum parallelism: batches never conflict, so the test controls
     # ordering through what it lands rather than through the analyzer.
     analyzer: {type: none}

service/submitqueue/demo/requests/BUILD.bazel

@@ -49,6 +49,7 @@ go_test(
         "//platform/base/change/git:go_default_library",
         "//platform/fakemarker:go_default_library",
         "//platform/gitexec:go_default_library",
+        "//platform/gitexec/gitexectest:go_default_library",
         "@com_github_stretchr_testify//assert:go_default_library",
         "@com_github_stretchr_testify//require:go_default_library",
     ],

service/submitqueue/demo/requests/git.go

@@ -136,13 +136,15 @@ func (s *gitSource) open(ctx context.Context, spec changeSpec) (openedChange, er
 
 	return openedChange{
 		headSHA: headSHA,
-		// The commits are real, but the fake change provider is what the
-		// orchestrator asks about them, and it cannot read a repository — so the
-		// paths just committed are stated on the URI for it to report back.
-		uri: withFiles(gitchange.ChangeID{
+		// Nothing is stated about what this change touches. The orchestrator
+		// keeps its own copy of this repository and reads that out of the
+		// commits, which is the whole difference between this rung and the fake
+		// one — and what makes a change pushed by hand behave the same as one
+		// from here.
+		uri: gitchange.ChangeID{
 			Scheme: "git", Remote: gitRemote, Repo: s.repo,
 			Ref: "refs/heads/" + spec.branch, CommitSHA: headSHA,
-		}.String(), spec.files),
+		}.String(),
 		// No pull request to number, so the branch names the change. Empty URL:
 		// a branch in a bare repository has nothing to open.
 		cell: client.Cell{Text: spec.branch},