Skip to content

Commit b56440e

Browse files
committed
feat(runway): git-backed merger with REBASE
## Summary ### Why? Runway's `merger` extension has had exactly one implementation — `noop`, which always succeeds. Nothing actually merges anything. This lands the first real backend: a `Merger` driven by the `git` CLI against a local checkout. It ships REBASE only. The strategy-specific apply paths are small and independent, but the machinery underneath them — the pinned git runtime, object resolution, the reset/apply/push cycle, contention retry, dry-run discard, conflict classification — is shared and is the bulk of what needs review. Landing it with one strategy keeps that review separable from the per-strategy mechanics that follow in this stack. ### What? Adds `runway/extension/merger/git`. **A change is a range of commits, not a commit.** A change URI pins a pull request to a single head SHA, but a pull request is routinely several commits. Applying the head alone applies only that commit's diff against its own parent: it conflicts against context its predecessors would have established, or — when the commits touch different files — succeeds while silently dropping everything before the head. So the unit replayed is the range from the change's merge base with the target up to its head. A change already contained in the target has an empty range and is a no-op, which is what keeps redelivery idempotent. A range runs through git's sequencer, which changes the control flow versus a single pick: the operation stops on a commit it declines and `--skip` advances to the next rather than ending. Two kinds of commit stop it harmlessly — one already present on the target, and one that was empty to begin with — and both are skipped. Anything else is a real conflict, and the in-progress pick is aborted so the checkout stays usable. The commits an apply produced are read back off the checkout rather than tracked per-invocation, which stays correct when the sequencer drops some. **Referenced commits are guaranteed present before anything is applied.** The default fetch refspec is `+refs/heads/*`, which does not cover a provider's change refs: a pull request head never also pushed as a branch — the normal case for a fork — is simply absent, and the apply then fails with git's "bad object", indistinguishable from a conflict. Every commit the request names is now fetched and verified up front, for all steps, so an unusable request fails without having mutated the checkout. Commits are requested by SHA (relying on the server serving a want for a reachable-but-unadvertised object, which github.com allows), falling back to the provider's canonical ref, with deployment-supplied refspecs as a last resort. Neither fetch is shallow — the range needs ancestry. A commit a *reachable* remote cannot supply is terminal; a remote that will not answer stays retryable. The first is a property of the request, the second a property of the moment. **One seam for change providers.** Every URI is reduced to the only three things the merger needs — the commit to apply, the ref the provider publishes it under, and a label for synthesized messages. `github://` and `git://` are supported; an unrecognized scheme is terminal. Adding a provider is one case in that mapping rather than a change to any apply path. **Optional staleness check.** Fetching by SHA guarantees the merger applies exactly the commit the URI names, not that it is still the change's head — a force-push leaves the superseded commit fetchable on most hosts. When enabled, each change's canonical ref is read (one ref advertisement, no object transfer) and a mismatch is terminal. **Atomicity, contention, dry run.** Nothing reaches the remote until the final push. If the push is rejected because the remote tip moved, the whole reset/apply/push cycle retries up to a bounded number of attempts. `CheckMergeability` runs the identical apply path but never pushes, then resets and reports empty outputs, committing intermediate steps locally so a multi-step check sees the same conflict surface a real merge would. **Runtime hygiene.** Every invocation uses an explicitly pinned git runtime and a scrubbed environment — no system or global config, no interactive prompts. That leaves no ambient identity, so the committer is injected per-invocation. `isConcreteStrategy` currently admits only REBASE; SQUASH_REBASE, MERGE and PROMOTE are rejected as invalid requests until their apply paths land later in this stack. The merger is not wired into the server yet, so this adds no production behavior. ## Test Plan ✅ `bazel test //runway/...` — 6/6 targets pass, including the new `//runway/extension/merger/git` suite (40s) The suite drives a real git binary against throwaway repositories. Beyond the single-commit cases (stacked URIs, already-landed changes, multi-step requests, conflicts, checkout recovery, contention retry, give-up after max attempts, DEFAULT resolution, dry runs), it covers what this change is actually about: a multi-commit change expressed as **one** URI for both disjoint and overlapping edits, a partially-landed change, an empty commit inside a range, a conflict inside a range leaving no sequencer state behind, stacked multi-commit changes not duplicating each other's commits, a head reachable only via its pull-request ref, an unavailable commit classified as invalid rather than conflicting, the staleness check on and off, and provider resolution across schemes. The regression tests were verified to fail against head-only picking: reverting just that line fails 5 of them, and they pass with the range applied.
1 parent d8ee8ae commit b56440e

6 files changed

Lines changed: 2094 additions & 0 deletions

File tree

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
load("@rules_go//go:def.bzl", "go_library", "go_test")
2+
3+
go_library(
4+
name = "go_default_library",
5+
srcs = [
6+
"changeref.go",
7+
"git_merger.go",
8+
"objects.go",
9+
],
10+
importpath = "github.com/uber/submitqueue/runway/extension/merger/git",
11+
visibility = ["//visibility:public"],
12+
deps = [
13+
"//api/base/mergestrategy/protopb:go_default_library",
14+
"//api/runway/messagequeue:go_default_library",
15+
"//api/runway/messagequeue/protopb:go_default_library",
16+
"//platform/base/change/git:go_default_library",
17+
"//platform/base/change/github:go_default_library",
18+
"//platform/metrics:go_default_library",
19+
"//runway/extension/merger:go_default_library",
20+
"@com_github_uber_go_tally//:go_default_library",
21+
"@org_uber_go_zap//:go_default_library",
22+
],
23+
)
24+
25+
go_test(
26+
name = "go_default_test",
27+
srcs = ["git_merger_test.go"],
28+
data = [
29+
"@git",
30+
"@git//:git_receive_pack",
31+
"@git//:git_upload_archive",
32+
"@git//:git_upload_pack",
33+
"@git//:templates",
34+
"@git//:templates/description",
35+
],
36+
embed = [":go_default_library"],
37+
env = {
38+
"SUBMITQUEUE_TEST_GIT": "$(location @git//:git)",
39+
"SUBMITQUEUE_TEST_GIT_TEMPLATE_DESCRIPTION": "$(location @git//:templates/description)",
40+
},
41+
deps = [
42+
"//api/base/change/protopb:go_default_library",
43+
"//api/base/mergestrategy/protopb:go_default_library",
44+
"//api/runway/messagequeue:go_default_library",
45+
"//api/runway/messagequeue/protopb:go_default_library",
46+
"//runway/extension/merger:go_default_library",
47+
"@com_github_stretchr_testify//assert:go_default_library",
48+
"@com_github_stretchr_testify//require:go_default_library",
49+
"@com_github_uber_go_tally//:go_default_library",
50+
"@org_uber_go_zap//zaptest:go_default_library",
51+
],
52+
)
Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
# git merger
2+
3+
A `merger.Merger` backed by the `git` CLI operating on a local checkout. It applies a merge request's ordered steps onto a target branch, honoring each step's strategy, and — for a committing merge — pushes the result. It is constructed by the wiring layer (see [`service/runway`](../../../../service/runway)) with the checkout path, remote, target branch, pinned git runtime, committer identity, and the default strategy; the request itself carries only a queue name and the changes, so the merger reads change URIs straight from the payload (no store, no resolver).
4+
5+
## Model
6+
7+
A request is an ordered list of steps; each step names a change (a set of provider URIs, each ending in a full head commit SHA) and a strategy. Steps are applied in order on top of the target tip — earlier steps are the in-flight base, the last step is the candidate. Each step yields one `StepResult`; the revisions a step produces on the target are its outputs, in application order.
8+
9+
A URI pins a change to one head commit, but a change is routinely several commits. The full set is recovered locally rather than from the wire: the commits to replay are the range from the change's merge base with the target up to its head. Applying the head commit alone would apply only that commit's diff against its own parent — conflicting against context its predecessors would have established, or silently dropping them when they touch different files.
10+
11+
## Change providers
12+
13+
Every URI is reduced to three things: the commit to apply, the ref the provider publishes that commit under, and a short label for synthesized commit messages. Adding a provider is one case in that mapping, not a change to any apply path.
14+
15+
| Scheme | Commit | Canonical ref | Label |
16+
|---|---|---|---|
17+
| `github://` | head commit SHA | `refs/pull/{n}/head` | `org/repo#n` |
18+
| `git://` | the URI's commit SHA | the URI's own ref | `repo@ref` |
19+
20+
An unrecognized scheme is a terminal invalid request.
21+
22+
## Object availability
23+
24+
The default fetch refspec is `+refs/heads/*`, which does not cover a provider's change refs — a pull request head never also pushed as a branch, the normal case for a fork, is simply absent locally. Every referenced commit is therefore fetched and verified before any step is applied, so a request naming an unreachable commit fails without having touched the checkout.
25+
26+
Commits are requested by SHA, which relies on the server serving a want for an object that is reachable but not advertised (`uploadpack.allowReachableSHA1InWant`, which github.com enables); the provider's canonical ref is the fallback. Neither fetch is shallow — the apply paths need ancestry, not just the commit. A remote that supports neither can supply explicit refspecs via configuration.
27+
28+
A commit that cannot be fetched from a reachable remote is a terminal invalid request, not a conflict. If the remote itself is unreachable the error stays retryable.
29+
30+
## Staleness
31+
32+
Fetching by SHA guarantees the merger applies exactly the commit a URI names — not that the commit is still the change's head. A force-push leaves the superseded commit fetchable for some time on most hosts, so a successful fetch says nothing about freshness. When enabled, each change's canonical ref is read (one ref advertisement, no object transfer) and a mismatch is terminal. A ref that no longer exists yields no verdict.
33+
34+
## Strategies
35+
36+
| Strategy | What it does | Outputs |
37+
|---|---|---|
38+
| `REBASE` | Cherry-picks every commit each change introduces onto the tip, in order. A commit already present on the target is skipped (no output), as is one that was empty to begin with. | one revision per newly-created commit |
39+
| `DEFAULT` | Resolved to the instance's configured default strategy before any step runs. | per the resolved strategy |
40+
41+
`REBASE` is the only strategy implemented so far. `SQUASH_REBASE`, `MERGE`, and `PROMOTE` are defined by the wire contract but not yet applied here — a step naming one is rejected as an invalid request.
42+
43+
## Committing, dry-run, atomicity, contention
44+
45+
`Merge` commits and reports outputs; `CheckMergeability` runs the identical apply but never pushes, then resets the checkout to discard the local commits and reports empty outputs. A multi-step check commits its intermediate steps locally so it sees the same conflict surface a real merge would.
46+
47+
For a committing merge nothing reaches the remote until the final push. A step that fails to apply aborts its in-progress git operation and returns without pushing. If the push fails because the remote tip moved between reset and push, the whole reset/apply/push cycle is retried up to a bounded number of attempts; detection re-fetches the tip and compares it to the SHA the cycle was based on.
48+
49+
## Failure classification
50+
51+
A merge conflict surfaces as `merger.ErrConflict`. An unusable request surfaces as `merger.ErrInvalidRequest`: an unsupported strategy or URI scheme, a malformed URI, a commit a reachable remote cannot supply, a change whose head has moved on, or a change sharing no history with the target under a picking strategy. Both are terminal — the controller publishes a `FAILED` result rather than retrying. Everything else (network/auth/push faults, and an unreachable remote) is returned as a plain error for the consumer to retry.
52+
53+
The distinction between the last two matters operationally: a commit that is missing while the remote answers is a property of the request, whereas a remote that will not answer is a property of the moment.
54+
55+
## Runtime and identity
56+
57+
Every git invocation uses the pinned runtime (explicit executable, exec-path, and template dir) and a scrubbed environment: no ambient configuration, no system or global git config, no interactive prompts. Because that leaves no ambient identity, the committer name and email are injected per-invocation, which the commit-creating `REBASE` strategy requires.
58+
59+
See [Object availability](#object-availability) for how referenced commits are obtained.
Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
// Copyright (c) 2025 Uber Technologies, Inc.
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// http://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
package git
16+
17+
import (
18+
"fmt"
19+
"strings"
20+
21+
entitygit "github.com/uber/submitqueue/platform/base/change/git"
22+
entitygithub "github.com/uber/submitqueue/platform/base/change/github"
23+
"github.com/uber/submitqueue/runway/extension/merger"
24+
)
25+
26+
// changeRef is everything the merger needs from a change URI, reduced to a
27+
// form that does not depend on which provider minted it. Adding a provider
28+
// means adding one case to resolveChange, not touching the apply paths.
29+
type changeRef struct {
30+
// SHA is the full commit hash the URI pins the change to. This is the
31+
// commit that gets fetched and applied.
32+
SHA string
33+
// Ref is the fully-qualified ref the provider publishes this change's head
34+
// under (e.g. "refs/pull/12/head"). Used as a fetch fallback and as the
35+
// staleness comparison point. Empty when the scheme has no such ref, in
36+
// which case both are skipped.
37+
Ref string
38+
// Label is a short human-readable identifier for the change, used in
39+
// commit messages the merger synthesizes.
40+
Label string
41+
}
42+
43+
// resolveChange maps a change URI onto the merger's provider-neutral view of
44+
// it. An unrecognized scheme is terminal: no retry produces a parser for it.
45+
func resolveChange(uri string) (changeRef, error) {
46+
scheme, _, ok := strings.Cut(uri, "://")
47+
if !ok {
48+
return changeRef{}, fmt.Errorf("%w: change URI %q has no scheme", merger.ErrInvalidRequest, uri)
49+
}
50+
51+
switch scheme {
52+
case "github":
53+
cid, err := entitygithub.ParseChangeID(uri)
54+
if err != nil {
55+
return changeRef{}, fmt.Errorf("%w: invalid change URI %q: %v", merger.ErrInvalidRequest, uri, err)
56+
}
57+
return changeRef{
58+
SHA: cid.HeadCommitSHA,
59+
// GitHub publishes every PR's head under refs/pull/<n>/head in the
60+
// base repository, including PRs opened from a fork.
61+
Ref: fmt.Sprintf("refs/pull/%d/head", cid.PRNumber),
62+
Label: fmt.Sprintf("%s#%d", cid.OwnerRepo(), cid.PRNumber),
63+
}, nil
64+
65+
case "git":
66+
cid, err := entitygit.ParseChangeID(uri)
67+
if err != nil {
68+
return changeRef{}, fmt.Errorf("%w: invalid change URI %q: %v", merger.ErrInvalidRequest, uri, err)
69+
}
70+
// A git:// URI already names its own fully-qualified ref, so the
71+
// staleness check reads exactly the ref the caller pinned.
72+
return changeRef{
73+
SHA: cid.CommitSHA,
74+
Ref: cid.Ref,
75+
Label: fmt.Sprintf("%s@%s", cid.Repo, cid.Ref),
76+
}, nil
77+
78+
default:
79+
return changeRef{}, fmt.Errorf("%w: unsupported change URI scheme %q in %q", merger.ErrInvalidRequest, scheme, uri)
80+
}
81+
}
82+
83+
// resolveStepChanges resolves every URI of every step, in application order.
84+
func resolveStepChanges(steps []resolvedStep) ([]changeRef, error) {
85+
var refs []changeRef
86+
for _, rs := range steps {
87+
for _, uri := range rs.step.GetChange().GetUris() {
88+
ref, err := resolveChange(uri)
89+
if err != nil {
90+
return nil, err
91+
}
92+
refs = append(refs, ref)
93+
}
94+
}
95+
return refs, nil
96+
}

0 commit comments

Comments
 (0)