feat(speculation): generator contract and bestfirst impl

Add submitqueue/extension/speculation/generator, the candidate-stream composition point (Generator/PathIterator) the standard Speculator pulls from, plus the bestfirst implementation and mocks.

bestfirst ranks each path by the product of its dependencies' landing chances (scored through an injected scorer) and generates lazily at two levels. A cross-head heap holds one real, non-terminal candidate per head — never an optimistic upper bound — so popping it yields the globally best-ranked path; each head is itself a lazy stream that materializes its next candidate only once the current one is consumed. Pulling k candidates materializes O(k) paths however large the space behind them, so a head with twelve unresolved dependencies costs no more at the front of the queue than one with two.

Within a head, the best path takes the preferred (likelier) bet on every unresolved dependency, and every other path flips some subset of those choices, each flip multiplying the score by that choice's penalty (alternative over preferred). Subsets are walked by a canonical expansion under which every subset has exactly one parent, so each combination is generated exactly once with no visited set, and a child never outranks its parent — which is what lets the heap pop in true descending order.

Also corrects the generator README: the leading path is the preferred-bet path, not the all-included one. The two differ whenever a dependency is likelier to fail than to land, so the old wording was wrong for any below-even dependency.

The two heaps (across heads, and within a head) order candidates identically, so they share one generic rankedHeap over the standard library's container/heap rather than repeating the heap.Interface boilerplate twice.

Test coverage is 22 cases, including the exact emitted sequence across heads, the preferred bet on a below-even dependency, mixed pinned dependencies dropping out of the search, terminal filtering across every terminal status (and deliberately not filtering the non-terminal ones), the materialization counts behind the laziness claim, deterministic tie-breaking, and a randomized cross-check of the whole walk against brute-force enumeration. The generator README gains a worked walkthrough tracing the frontier pop by pop.

A head over the depth bound is now recognised from its dependencies' states alone, before anything is scored. Previously every unfinished dependency was scored and only then was the head found to be over the bound. Scoring can be expensive and a head we skip should not pay for it. One visible consequence: a scorer error on a head that was going to be skipped no longer fails Open.

Author: behinddwalls

submitqueue/extension/speculation/generator/BUILD.bazel

@@ -0,0 +1,9 @@
+load("@rules_go//go:def.bzl", "go_library")
+
+go_library(
+    name = "go_default_library",
+    srcs = ["generator.go"],
+    importpath = "github.com/uber/submitqueue/submitqueue/extension/speculation/generator",
+    visibility = ["//visibility:public"],
+    deps = ["//submitqueue/entity:go_default_library"],
+)

submitqueue/extension/speculation/generator/README.md

@@ -0,0 +1,19 @@
+# generator
+
+The `generator` package is a piece the `standard` `Speculator` is built from: a `Generator` produces the queue's candidate paths as one stream, best first, across all heads. It is **not** controller-facing — the speculate controller only knows the `Speculator` contract, and a different `Speculator` need not split its work this way. So there is no `Config` or `Factory` here; a `Generator` is chosen when the `standard` `Speculator` is constructed.
+
+`Open` starts the stream over the queue's live batches and their path sets and returns a `PathIterator`. The caller pulls one candidate at a time and the generator does only the work that answer needs. Candidates descend in score, never repeat, and never contradict a known fact. A candidate's score means something only for the current run; scores are never stored.
+
+## `bestfirst`
+
+A head waiting on unfinished dependencies has one path per combination of guesses — 2ⁿ of them — while callers want the best few. `bestfirst` hands them out in score order without building the rest.
+
+Dependencies that already finished are not guesses: landed means included, failed or cancelled means excluded. They stay in the path but drop out of the search. Each unfinished one is a two-way guess whose chance of landing comes from an injected `scorer.Scorer`, remembered per dependency so one shared by several heads is scored once; a dependency that is not a live batch is treated as a coin flip. A path's score is its guesses' chances multiplied together. Heads with more unfinished dependencies than the depth bound are skipped until some resolve, and paths that already finished are passed over — so a failed path drops out and the next-best takes its place.
+
+**The best path is not "bet everything lands."** Each guess takes the *likelier* outcome, so a dependency that will probably fail is excluded. Every other path flips some of those guesses, and each flip costs a known factor. One consequence worth knowing: flipping two cheap guesses can beat flipping one expensive guess, and the ordering handles that correctly.
+
+**Laziness at two levels.** Across heads, a heap holds each head's current offer — a real, already-built path, never an estimate — so the top of the heap is the best path in the queue. Within a head the same shape repeats: it keeps a small heap, hands out its best, and builds only the one or two paths that come after it. Pulling *k* candidates builds about *k* paths, whatever the size of the set behind them.
+
+`bestfirst` discards nothing: pull long enough and every path within the depth bound comes out. It does no conflict relaxation (`dropped` bets) — that is the `Speculator`'s call, not the generator's.
+
+The ordering trick behind all this — how the walk reaches every path exactly once, in score order, without tracking what it has already seen — is documented with a worked example on `expand` in `bestfirst.go`. The behavior is pinned by tests in `bestfirst_test.go`.

submitqueue/extension/speculation/generator/bestfirst/BUILD.bazel

@@ -0,0 +1,26 @@
+load("@rules_go//go:def.bzl", "go_library", "go_test")
+
+go_library(
+    name = "go_default_library",
+    srcs = ["bestfirst.go"],
+    importpath = "github.com/uber/submitqueue/submitqueue/extension/speculation/generator/bestfirst",
+    visibility = ["//visibility:public"],
+    deps = [
+        "//submitqueue/entity:go_default_library",
+        "//submitqueue/extension/scorer:go_default_library",
+        "//submitqueue/extension/speculation/generator:go_default_library",
+    ],
+)
+
+go_test(
+    name = "go_default_test",
+    srcs = ["bestfirst_test.go"],
+    embed = [":go_default_library"],
+    deps = [
+        "//submitqueue/entity:go_default_library",
+        "//submitqueue/extension/scorer:go_default_library",
+        "//submitqueue/extension/speculation/generator:go_default_library",
+        "@com_github_stretchr_testify//assert:go_default_library",
+        "@com_github_stretchr_testify//require:go_default_library",
+    ],
+)