spec(SPEC-BPE-QUADRATIC-MERGE): the merge loop is quadratic before the length check, and the growth gate is dropped

CMakeLists.txt

@@ -2399,6 +2399,34 @@ target_include_directories(vllm_music3_vocoder_conv_ab SYSTEM PRIVATE
 target_compile_features(vllm_music3_vocoder_conv_ab PRIVATE cxx_std_20)
 vllm_cpp_set_warnings(vllm_music3_vocoder_conv_ab)
 
+# ── BIT-ROT GUARD: the BPE encode-cost harness (#1365) ───────────────────────
+# `tools/bench/bpe_encode_cost.cpp` is the artifact behind the timing figures in
+# `.agents/specs/bpe-quadratic-merge.md`. It exists because three earlier
+# harnesses for that row lived in prose only, so no reader could re-derive one
+# figure and three rounds of certified constants each failed to reproduce.
+#
+# CI COMPILES IT AND RUNS NOTHING, and that is not an oversight. The row's own
+# ruling is that a growth assertion over these figures is NOT viable on a shared
+# runner, so there is nothing here for CI to decide; and one leg on the O(n^2)
+# code costs tens of seconds of one core, which no test tier should spend. What
+# CI does buy is that the file cannot rot behind a `Tokenizer::Encode` or
+# `FromHfJson` signature change while still being the only artifact a reader can
+# reproduce the row's numbers from — the #1246 failure, where the sole named
+# harness for a measurement was compiled by nothing.
+#
+# An OBJECT library, like `vllm_music3_depth_stage_ab_*` above and unlike
+# `vllm_music3_vocoder_conv_ab`: the recorded run recipe in the file's own header
+# compiles the four tokenizer translation units directly with `g++` rather than
+# linking `libvllm.a`, so a linked target here would guard nothing the compile
+# does not and would only add an entry point nobody calls.
+add_library(vllm_bpe_encode_cost OBJECT tools/bench/bpe_encode_cost.cpp)
+target_include_directories(vllm_bpe_encode_cost PRIVATE
+  ${CMAKE_SOURCE_DIR}/include ${CMAKE_SOURCE_DIR}/src ${CMAKE_BINARY_DIR}/include)
+target_include_directories(vllm_bpe_encode_cost SYSTEM PRIVATE
+  ${CMAKE_SOURCE_DIR}/third_party)
+target_compile_features(vllm_bpe_encode_cost PRIVATE cxx_std_20)
+vllm_cpp_set_warnings(vllm_bpe_encode_cost)
+
 # ── Packaged shared/static libvllm (M3.5 Task 3, library-first packaging) ─────
 # The static `vllm` core lib above stays as-is (tests + examples link it and
 # reach internal C++ symbols). Here we ALSO package a SHARED libvllm.so that

docs/USAGE.md

@@ -471,6 +471,29 @@ two edits back gives the GPT-4o row above, so the two share one scanner's
 character classes but stay separate patterns: they disagree on `don't` and on
 every digit run longer than one.
 
+### Timing an encode on your own box
+
+`tools/bench/bpe_encode_cost.cpp` times `Tokenizer::Encode` on one synthetic
+input, at the sizes you name, through a `tokenizer.json` you name. Use it when
+you want to know what a prompt of some shape costs to tokenize here, or to
+re-derive a figure somebody else recorded instead of trusting it.
+
+Nothing RUNS it: it is registered as no test and it is not a gate. Both halves
+of that are deliberate — a growth ratio over these timings is not stable enough
+to gate on a shared machine, and one leg on a long single-class input can cost
+tens of seconds of one core. It IS compiled, as the never-linked OBJECT library
+`vllm_bpe_encode_cost`, so it cannot rot behind a `Tokenizer::Encode` or
+`FromHfJson` signature change while still being the artifact those figures are
+reproducible from. Its header carries the exact `g++` and run lines; it builds
+from the four tokenizer translation units directly and needs no `libvllm.a`.
+
+It prints one row per case and size, with the ids it produced and the
+1/5/15-minute load average sampled around each row, under a banner saying the
+output is a session reading and not a bound. Read it that way: on a 20-core box
+the same input on the same binary has read 1.7x apart on load alone, while the
+id counts came back identical. Quote a number from it only with its load beside
+it, and take the minimum of several repetitions rather than one shot.
+
 ### How much memory a Vulkan load needs
 
 On a unified-memory device (a DGX Spark) the Vulkan heap and system RAM are the

scripts/check-agent-record.py

@@ -610,8 +610,29 @@
 # implementation, `model_loader.cpp:279-303`, reads an existing cache for the
 # DFlash draft alone and never downloads. `READY`, spec
 # `specs/hf-model-download.md`, issue #1280.
+# 168 since 2026-08-19: +`SPEC-BPE-QUADRATIC-MERGE` (the BPE merge loop is O(n^2)
+# in pretoken length, on the request path, before `ValidatePromptLen`). Genuinely
+# new and not expressible by the two tokenizer rows beside it: `LOAD-HF-BPE` and
+# `LOAD-SENTENCEPIECE` both own a FORMAT -- which `tokenizer.json` shapes parse and
+# which token identifiers come out -- and both are token-exact against HF goldens
+# today and stay that way. This row changes no identifier at all. It replaces the
+# algorithm underneath both of them, and its gate is a COST bound, which is the one
+# thing a token gate provably cannot see. It is also not a benchmark row: the encode
+# runs synchronously on the HTTP worker five lines before the only length check, so
+# `max_model_len` bounds none of it and `/tokenize` reaches it with no engine.
+# MEASURED, and stated as the two SESSION-INVARIANT quantities only: over 1 KB to
+# 64 KB of ordinary English prose the fit through the committed Mistral golden has
+# exponent 2.01, and at 64 KB our cost is 2,507x HF `tokenizers` 0.22.2's on the
+# same file for byte-identical identifiers. Both are ratios taken inside one
+# session, so contention cancels. The ABSOLUTE milliseconds are deliberately not
+# repeated here: they moved 54% between two runs of one binary on one input, so a
+# constant copied into this comment would be a fourth place for a number nobody
+# can reproduce to drift. They live in the spec's tables, each beside its own load
+# average, and `## Gates` owes the idle-host re-measure. The exponent, not the
+# constant, is what makes this a row.
+# `READY`, spec `specs/bpe-quadratic-merge.md`, issue #1365.
 # Bumped for a real new row, never to make a failing state transition pass.
-ENGINE_ROWS = 167
+ENGINE_ROWS = 168
 
 ENGINE_SUMMARY_SECTIONS = (
     ("Engine and scheduling", "Engine core and scheduling"),

scripts/check-gate-commands.py

@@ -460,6 +460,16 @@ def audit() -> list[dict]:
     "SERVE-ASYNC-LLM",
     "SERVE-STREAM-USAGE",
     "TOOLS-STREAMING-PARSER",
+    # SPEC-BPE-QUADRATIC-MERGE (2026-08-19, issue #1365): GROWTH, re-pinned in
+    # the same change. The row was gated and command-free; its spec's `## Gates`
+    # now carries the `g++` build and the two run lines for
+    # `tools/bench/bpe_encode_cost.cpp`, the harness that replaces three
+    # prose-only ones, so the section names a command that can fail and the row
+    # joins this population. Named here so the growth is not silent: the harness
+    # itself is deliberately NOT a gate -- the row rules that a growth assertion
+    # over its figures is not viable on a shared runner -- and what became
+    # runnable is the RECIPE the section owes under AGENTS.md §Gates.
+    "SPEC-BPE-QUADRATIC-MERGE",
 })
 
 

tests/scripts/test_agent_record.py

@@ -1730,5 +1730,92 @@ def test_the_pin_agrees_with_the_matrix_it_counts(self) -> None:
         self.assertIn(self.ROW, {r.item_id for r in engine})
 
 
+class BpeQuadraticMergeRowIsCounted(unittest.TestCase):
+    """The ENGINE ratchet bump 167 -> 168 is backed by a real row (#1365).
+
+    Same shape and the same reason as `HfModelDownloadRowIsCounted`, applied to
+    the pin this change moves. `ENGINE_ROWS` is re-pinned by hand, so a bump
+    with nothing behind it looks exactly like a bump for a row that landed.
+    `test_engine_row_ratchet_is_load_bearing` proves the pin BINDS by moving it,
+    which holds for any value and cannot say whether 168 is the right one. This
+    class says that, by tying the pin to the row the matrix carries.
+
+    This class asserts nothing about `.agents/issue-index.md`, where the sibling
+    classes assert their issue number, and this change appends no row there.
+    #1365's row already landed in `9e1a5e573` and a second row for one issue
+    number is what `check-agent-record.py` reports as `issue #1365 listed
+    twice`. The row's TEXT is stale, because #1365 was re-scoped in place from
+    the symptom onto the cause after the row landed, so `assertIn("issues/1365)",
+    index)` would pass here against a row describing the symptom and would
+    measure nothing about this row's work. The staleness is recorded in the
+    spec's `## Dependencies` instead, where prose can say it.
+    """
+
+    ROW = "SPEC-BPE-QUADRATIC-MERGE"
+
+    def test_the_row_exists_in_the_engine_matrix(self) -> None:
+        text = (ROOT / ".agents/engine-matrix.md").read_text(encoding="utf-8")
+        matching = [
+            line for line in text.splitlines() if line.startswith(f"| `{self.ROW}` |")
+        ]
+        self.assertEqual(len(matching), 1, f"{self.ROW} must appear exactly once")
+
+    def test_the_row_names_its_spec(self) -> None:
+        """A row whose spec lives only in the PR body is untraceable."""
+        text = (ROOT / ".agents/engine-matrix.md").read_text(encoding="utf-8")
+        row = next(l for l in text.splitlines() if l.startswith(f"| `{self.ROW}` |"))
+        self.assertIn("bpe-quadratic-merge.md", row)
+        self.assertTrue(
+            (ROOT / ".agents/specs/bpe-quadratic-merge.md").is_file(),
+            "the spec the row cites must exist",
+        )
+
+    def test_the_engine_pin_is_load_bearing_for_this_row(self) -> None:
+        """MUTATION: with this row removed, the pinned count must disagree.
+
+        Redirects only the ENGINE matrix at a mutated copy on disk, for the
+        reason `TenstorrentMistralRowIsCounted` records: patching `read_text`
+        globally would feed engine content to every matrix, and this case would
+        then pass on errors that have nothing to do with the removal.
+        """
+        clean: list[str] = []
+        agent_record.check_matrices(clean)
+        self.assertEqual([e for e in clean if "engine rows" in e], [])
+
+        path = agent_record.ENGINE_MATRIX
+        text = path.read_text(encoding="utf-8")
+        without = "\n".join(
+            l for l in text.splitlines() if not l.startswith(f"| `{self.ROW}` |")
+        )
+        self.assertNotEqual(without, text, "the row must be present to remove")
+
+        with tempfile.TemporaryDirectory(dir=agent_record.ROOT) as tmp:
+            mutated = Path(tmp) / "engine-matrix.md"
+            mutated.write_text(without, encoding="utf-8")
+            paths = [mutated if q == path else q for q in agent_record.MATRIX_PATHS]
+            errors: list[str] = []
+            with mock.patch.object(agent_record, "MATRIX_PATHS", paths), \
+                 mock.patch.object(agent_record, "ENGINE_MATRIX", mutated):
+                agent_record.check_matrices(errors)
+        self.assertTrue(
+            any("engine rows" in e for e in errors),
+            f"removing {self.ROW} must break the engine count; got {errors}",
+        )
+
+    def test_the_pin_agrees_with_the_matrix_it_counts(self) -> None:
+        """MUTATION TARGET: `ENGINE_ROWS` back at 167 must be an error.
+
+        The pin and the matrix are two hand-maintained records of one number.
+        This asserts they agree at the value this change lands, so lowering the
+        constant to the previous 167 while the row is present reds here.
+        """
+        errors: list[str] = []
+        rows, _ = agent_record.check_matrices(errors)
+        self.assertEqual([e for e in errors if "engine rows" in e], [])
+        engine = [r for r in rows if r.path == agent_record.ENGINE_MATRIX]
+        self.assertEqual(len(engine), agent_record.ENGINE_ROWS)
+        self.assertIn(self.ROW, {r.item_id for r in engine})
+
+
 if __name__ == "__main__":
     unittest.main()

tests/scripts/test_check_gate_commands.py

@@ -815,6 +815,27 @@ def test_hf_model_download_earns_its_runnable_baseline_entry(self):
         self.assertEqual(runnable - reduced, {"ENG-HF-MODEL-DOWNLOAD"})
         self.assertEqual(runnable, set(gates.RUNNABLE_BASELINE))
 
+    def test_bpe_quadratic_merge_earns_its_runnable_baseline_entry(self):
+        # SPEC-BPE-QUADRATIC-MERGE (#1365) was already gated and already carried
+        # a `## Gates` section; what changed is that the section now names the
+        # `g++` build and the run lines for `tools/bench/bpe_encode_cost.cpp`.
+        # So this is growth WITHIN the population, not arrival into it, and the
+        # baseline grows by one either way.
+        #
+        # Same shape and same reason as the two rows above: set equality holds
+        # for any membership and cannot say this row belongs, so this case says
+        # it by removing the entry and requiring the equality to go red. Without
+        # it, an entry added to quiet a red gate is indistinguishable from an
+        # entry added because a row gained a command.
+        verdicts = {r["id"]: r["verdict"] for r in gates.audit()}
+        self.assertEqual(verdicts.get("SPEC-BPE-QUADRATIC-MERGE"), "runnable")
+        reduced = set(gates.RUNNABLE_BASELINE) - {"SPEC-BPE-QUADRATIC-MERGE"}
+        self.assertNotEqual(reduced, set(gates.RUNNABLE_BASELINE))
+        runnable = {r["id"] for r in gates.audit() if r["verdict"] == "runnable"}
+        self.assertNotEqual(runnable, reduced)
+        self.assertEqual(runnable - reduced, {"SPEC-BPE-QUADRATIC-MERGE"})
+        self.assertEqual(runnable, set(gates.RUNNABLE_BASELINE))
+
 
 if __name__ == "__main__":
     unittest.main()