From 03f0a8f5bf2c96015009f512b87088a743619e4a Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 19 Aug 2026 17:13:13 +0000 Subject: [PATCH 1/4] spec(SPEC-BPE-QUADRATIC-MERGE): the merge loop is quadratic in pretoken length, and one long prompt is a denial of service (#1365) leg". The cause is measured here and it is not a benchmark curiosity. `src/vllm/tokenizer/bpe.cpp::BpeMerge` rescans every adjacent pair per merge, on the premise its own comment states: "O(n^2) scan; pretokens are tiny". Nothing bounds a pretoken. Four of the seven pretokenizer rules return an unbounded run, and on the SentencePiece family `EncodePlainSp` merges the whole prompt as one word because Mistral and Gemma declare `split: false`. Measured at 31f93787c on one core: 65,535 bytes of one repeated character costs 28.1 s, 65,535 ASCII spaces cost 48.1 s, and 64 KB of ordinary English prose through the committed Mistral golden costs 24.1 s, growing n^2.0. The encode runs synchronously on the HTTP worker at input_processor.cpp:260, five lines before ValidatePromptLen at :265, so max_model_len bounds none of it, and /tokenize reaches the same path with no engine at all. That is a remote denial of service in a shipped path, so the issue is re-titled to say so. Spec before code, and this one needs agreement before anyone edits a token-exact path. The design mirrors HF tokenizers 0.22.2 Word::merge_all: a heap over candidate merges and an identifier-keyed merge table. Two shortcuts are closed in writing. A pretoken cap is refused because it changes the token identifiers we emit. Removing the per-probe std::string alone was measured rather than assumed, and it is slower than the current code on the largest case, so the allocation is not the cost. The issue index is deliberately untouched. It is append-only, #1365 already has a row, and a second row for the same number is what check-agent-record.py reports as "issue #1365 listed twice". ENGINE_ROWS moves 167 -> 168 for the new row, with the justification the neighbouring entries carry. That 15-line comment shifted every line below it in check-agent-record.py, which staled the five anchors ENG-RECORD-ANCHOR-RATCHET cites into that file. The ratchet caught its own file moving and the five anchors are repaired in the same change. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:Opus-5 [Claude Code] --- .agents/engine-matrix.md | 7 +- .agents/specs/bpe-quadratic-merge.md | 457 +++++++++++++++++++++++++++ scripts/check-agent-record.py | 16 +- 3 files changed, 476 insertions(+), 4 deletions(-) create mode 100644 .agents/specs/bpe-quadratic-merge.md diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index 236c960ac..7a42aa77b 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -47,8 +47,8 @@ forensics: roadmap_v1.md and the parity ledger. | Serving, API, CLI, library | 35 | 10 | 2 | 0 | 3 | 11 | 2 | 3 | 4 | | LoRA and adapters | 2 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 1 | | Long context and attention | 10 | 5 | 0 | 0 | 1 | 0 | 1 | 0 | 3 | -| Loading, tokenizer, config | 12 | 3 | 3 | 0 | 2 | 1 | 1 | 1 | 1 | -| **Total** | **167** | **35** | **17** | **4** | **14** | **39** | **8** | **11** | **38** | +| Loading, tokenizer, config | 13 | 3 | 3 | 0 | 3 | 1 | 1 | 1 | 1 | +| **Total** | **168** | **35** | **17** | **4** | **15** | **39** | **8** | **11** | **38** | ## Engine core and scheduling @@ -224,7 +224,7 @@ claims it. | `ENG-RELEASE-WINDOWS` | Native Windows x86_64 pre-alpha release extension: one adaptive MSVC/UCRT CPU bundle with AVX2 executed in CI and one Vulkan preview bundle, both deterministic ZIPs and authenticated by the existing release handoff | T0 | vLLM has no Windows release path; runtime behavior remains pinned to vLLM `555967922`. Platform substrate reference: llama.cpp `src/llama-mmap.cpp:520-590` @ `237ad9b961f009ae19ac29dbce4cd0c1251f94b3`; Win32 API is the OS authority | W14 Win32 portability/MSVC CPU, W15 deterministic ZIP/PE packaging + Vulkan, and W16 ten-tuple prerelease workflow/version/docs implemented for one PR | Linux portability/release mutation gates are local evidence only. Native `windows-2022` MSVC `/W4 /WX`, extracted runtime/ISA smokes, merged-SHA ten-tuple dry run, `v0.0.3-pre.1` publication, attestations, and exact 32-asset audit remain pending; no Windows ZIP exists yet | [windows-binary-release.md](specs/windows-binary-release.md); [#117](https://github.com/mudler/vllm.cpp/issues/117) | `ACTIVE` | `CLAIM-ENG-RELEASE-WINDOWS` | | `ENG-RELEASE-CONTAINERS` | Published OCI container images on GHCR, built by GitHub Actions: the same staged server bundle as `ENG-RELEASE-BINARIES`, shipped from one package `ghcr.io/mudler/vllm.cpp` with the lane in the tag — `:-cuda` / `-vulkan` / `-cpu`, the moving `:latest-cuda` / `:latest-vulkan` / `:latest-cpu`, and a bare `:latest` aliasing the cpu lane, with `ENTRYPOINT vllm-server`. Lanes `cuda` (one fat image covering every supported SM), `vulkan`, `cpu` (adaptive baseline); `rocm` blocked-preview, tracking its binary channel. Version tags are immutable; every `latest-` moves. Each lane is a `linux/amd64` + `linux/arm64` multi-arch manifest built on native runners — aarch64 is first-class here because GB10 (sm_121a), Thor (sm_110) and Orin (sm_87) are all arm64. The image contains the bundle and nothing else: no weights, no Python, no PyTorch, no compiler, no build tree. BOUNDARY: the GPU driver and container runtime stay on the host and are never bundled; Metal and MLX are NOT-CONTAINERIZABLE (no macOS container runtime and no Metal passthrough exists) and remain static-binary-only lanes, recorded as a permanent boundary rather than pending work. No image, workflow, registry package or pull is claimed to exist. | T0 | release image lanes `.buildkite/release-pipeline.yaml:34-170` and the published-image dependency boundary `docker/Dockerfile.cpu:262-290` @ `555967922` | `docker/Dockerfile` (cpu/vulkan/cuda targets calling the release scripts); `docker/healthcheck.sh`; `release/container-matrix.json`; `scripts/check-container-matrix.py`; `scripts/check-container-workflow.py`; `scripts/validate-container-image.py`; `scripts/container_tags.py`; `.github/workflows/containers.yml`; SIGTERM handler `src/vllm/entrypoints/openai/server_main.cpp` (`SignalShutdown`, all three `listen()` sites); the pre-existing `docker/Dockerfile.arm64` is an unrelated CPU bench cross-check | issues `#170`, `#312`, `#394`; `tests/scripts/test_check_container_matrix.py` 31/31; `test_check_container_workflow.py` 29/29; `test_check_cuda_fat_gencode.py` 7+4 subtests. **GB10 2026-08-11 (`promaxgb10-4ad8`, `sm_121a`, CUDA 13.3): arm64 cuda image 1.71 GB, 673/673 objects, ten-SM gencode audit PASS, and a REAL GPU boot -- `/health` 200, `/version` 200, in-container healthcheck, clean SIGTERM, `--gpus all`, host driver 580.159.03 injected.** cpu amd64 783 MB gated locally; cpu+vulkan amd64 green on hosted CI **arm64 cuda lane RUNTIME-VERIFIED on GB10 2026-08-11** -- the first accelerator-hardware evidence for any lane. Four defects were removed to get there, each found by building rather than reading: the CUDA 12.9 base could not compile `sm_110`, the BuildKit cache mount outlived its toolchain (both #366), Marlin gencode had drifted from the feature table and failed the audit on 14 correctly-compiled TUs (#394, blocking BOTH cuda tuples project-wide), and the validator could only ever produce build evidence because its boot smoke never passed `--gpus`. **NOT established: nothing is published to GHCR; amd64 cuda is unbuilt; the published arm64 image is SBSA (`targets/sbsa-linux`), so Tegra -- Thor `sm_110`, Orin `sm_87` -- is untested and NOT covered** **ORIN (Tegra) 2026-08-11: the SBSA image RUNS on Jetson AGX Orin `sm_87` (L4T R36.4.3, Docker 27.5.1) -- Qwen3-0.6B (rev `c1899de2`) loads and GENERATES via `/v1/completions`, tegrastats GR3D 95-97% during decode vs 14-15% idle.** Tegra needs `--runtime nvidia --gpus all`: `--gpus` alone is refused by the hook and `--runtime` alone mounts no driver | [container-images.md](specs/container-images.md); issues [#170](https://github.com/mudler/vllm.cpp/issues/170), [#312](https://github.com/mudler/vllm.cpp/issues/312), [#394](https://github.com/mudler/vllm.cpp/issues/394) | `ACTIVE` | `CLAIM-ENG-RELEASE-CONTAINERS-W1-W7` | | `ENG-DOCS-SITE` | Publish the 11 `docs/*.md` as a browsable GitHub Pages site at `https://mudler.github.io/vllm.cpp/` WITHOUT a second copy of the prose. A Hugo site at `website/` mounts `../docs` READ-ONLY and derives everything else from what is already in the files: each page title from the file's first `# H1`, the sidebar order from `website/data/nav.yaml`, and links through a Goldmark render hook (internal `.md` → site URL; the 139 `../.agents/**` and `../AGENTS.md` escapes → GitHub blob URLs, since the protocol tree is deliberately NOT published). **No file under `docs/` is modified, moved, renamed, or given front matter**, so `check-doc-checkpoint.py` and every protocol path reference keep working and there is no second surface that can drift — the whole point of the row. Custom lean layouts, NO theme and NO submodule: off-the-shelf docs themes read titles, weights and menus out of front matter this design deliberately does not have, so each would need its title partial, menu and link hook overridden anyway, and hugo-book additionally floors at Hugo 0.158 against the 0.146.3 pin CI and the local toolchain share. Hard prerequisite inside the repo: `classify_path` in `scripts/check-pr-size.py` FAILS CLOSED on `website/**` (verified: raises `ValueError: unclassified repository path`), so the classifier must learn the path or the PR cannot pass the project's own size gate. Hard prerequisite outside it: GitHub Pages must be enabled with the source set to GitHub Actions — the workflow is inert otherwise. A marketing landing page is explicitly OUT of scope (`README.md` stays the front door), as is any restructuring of `docs/`; the custom domain is parked behind the pending vLLM trademark question | T1 | NO vLLM analogue — upstream's docs are a separate mkdocs site and nothing in this row mirrors upstream *behavior*, so it carries no parity obligation. The STRUCTURAL reference is LocalAI's `.github/workflows/gh-pages.yml` (two Hugo sites merged into one Pages artifact), reduced to the docs half | read-only mount `website/hugo.toml:29`; title-from-H1 `website/layouts/partials/title.html:10`; link rewriting `website/layouts/_default/_markup/render-link.html:27`; guard `scripts/check-site.py:70`; deploy `.github/workflows/gh-pages.yml` | `tests/scripts/test_check_site.py:51,56,66,80,89,97` (6 mutation cases: clean tree, H1 stripped, doc absent from nav, nav entry with no file, duplicated entry, missing nav file); build evidence 14 pages with `docs/bench-evidence` + `docs/superpowers` absent from `public/` and no `href` ending in `.md`; 48 protocol links rewritten in `docs/status/`. NO published page is claimed: GitHub Pages is not yet enabled on the repository, which is the recorded stop condition holding this row at `GATING` | [gh-pages-docs-site.md](specs/gh-pages-docs-site.md); issue [#224](https://github.com/mudler/vllm.cpp/issues/224) | `READY` | `CLAIM-ENG-DOCS-SITE` | -| `ENG-RECORD-ANCHOR-RATCHET` | **The record's `path:line` citations were range-checked and never reported.** `check-agent-record.py` parsed BOTH forms: markdown links, and bare `` `file.cpp:123` `` through `RAW_LOCAL_ANCHOR_RE` since `ee511ca8a`. On a missing file or an out-of-range line `local_line_anchors` runs `continue`, so the bad anchor never reaches the caller, and `is_code_anchor` then answers with **any**, so one good sibling covers the rest. There was no symbol test and no report, and **32 of the 38** offenders are IN RANGE, so range-checking could not have found them. Measured at `8daa67b39`: **832 of 867** in-scope citations (**96.0%**) were already parsed and range-checked, and the **35** new to parsing sit under `.agents/`, `docs/` and `website/`; `EVIDENCED_STATES` omits `ACTIVE`/`READY` entirely and is deliberately NOT widened, because requiring an anchor there raises 85 errors across 53 rows. Even the fraction it saw was only range-checked, never checked to CONTAIN the symbol named beside it — every stale anchor found in the 2026-08-13/14 campaign was in range. LANDED as a device-leakage-shaped ratchet over a recorded baseline, never a bulk cleanup: the backlog is fixed by whoever next touches each row | T1 | none — this is our own record surface; the discipline mirrors AGENTS.md §Records ("cite the `file:line` you ported from") | parser + classifier + ratchet in `check-agent-record.py`: `BARE_CITATION_RE` `scripts/check-agent-record.py:1237` (the bare form), `cell_citations` `scripts/check-agent-record.py:1291` (both forms, with the adjacent-symbol rule), `classify_citation` `scripts/check-agent-record.py:1348` (OK / STALE / BROKEN), `RECORD_ANCHOR_STATES` `scripts/check-agent-record.py:1235` (gap 3: `ACTIVE` and `READY` join the count), `check_record_anchors` `scripts/check-agent-record.py:1471` (the two-way gate); budget in `scripts/record-anchor-baseline.json` | `RecordAnchorRatchet` `tests/scripts/test_agent_record.py:1397` — 10 cases, RED-first, including `test_one_good_link_does_not_cover_a_rotted_bare_citation` `tests/scripts/test_agent_record.py:1465`, the `any()` shape the rot hid in. Five mutants red it: report-only, `EVIDENCED_STATES` restored, links-only, first-citation-only, range-only. Measured baseline **38** (32 STALE + 6 BROKEN); gate wired in `scripts/agent-preflight.sh` and the `agent-record` CI job (`--report`) | [record-anchor-ratchet.md](specs/record-anchor-ratchet.md) | `ACTIVE` | `CLAIM-ENG-RECORD-ANCHOR-RATCHET` | +| `ENG-RECORD-ANCHOR-RATCHET` | **The record's `path:line` citations were range-checked and never reported.** `check-agent-record.py` parsed BOTH forms: markdown links, and bare `` `file.cpp:123` `` through `RAW_LOCAL_ANCHOR_RE` since `ee511ca8a`. On a missing file or an out-of-range line `local_line_anchors` runs `continue`, so the bad anchor never reaches the caller, and `is_code_anchor` then answers with **any**, so one good sibling covers the rest. There was no symbol test and no report, and **32 of the 38** offenders are IN RANGE, so range-checking could not have found them. Measured at `8daa67b39`: **832 of 867** in-scope citations (**96.0%**) were already parsed and range-checked, and the **35** new to parsing sit under `.agents/`, `docs/` and `website/`; `EVIDENCED_STATES` omits `ACTIVE`/`READY` entirely and is deliberately NOT widened, because requiring an anchor there raises 85 errors across 53 rows. Even the fraction it saw was only range-checked, never checked to CONTAIN the symbol named beside it — every stale anchor found in the 2026-08-13/14 campaign was in range. LANDED as a device-leakage-shaped ratchet over a recorded baseline, never a bulk cleanup: the backlog is fixed by whoever next touches each row | T1 | none — this is our own record surface; the discipline mirrors AGENTS.md §Records ("cite the `file:line` you ported from") | parser + classifier + ratchet in `check-agent-record.py`: `BARE_CITATION_RE` `scripts/check-agent-record.py:1251` (the bare form), `cell_citations` `scripts/check-agent-record.py:1305` (both forms, with the adjacent-symbol rule), `classify_citation` `scripts/check-agent-record.py:1362` (OK / STALE / BROKEN), `RECORD_ANCHOR_STATES` `scripts/check-agent-record.py:1249` (gap 3: `ACTIVE` and `READY` join the count), `check_record_anchors` `scripts/check-agent-record.py:1485` (the two-way gate); budget in `scripts/record-anchor-baseline.json` | `RecordAnchorRatchet` `tests/scripts/test_agent_record.py:1397` — 10 cases, RED-first, including `test_one_good_link_does_not_cover_a_rotted_bare_citation` `tests/scripts/test_agent_record.py:1465`, the `any()` shape the rot hid in. Five mutants red it: report-only, `EVIDENCED_STATES` restored, links-only, first-citation-only, range-only. Measured baseline **38** (32 STALE + 6 BROKEN); gate wired in `scripts/agent-preflight.sh` and the `agent-record` CI job (`--report`) | [record-anchor-ratchet.md](specs/record-anchor-ratchet.md) | `ACTIVE` | `CLAIM-ENG-RECORD-ANCHOR-RATCHET` | | `ENG-RECORD-CONFLICT-SURFACES` | Retire the shared record surfaces that make concurrent PRs conflict by construction. MEASURED at `origin/main` `d928e2c3` with `git merge-tree --write-tree` over every open PR: **16 of 29 conflict (55%), and 13 of the 16 conflict in bookkeeping files ONLY**, with no product code involved — `.agents/coordination.md` in 8, `.agents/NOW.md` in 5, `.agents/roadmap_v1.md` in 4, `scripts/check-public-doc-tables.py` in 4, `docs/STATUS.md` in 4, and any `src/`/`tests/` path in just 3. Three defects, each of which GUARANTEES rather than risks a collision. (1) `.agents/NOW.md` is a fixed-size shared buffer at EXACTLY 6000/6000 chars (`check-now-current.py:31`), so adding a row requires evicting another and every PR is a read-modify-write of one global — and the conflict is the LUCKY outcome, since a clean three-way merge would apply both evictions and both additions, silently dropping live rows and blowing the very budget the checker defends. (2) `STATUS_RATCHET = {"chars": 243245}` (`check-public-doc-tables.py:557`) is a hardcoded byte count of a DIFFERENT file that may only fall, so a PR owing `docs/STATUS.md` one lifecycle line must delete unrelated prose from another row to pay for it and edit the checker too; the checker's own comment at `:331` already records the failure (*"a ratchet pinned to the byte turns every concurrently merged row's one-line status edit into a spurious failure"*) and answered it with slack instead of removing the coupling. (3) `.agents/coordination.md`'s active-claims table is insert-at-one-anchor: the six ROCm GDN PRs (#334 #336 #341 #343 #345 #348) are ONE author's sequential stack that conflicts on nothing else, each appending a ~1,500-char row — the PR description, transcribed into a file every other claim also writes. It also contradicts the protocol it serves: `AGENTS.md` holds that *"History is git"* and *"There is no state log"*, yet both claims tables ARE state logs duplicating `gh pr list`, `row/` branch names and issue state; the argument that refuses a waiver registry applies unchanged to a claims registry. Precedent twice over — `policy.csv` retired in `0f3e44ee`, per-class line budgets retired 2026-08-10 because the gate fired on ordinary work. The exonerated surfaces share ONE property, one writer per file: `.agents/specs/.md` (one file per row, **zero conflicts** in the sample), the `*-matrix.md` inventories, and the append-only `.agents/benchmark-record.md`. SCOPE: remove `STATUS_RATCHET` and the doc-gating global counters while KEEPING the per-cell/per-paragraph caps (local, so they couple nothing); remove the active-claims table and derive claims from open PRs and branch names; drop `NOW.md`'s byte budget; order the roadmap's keyed tables by ID so distinct keys stop colliding at one anchor; and record the invariant — **no surface that every PR must write** — in `AGENTS.md`. No product source, kernel or gate semantic moves | T0 | NO vLLM analogue — this is local protocol machinery, so the mirror rule does not apply and no upstream `file:line` exists to port from. Governed instead by `AGENTS.md` §"Changing the rules or a checker", which requires a spec, a red-before test or mutation, and green-after evidence | - | - (spec-before-code: the red-before suites are named in the spec's Tests section — `tests/scripts/test_check_public_doc_tables.py`, `tests/scripts/test_check_now_current.py`, a mutation case per removed rule proving the obligation survives in the retained caps and `check-doc-checkpoint.py`, and a `git merge-tree` merge-shape regression that must be RED before the `NOW.md`/roadmap work and GREEN after) | [retire-shared-record-surfaces.md](specs/retire-shared-record-surfaces.md); issue [#364](https://github.com/mudler/vllm.cpp/issues/364) | `READY` | `CLAIM-ENG-RECORD-CONFLICT-SURFACES` | | `ENG-TRAILER-MERGE-ARTIFACTS` | The trailer gate rejects CORRECT commits because of paragraph placement, and that is why `main` is red on `agent-record`. `check-commit-trailers.py` reads trailers via `git interpret-trailers --parse`, which treats ONLY the final paragraph as the block; GitHub appends `Co-authored-by:` as a SEPARATE trailing paragraph on a squash merge, so a complete correct block becomes invisible and the gate reports it missing. MEASURED: piping `git show -s --format=%B dbd0d51c` into `git interpret-trailers --parse` prints nothing but the co-author line, and 13 of the last 30 commits on `main` fail the check -- unnoticed only because those runs were cancelled (#274), which HID the defect rather than causing it. FIX: fuse consecutive trailing TRAILER-SHAPED paragraphs before parsing. Nothing is relaxed -- the block must still exist, the marker must still sit above it, each declaration must still appear exactly once, and an AI co-author is still forbidden; the block is merely FOUND where the merge tool left it. A prose paragraph still terminates it. REJECTED IN FLIGHT and recorded because it is the more instructive half: a first attempt also collapsed identical duplicate trailers to fix the multi-commit-squash shape, which relaxes the uniqueness rule an existing test already pins. Rewriting that assertion to suit the change is what AGENTS.md forbids, and the distinction is real -- a doubled block is genuinely malformed and fixable at source, whereas the co-author case is a correct commit defeated by the parser. Reverted in full. SCOPE LIMIT, stated rather than implied: this fixes ONE of five observed shapes. `f64f2b71` (bot co-author) is a REAL violation the parse had been hiding and now correctly fails; `87308dea` (GitHub's `---------` separator), `b8293c88` (squash doubled the block) and `b580452d` (merge button, no trailers) stay red by design. Closing those is a merge-method change, not a checker change | T0 | NO vLLM analogue -- local protocol machinery, so the mirror rule does not apply and there is no upstream `file:line` to port from. Governed by `AGENTS.md` §"Changing the rules or a checker" | `scripts/check-commit-trailers.py:60` (`join_trailing_trailer_paragraphs`, `_is_trailer_paragraph`, and the fused `parsed_trailers`) | `tests/scripts/test_check_commit_trailers.py:1` 21 cases -- the RED-BEFORE appended-co-author case plus four GUARDS that keep the fusion bounded (doubled block still fails, contradictory declarations still fail, a no-trailer merge message still fails, prose after the block still fails), all four green before and after; closure [parity-ledger.md#L941](parity-ledger.md#L941) | [trailer-merge-artifacts.md](specs/trailer-merge-artifacts.md); issue [#406](https://github.com/mudler/vllm.cpp/issues/406) | `DONE` | `157080c8` | | `ENG-FORGE-COAUTHOR` | The forbidden-AI-trailer rule was catching ATTRIBUTION rather than an authorship claim, which is why bot-opened PRs red `main` on merge. GitHub composes the squash message itself and appends the account that opened the PR — `Co-authored-by: localai-org-maint-bot <...@users.noreply.github.com>` — and most PRs here are opened by a bot, so nearly every squash trips the AI-identity check. Real instance `f64f2b71`, invisible until #406 repaired the parse, which is why it reads as a new failure and is not one. The rule exists so an AI cannot claim it WROTE the code, and that stays; GitHub is recording who pressed the button, and the AI-involvement claim is already carried separately by `AI-Assisted` and `Assisted-by` in the same block. FIX: accept a `Co-authored-by` at a GitHub account noreply address even when the name matches an AI identity token, keyed on the FORGE'S OWN DOMAIN rather than the name so the exemption cannot be borrowed. A hand-written `Co-authored-by: Claude ` still fails; `Signed-off-by` is excluded from the exemption entirely, because a sign-off is a legal assertion about provenance rather than attribution. `AGENTS.md` records the same distinction in the same change so prose and checker cannot drift | T0 | NO vLLM analogue -- local protocol machinery, so the mirror rule does not apply and there is no upstream `file:line` to port from. Governed by `AGENTS.md` §"Changing the rules or a checker" | `scripts/check-commit-trailers.py:38` (`FORGE_ACCOUNT_EMAIL` and the forbidden-trailer skip) | `tests/scripts/test_check_commit_trailers.py:1` 25 cases -- the RED-BEFORE forge-bot case plus THREE guards that matter more than the relaxation because this LOOSENS a rule: a hand-written AI co-author still fails, `Signed-off-by` at the same noreply address still fails, and a human co-author still passes; all three green before and after. Real commit `f64f2b71` re-verified per commit | [forge-coauthor-attribution.md](specs/forge-coauthor-attribution.md); issue [#418](https://github.com/mudler/vllm.cpp/issues/418) | `ACTIVE` | `CLAIM-ENG-FORGE-COAUTHOR` | @@ -277,6 +277,7 @@ claims it. | `LOAD-GGUF-MMPROJ` | A SECOND, `clip`-architecture GGUF projector file beside the language file, and the Qwen3-VL vision tower loaded out of it. The single-file assumption is structural, not incidental: `ModelSource` carries a VECTOR of safetensors shards and exactly ONE `GgufFile*`, and `EngineParams` has no projector field at all, so an mmproj has nowhere to arrive. `GgufFile::Open` does handle SHARDS (`DetectSplit` parses `-NNNNN-of-MMMMM.gguf`), but shards of one split are not a second, differently-architected file. Nothing in this tree loads a `clip` projector today and the two apparent counterexamples are neither: MiniMax-H3 reads `visual.*` out of the SAME encoder GGUF and only from a test, and MuseGlimmer's mmproj path is a REFUSAL whose only caller is a test — there is no production call site because no production path accepts a second GGUF path. First consumer is `Qwen3.8-27B` ([#821](https://github.com/mudler/vllm.cpp/issues/821)), whose `mmproj-BF16.gguf` (334 tensors, `clip.projector_type = qwen3vl_merger`, header-verified 2026-08-18: data end == file size 931,146,432) ships BOTH halves of the temporal patch embedding as `v.patch_embd.weight` + `v.patch_embd.weight.1` — precisely what MuseGlimmer's lacks, so the check that distinguishes them is this row's, not a reason to refuse. Second consumer is MuseGlimmer, whose refusal becomes REACHABLE production code the moment this lands | T0 | Pinned vLLM has no GGUF loader at all (`555967922`, `model_loader/__init__.py:33-49`), so the compatibility reference is llama.cpp `b10451` = `10bf611e5` (`PROJECTOR_TYPE_QWEN3VL`; the previously recorded `tools/mtmd/clip-impl.h:330` was read at the SUPERSEDED local fork `237ad9b96` and is owed re-anchoring, [#1003](https://github.com/mudler/vllm.cpp/issues/1003)) | the one open site [model_loader.cpp:1679](../src/vllm/entrypoints/model_loader.cpp#L1679), the single pointer [model_registry.h:98](../include/vllm/model_executor/models/model_registry.h#L98), the arch dispatch [model_loader.cpp:794](../src/vllm/entrypoints/model_loader.cpp#L794), the refusal precedent [muse_glimmer_gguf_weights.cpp:695](../src/vllm/model_executor/models/muse_glimmer_gguf_weights.cpp#L695) | none yet — this row is spec-only. The port target for the vision config mapping is [minimax_h3_vision_gguf.cpp:32](../src/vllm/model_executor/models/minimax_h3_vision_gguf.cpp#L32), which builds the same `Qwen3VLVisionConfig` from `visual.*` | [quantized arms of Qwen3.8-27B](specs/qwen38-27b-quant-arms.md) | `READY` | - | | `LOAD-HF-BPE` | HF tokenizer.json byte-level BPE and incremental detokenization | T0 | `vllm/tokenizers/registry.py:176`; `vllm/tokenizers/hf.py:163`; `tests/tokenizers_/test_hf.py:18`; `tests/tokenizers_/test_detokenize.py:148` | `src/vllm/tokenizer/tokenizer.cpp:240,484,512`; `src/vllm/v1/engine/detokenizer.cpp:409` | `tests/vllm/test_bpe.cpp:226`; `tests/vllm/test_tokenizer_parity.cpp:66,74,82,90`; `tests/vllm/test_pretokenizer.cpp:130` | `planned: specs/hf-tokenizer.md` | `ANCHOR-BACKFILL` | - | | `LOAD-SENTENCEPIECE` | SentencePiece (Metaspace + byte-fallback) BPE tokenizer family — the gate to Mistral/Gemma/SentencePiece `tokenizer.json`. `Tokenizer::FromHfJson` now dispatches on the pre_tokenizer FAMILY: a bare `Metaspace` node selects the SP path (space→▁ U+2581; `prepend_scheme` first/always/never with the starts-with-▁ guard; `split=false`; out-of-vocab char → `<0xNN>` byte-fallback tokens; merge-ranked BPE over the raw-UTF-8 string), else the BYTE-IDENTICAL byte-level path (`DetectPattern` still fails loudly on Metaspace, so families never overlap). Decode + the incremental detokenizer mirror HF's Sequence decoder (Replace ▁→space, ByteFallback, Fuse, Strip 1 leading space). **UNBLOCKS the Mistral (`MODEL-TEXT-mistral-mistral-for-causal-lm`) paged-engine SACRED gate.** `split=true` fails loudly (no golden in scope). Mirrors HF `tokenizers` 0.22 | T1 | `vllm/tokenizers/hf.py:163`; `vllm/tokenizers/mistral.py:235,467`; HF `tokenizers` 0.22 `pre_tokenizers/metaspace.rs`, `models/bpe/model.rs::merge_word`, `decoders/{replace,byte_fallback,fuse,strip}.rs` | dispatch `src/vllm/tokenizer/tokenizer.cpp:184` (`DetectMetaspace`), `:732` (`EncodePlainSp`), `:878` (`SpDecodeTokens`), `:927` (`Decode` SP branch); merge-loop factor `src/vllm/tokenizer/bpe.cpp:91` (`BpeMerge`); family/params/accessors `include/vllm/tokenizer/tokenizer.h:88,98`; incremental dispatch `src/vllm/v1/engine/detokenizer.cpp:357`; generator `tools/parity/dump_tokenizer_mistral.py` | `tests/vllm/test_tokenizer_parity_mistral.cpp:78` **6/6, 421 assertions byte-exact** vs HF `tokenizers` 0.22.2 (= vLLM 0.25.0 backend) over a 45-entry Metaspace/byte-fallback/special-token corpus (goldens `tests/parity/goldens/tokenizer_mistral/{tokenizer.json,encodings.json}`); SACRED cross-check vLLM `AutoTokenizer` (transformers 5.13.1) 0/45 mismatch (±BOS); byte-level suites byte-identical `tests/vllm/test_bpe.cpp` 852, `tests/vllm/test_detokenizer.cpp` 221, `tests/vllm/test_tokenizer_parity.cpp` 1175, `tests/vllm/test_tokenizer_parity_deepseek.cpp` 2461 | [specs/sentencepiece.md](specs/sentencepiece.md) | `ANCHOR-BACKFILL` | `CLAIM-LOAD-SENTENCEPIECE` | +| `SPEC-BPE-QUADRATIC-MERGE` | **The BPE merge loop is O(n^2) in pretoken length, on the request path, before the length check.** `src/vllm/tokenizer/bpe.cpp::BpeMerge` rescans every adjacent pair per merge and builds one `std::string` key per probe through `src/vllm/tokenizer/bpe.cpp::MergeKey`, on the premise its own comment states: "pretokens are tiny". Four of the seven pretokenizer rules return an unbounded run (`src/vllm/tokenizer/pretokenizer.cpp::MatchLetterRun`, `src/vllm/tokenizer/pretokenizer.cpp::MatchPunctRun`, `src/vllm/tokenizer/pretokenizer.cpp::MatchWsNotBeforeNonSpace`, `src/vllm/tokenizer/pretokenizer.cpp::MatchWs`), and on the SentencePiece family `src/vllm/tokenizer/tokenizer.cpp::EncodePlainSp` merges the WHOLE prompt as one word because Mistral and Gemma declare `split: false`. MEASURED at `31f93787c`: 65,535 bytes of one repeated character costs 28.1-48.1 s of one core; 64 KB of ordinary English prose through the committed Mistral golden costs 24.1 s, growing n^2.0. `src/vllm/v1/engine/input_processor.cpp::ValidatePromptLen` runs FIVE LINES AFTER the encode, so `max_model_len` bounds none of it, and `/tokenize` reaches the same path with no engine. This is a remote denial of service, not a latency curiosity. Fix mirrors HF `tokenizers` 0.22.2 `Word::merge_all` (heap over candidate merges, identifier-keyed merge table); a prototype is bit-identical on ten inputs and 3,459x faster on the 65,535-byte case. A pretoken cap is REFUSED: it changes the token identifiers. Removing the string key alone was MEASURED and is slower, so it is not a separable win | T0 | HF `tokenizers` 0.22.2 `tokenizers/src/models/bpe/word.rs:162-250` (`Word::merge_all`), `:28-35` (`Ord for Merge`), `tokenizers/src/models/bpe/mod.rs:9` + `model.rs:19,174-192` (identifier-keyed merge table and its load-time refusal) | `src/vllm/tokenizer/bpe.cpp::BpeMerge` (the defect); callers `src/vllm/tokenizer/tokenizer.cpp::EncodePlain` and `src/vllm/tokenizer/tokenizer.cpp::EncodePlainSp` | none yet — the corpus at `tests/parity/goldens/tokenizer_qwen36/` has a longest pretoken of 54 bytes, which is why every existing gate is green | [specs/bpe-quadratic-merge.md](specs/bpe-quadratic-merge.md) | `READY` | - | | `LOAD-CONFIG-SURFACE` | Dataclass-for-dataclass config and serve-compatible flags | T0/T1 | `vllm/config/scheduler.py:26`; `vllm/config/cache.py:25`; `vllm/config/compilation.py:378` | `include/vllm/config/scheduler.h:67`; `src/vllm/config/scheduler.cpp:11`; `src/vllm/transformers_utils/hf_config.cpp:83`; limited flags incl. `max_num_seqs`/`max_num_batched_tokens` `examples/server/main.cpp:63,116,170` | `tests/vllm/test_scheduler_config.cpp:10`; `tests/vllm/test_hf_config.cpp:131,224,265`; `examples/CMakeLists.txt:34` | `planned: specs/config-surface.md` | `PARTIAL` | - | | `ENG-HF-MODEL-DOWNLOAD` | Fetch a checkpoint from HuggingFace so `--model` accepts a repository identifier and not a local path only. Two forms behind one flag, with the local path probed first: `org/repo` mirrors vLLM's full-snapshot download, and `org/repo:Q4_K_M` fetches one GGUF file, a form vLLM does not implement and which llama.cpp supplies as the secondary oracle. Mirrors vLLM's two-phase fetch (config JSON first, so a bad repository fails after 200 KB and not 60 GB) and its index-driven file selection, which reads `model.safetensors.index.json` and fetches the exact names in `weight_map` instead of every match for `*.safetensors`. Reference is resolved to a commit before any byte is fetched, so a moving `main` cannot change what a second run loads. Cache layout is HuggingFace's documented local cache, so a host holding a Python `huggingface_hub` cache gets a hit and `model_loader.cpp:279-303` reads it unchanged. Transport is the already-vendored cpp-httplib with OpenSSL, matching llama.cpp, which retired libcurl at `CMakeLists.txt:195` at stock tag `b10451`. NOT a verbatim port on one point: llama.cpp's `is_valid_oid` accepts any 40- or 64-character hexadecimal string, and on 17 August 2026 this project measured the tree API answering an unauthenticated caller on gated `Lightricks/LTX-2.5` with an `lfs.oid` of one character repeated 64 times, identical for all 14 large-file-storage files, so this row treats an untokenized `lfs.oid` as absent and proves completeness structurally instead | T1 | `vllm/model_executor/model_loader/weight_utils.py:345,349-357,459,472-490,493-496,506`; `vllm/model_executor/model_loader/default_loader.py:167-184`; `vllm/engine/arg_utils.py:839`; `vllm/config/model.py:183` at pin `5559679229`. Secondary oracle llama.cpp for the `:QUANT` form only, stock tag `b10451`, anchors re-verified by W1 at commit `10bf611e533d81f739128304991c5e133c6aebd8` | W1 and W2 only: `src/vllm/transformers_utils/hf_hub.cpp:1`, `src/vllm/transformers_utils/hf_cache.cpp:1`, reached from `src/vllm/entrypoints/model_loader.cpp:280`. No downloader, no `--model` grammar, no TLS option | `tests/vllm/transformers_utils/test_hf_cache.cpp:1`, `tests/vllm/transformers_utils/test_hf_hub.cpp:1` (in-process fake hub), `tests/vllm/entrypoints/test_dflash_draft_hf_cache.cpp:1` (the loader reach, red when the call site is deleted) | [hf-model-download.md](specs/hf-model-download.md) | `READY` | - | | `LOAD-LONGTAIL` | Sharded-state, tensorizer, RunAI, BitsAndBytes loaders | T3 | `vllm/model_executor/model_loader/__init__.py:33-65` | - | - | `planned: specs/loader-longtail.md` | `INVENTORIED` | - | diff --git a/.agents/specs/bpe-quadratic-merge.md b/.agents/specs/bpe-quadratic-merge.md new file mode 100644 index 000000000..9457e9059 --- /dev/null +++ b/.agents/specs/bpe-quadratic-merge.md @@ -0,0 +1,457 @@ +# SPEC-BPE-QUADRATIC-MERGE — the merge loop is quadratic, and the premise that made it look safe + +**Issue:** [#1365](https://github.com/mudler/vllm.cpp/issues/1365), re-titled and +re-scoped on 2026-08-19 from "a reproducible ~4 s TTFT outlier on request 3 of +every leg" to the cause measured below. +**Kind:** one algorithm replaced inside the tokenizer, mirrored from HF +`tokenizers`. No kernel, no device code, no model numerics, no new public +surface. Every gate in this spec runs on a CPU host. +**Row:** `SPEC-BPE-QUADRATIC-MERGE` in +[`engine-matrix.md`](../engine-matrix.md), section "Loading, tokenizer, and config". +**Base:** `31f93787c`. Every local line number in this document is read there. + +## Now + +`origin/main` at `31f93787c` encodes every prompt through +`src/vllm/tokenizer/bpe.cpp::BpeMerge`. That function is O(n^2) in the number of +symbols of one pretoken, and the comment above it states the premise the design +rests on: + +``` +$ sed -n '91,93p' src/vllm/tokenizer/bpe.cpp +void BpeMerge(std::vector& symbols, const MergeRanks& ranks) { + // Repeatedly merge the lowest-ranked adjacent pair; leftmost wins ties + // (strict < keeps the first best). O(n^2) scan; pretokens are tiny. +``` + +Nothing bounds a pretoken. Four of the seven pretokenizer rules return a run of +unbounded length, and on the SentencePiece family the pretoken is the entire +prompt. Measured on this tree, 64 KB of the letter `a` costs 28.1 s of one core +and 64 KB of ASCII spaces costs 48.1 s, both on a path that runs before any +length check. The row is `READY`: this spec is committed before any product +code, because the fix replaces the merge algorithm on a token-exact path and the +design has to be agreed before anyone edits the tokenizer. + +## Scope + +In scope: + +- `src/vllm/tokenizer/bpe.cpp::BpeMerge`, replaced by a heap-driven merge that + mirrors HF `tokenizers` `Word::merge_all`. +- Whatever `src/vllm/tokenizer/bpe.h` has to expose so that + `src/vllm/tokenizer/tokenizer.cpp::EncodePlain` and + `src/vllm/tokenizer/tokenizer.cpp::EncodePlainSp`, the only two callers, reach + the new form. +- The interning decision described under `## Design`, and the load-time refusal + it needs. +- Red-first equivalence and cost cases in `tests/vllm/test_bpe.cpp` and one new + CPU-tier binary described under `## Tests`. +- `docs/STATUS.md` and `docs/BENCHMARKS.md` when the row moves to `DONE`. + +Out of scope, each with a stated reason: + +- **A cap on pretoken length.** Argued and rejected under `## Design`. +- **The pretokenizer.** `src/vllm/tokenizer/pretokenizer.cpp` mirrors an + upstream regex. Its unbounded runs are the regex's behaviour, not a defect, + and shortening one changes the token identifiers we emit. +- **A word cache.** Upstream has one (`tokenizers/src/models/bpe/model.rs:475-496`) + and it stores only sequences below `MAX_LENGTH = 256` + (`tokenizers/src/utils/cache.rs:10`), so it does not touch the case this row + exists for. Adding it is a separate, measurable question. +- **A request-size limit at the HTTP boundary.** Discussed under + `## Defence in depth`, and deliberately not bundled here. +- **`ignore_merges`, the added-token split, the detokenizer.** Untouched. + +## Our baseline + +Every figure below was reproduced in this row, on an AMD Ryzen 9 9950X3D at +`-O2`, against a driver built from this tree's own `bpe.cpp`, +`pretokenizer.cpp` and `unicode_data.cpp`. The box carried a load average +between 130 and 190, so these are contended numbers. The **shape** is the +result. A same-binary idle re-measure belongs to the implementing row, and +`## Gates` requires it before any speed claim is accepted. + +Merge table: `merges.txt` of `qwen3.8-27b-hf`, 247,586 merges, sha256 +`a9d356d7bdf1ef4949e3e748e95b8e10ad9d4e2e838eddc38a0a7b6b94d1db8d`. + +### The benchmark prompts that found it + +The six reconstructed `vllm bench serve` prompts of the #1365 legs, timing +`BpeSplit` only: + +| prompt | bytes | pretokens | longest pretoken | current | heap prototype | +|---|---:|---:|---:|---:|---:| +| 0 | 8,798 | 698 | 88 | 3.70 ms | 3.27 ms | +| 1 | 6,629 | 344 | 119 | 2.94 ms | 2.93 ms | +| **2** | **8,034** | **1** | **8,034** | **1,356.89 ms** | **6.75 ms** | +| 3 | 6,682 | 723 | 47 | 1.65 ms | 1.93 ms | +| 4 | 6,939 | 737 | 91 | 1.69 ms | 2.01 ms | +| 5 | 5,863 | 709 | 73 | 3.72 ms | 2.59 ms | + +Index 2 is the outlier request of #1365's four legs. It is one pretoken. + +### The growth is quadratic + +Prompt 2 truncated and tripled, one pretoken throughout: + +| bytes | BPE merge | +|---:|---:| +| 2,007 | 81.57 ms | +| 4,017 | 350.61 ms | +| 6,024 | 865.44 ms | +| 8,034 | 1,933.60 ms | +| 24,102 | 14,794.38 ms | + +Three times the bytes costs 7.65 times the time. + +### Four rules are unbounded, not one + +`src/vllm/tokenizer/pretokenizer.cpp::MatchLetterRun` (rule 2), +`src/vllm/tokenizer/pretokenizer.cpp::MatchPunctRun` (rule 4), +`src/vllm/tokenizer/pretokenizer.cpp::MatchWsNotBeforeNonSpace` (rule 6) and +`src/vllm/tokenizer/pretokenizer.cpp::MatchWs` (rule 7) each scan `while (p < +t.size())` with no cap. Only +`src/vllm/tokenizer/pretokenizer.cpp::MatchNumbers` is capped, at one codepoint +for Qwen and three for Llama-3. Each input below is 8,192 bytes returned as one +pretoken: + +| input | rule | current | heap prototype | +|---|---|---:|---:| +| `a` x 8,192 | 2, letter run | 514.19 ms | 2.33 ms | +| `中` x 2,731 | 2, letter run | 438.58 ms | 0.85 ms | +| `ก` x 2,731 | 2, letter run | 505.81 ms | not run | +| `~` x 8,192 | 4, punct run | 451.81 ms | 2.20 ms | +| space x 8,192 | 7, whitespace | 813.02 ms | 3.16 ms | + +The trigger is a long run of one character class, not one script. ASCII reaches +it. + +### At 65,535 bytes + +| input | current | heap prototype | +|---|---:|---:| +| `的` x 21,845 | 28,746.70 ms | 8.31 ms | +| `a` x 65,535 | 28,130.35 ms | not run | +| space x 65,535 | 48,098.11 ms | not run | + +### The SentencePiece family pays it on ordinary English + +`src/vllm/tokenizer/tokenizer.cpp::EncodePlainSp` calls the same `BpeMerge`, and +its pretokenization is Metaspace, not the split regex. When `metaspace_split_` +is false it calls `encode_piece` on the whole input +(`src/vllm/tokenizer/tokenizer.cpp:1005-1007`). Two shipped families are in that +state: `src/vllm/tokenizer/tokenizer.cpp:566` hardcodes `false` for the Gemma +metaspace layout, and the committed Mistral golden +`tests/parity/goldens/tokenizer_mistral/tokenizer.json` declares +`"split": false` in its own `pre_tokenizer`. + +Full `Tokenizer::Encode` through that committed golden, on "The quick brown fox +jumps over the lazy dog. " repeated to length. The Qwen3.6 golden is the +byte-level control on the identical text: + +| prompt bytes | Mistral golden | Qwen3.6 golden | +|---:|---:|---:| +| 1,000 | 5.59 ms | 0.15 ms | +| 2,000 | 21.73 ms | 0.11 ms | +| 4,000 | 87.15 ms | 0.23 ms | +| 8,000 | 357.01 ms | 0.47 ms | +| 16,000 | 1,410.42 ms | 0.92 ms | +| 32,000 | 5,804.42 ms | not run | +| 65,536 | 24,144.72 ms | not run | + +Sixteen times the bytes costs 252 times the time. That exponent is 2.0. A +Mistral or Gemma server spends 24 s of one core to tokenize a 64 KB English +document, and a user who pastes a document sends exactly that. + +### Why no gate saw it + +The committed 64-entry corpus at `tests/parity/goldens/tokenizer_qwen36/` has a +longest pretoken of **54 bytes**. The failing regime starts two orders of +magnitude above it. A token gate also cannot see this class of defect at all, +because the identifiers it compares are correct. + +## Reachability + +The encode is on the request path of a shipped server, and it runs before the +length check that a reader would expect to bound it: + +``` +$ sed -n '259,265p' src/vllm/v1/engine/input_processor.cpp + std::vector prompt_token_ids = + tokenizer_.EncodeWithSpecialTokens(prompt); + ... + ValidatePromptLen(prompt_token_ids.size()); +``` + +`src/vllm/v1/engine/input_processor.cpp::ValidatePromptLen` runs at `:265`, five +lines after the encode at `:260`. `max_model_len` therefore bounds nothing here: +the cost is paid in full, and only then does the server decide the prompt is too +long to serve. `src/vllm/entrypoints/openai/api_server.cpp:763-764` reaches the +same encode from `/tokenize`, which needs no engine. The worker pool is sized by +`src/vllm/entrypoints/openai/api_server.cpp::HttpWorkerCount` as +`max_concurrent_streams + kControlWorkerHeadroom`, so one caller can occupy +every worker at once. + +This is a remote denial of service in a shipped path. It is not a latency +curiosity, and the row is scoped that way. + +## Upstream chain + +vLLM does not implement a tokenizer. `vllm/tokenizers/hf.py` delegates to +HuggingFace `transformers`, which delegates to the Rust `tokenizers` crate, and +that crate holds the merge loop. So the executing chain for a vLLM encode is: + +| Layer | What it decides | Anchor | +|---|---|---| +| vLLM | which tokenizer to build | `vllm/tokenizers/registry.py:176`, `vllm/tokenizers/hf.py:163` | +| `transformers` 5.14.1 | the `tokenizer.json` contract | the pin resolved inside the pinned vLLM environment, [`../oracles/transformers.md`](../oracles/transformers.md) | +| `tokenizers` 0.22.2 | **the merge algorithm** | `tokenizers/src/models/bpe/word.rs:162-250` (`Word::merge_all`) | +| `tokenizers` 0.22.2 | the merge-table representation | `tokenizers/src/models/bpe/mod.rs:9` (`type Pair = (u32, u32)`), `model.rs:19` (`MergeMap`), `model.rs:174-192` (built at load, refuses an out-of-vocabulary merge) | +| `tokenizers` 0.22.2 | the tie-break order | `tokenizers/src/models/bpe/word.rs:28-35` (`Ord for Merge`) | +| `tokenizers` 0.22.2 | the symbol list | `tokenizers/src/models/bpe/word.rs:37-53` (`Symbol`, `merge_with`) | +| `tokenizers` 0.22.2 | word construction, byte fallback, unk fusing | `tokenizers/src/models/bpe/model.rs:382-460` (`merge_word`) | +| `tokenizers` 0.22.2 | the word cache, out of scope here | `model.rs:475-496`, `tokenizers/src/utils/cache.rs:7,10` | + +The crate source read for this spec is `tokenizers-0.22.2.crate`, sha256 +`b238e22d44a15349529690fb07bd645cf58149a1b1e44d6cb5bd1641ff1a6223`, fetched from +`static.crates.io`. `tokenizers` is not a registered oracle in its own right; it +is the code `transformers` executes, and +[`../oracles/transformers.md`](../oracles/transformers.md) is the pin that +selects this version. The implementing row records the version it read. + +## Port map + +| Upstream | Ours today | After | +|---|---|---| +| `word.rs:162-250` `Word::merge_all` | `src/vllm/tokenizer/bpe.cpp::BpeMerge`, an O(n^2) rescan | the same signature, heap-driven | +| `word.rs:37-43` `Symbol {c, prev, next, len}` | `std::vector` with `erase` per merge | a linked symbol list with `len = 0` tombstones | +| `word.rs:28-35` `Ord for Merge` | `strict <` inside the scan | an explicit comparator, with the tie pinned by a test | +| `mod.rs:9`, `model.rs:19` identifier-keyed `MergeMap` | `MergeRanks = unordered_map` in `src/vllm/tokenizer/bpe.h` | an identifier-pair-keyed table built at load | +| `src/vllm/tokenizer/bpe.cpp::MergeKey` | one `std::string` per probe | deleted; there is no key to build | +| `model.rs:180-189` `MergeTokenOutOfVocabulary` | no rule; the failure appears per request in `src/vllm/tokenizer/tokenizer.cpp::EncodePlain` | refused at load, naming the missing token | +| `model.rs:382-460` `merge_word` | `src/vllm/tokenizer/tokenizer.cpp::EncodePlainSp`'s symbol builder | unchanged in behaviour, emitting identifiers | +| `model.rs:475-496` word cache | absent | still absent, and out of scope | + +## Design + +**Mirror `Word::merge_all`.** HF `tokenizers` 0.22.2, the backend the pinned +`transformers` 5.14.1 resolves, solves this at +`tokenizers/src/models/bpe/word.rs:162-250`. The crate source is +`tokenizers-0.22.2.crate`, sha256 +`b238e22d44a15349529690fb07bd645cf58149a1b1e44d6cb5bd1641ff1a6223`. Its shape: + +1. A doubly linked list of symbols, each carrying `c`, `prev`, `next`, `len` + (`word.rs:37-43`). A removed symbol is tagged `len = 0` rather than erased, + so no index shifts (`word.rs:210`) and the final `retain` drops them all at + once (`word.rs:249`). +2. A `QuaternaryHeap` of candidate merges, seeded once from every adjacent pair + (`word.rs:163-178`). +3. Pop the best candidate, apply it, and push only the two pairs the merge + creates: the one with the previous symbol and the one with the next + (`word.rs:217-244`). +4. A popped candidate is validated before use rather than removed when it goes + stale: skip it when its left symbol was removed (`word.rs:187`), when it has + no right neighbour (`word.rs:191`), or when the pair it names is no longer + the pair at that position (`word.rs:198-205`). + +That is O(n log n) merges of O(1) work each, against our O(n) rescan per merge. + +**The order must stay identical, and upstream makes it explicit.** `Ord for +Merge` (`word.rs:28-35`) inverts both comparisons, so the max-heap yields the +lowest `rank` first and the lowest `pos` on a tie. That is the same leftmost +rule our loop gets from `strict <`. Our implementation states this in a comment +and a test pins it, because a heap silently reversing a tie is exactly the +change a token gate over ordinary text would not catch. + +**Intern the symbols, which removes the string key rather than optimising it.** +Upstream's merge table is `AHashMap<(u32, u32), (u32 rank, u32 new_id)>` +(`tokenizers/src/models/bpe/mod.rs:9`, `model.rs:19`), built once at load +(`model.rs:174-192`). A symbol is a vocabulary identifier, a pair is two of +them, and `MergeKey` has no counterpart upstream. Our `MergeRanks` is +`unordered_map`, so we build one key string per probe. Mirroring +upstream deletes that whole class of work. + +Interning has one precondition, and upstream states it as a refusal: a merge +whose left token, right token, or concatenation is absent from the vocabulary is +`MergeTokenOutOfVocabulary` at load (`model.rs:180-189`). Our string form has no +such rule, so a merge table can currently produce a symbol that +`src/vllm/tokenizer/tokenizer.cpp::EncodePlain` then fails on at encode time +with `symbol "..." not in vocab`. Mirroring the upstream refusal moves that +failure from per-request to load, which is where upstream puts it and where a +user can act on it. **This is a behaviour change on malformed inputs and it must +be called out in review**: a checkpoint that loads today and fails on some +prompts would be refused at load instead. That is the upstream behaviour, so it +is the behaviour we mirror, but the implementing row records which committed +goldens it verified still load. + +**The two callers differ and both must be carried.** +`src/vllm/tokenizer/tokenizer.cpp::EncodePlain` starts from single mapped +codepoints, each of which is in the vocabulary for a complete byte-level +alphabet. `src/vllm/tokenizer/tokenizer.cpp::EncodePlainSp` builds symbols +first, with byte-fallback decomposition and a `kUnk` sentinel that is +deliberately not a real symbol, and only then merges. Upstream does the same +split (`model.rs:382-460` builds the `Word`; `merge_all` runs after), so the +seam survives. The sentinel needs a reserved identifier that no merge can name. + +**Rejected: a cap on pretoken length.** It changes the token identifiers we +emit, so it breaks the mirror on exactly the inputs it claims to protect. +Truncation is worse, because it changes the output silently. Nobody should +reach for this as a shortcut, which is why it is written here. + +**Rejected on measurement: removing the per-probe `std::string` on its own.** +This was the low-risk option and it does not work. The same loop with one +reused key buffer, measured in the same harness on the same inputs, scored +31,076.04 ms on the 65,535-byte case against the current 28,746.70 ms, and +639.39 ms against 813.02 ms on 8,192 spaces. It is within noise, and slower on +the largest case. The allocation is not the cost. The rescan is. The separable +question therefore has a measured answer of no, and the two changes are one +change: interning removes the key because the algorithm no longer names one. + +## Prototype evidence + +A prototype of the design, built against this tree's sources, produced symbol +output **bit-identical** to `BpeMerge` on all ten inputs it was run against: the +six benchmark prompts, 8,192 spaces, 8,192 `~`, 8,192 `a`, 2,731 `中`, and the +65,535-byte `的` case. It is a prototype, not the implementation. It validates +staleness by comparing the merged string, where the implementation must mirror +upstream's identifier comparison (`word.rs:198-205`), and it uses +`std::priority_queue` where upstream uses a 4-ary heap. Neither difference +changes the order, and both are named so the implementer does not inherit them +by accident. + +## Defence in depth + +A length guard at the API boundary is worth having **after** the algorithmic +fix, not instead of it, and it is not in this row's scope. Two constraints bind +it if a later row adds one: + +- It must **refuse**, with an explicit error naming the limit. Truncation would + silently change the token identifiers we return, which is the same defect as + the rejected cap. +- It belongs where a caller can be told, at the request boundary, not inside the + tokenizer. `src/vllm/v1/engine/input_processor.cpp::ValidatePromptLen` is the + wrong place, because it needs the token count that the expensive step + produces. + +After the fix, 64 KB encodes in single-digit milliseconds, so the guard protects +against a future regression rather than against today's cost. File it as its own +issue when the row lands. + +## Dependencies + +None. The row is self-contained host code on base `31f93787c`. It needs no +oracle run, no lease, no checkpoint mount, and no other row. The two goldens its +gates read are already committed. It does not block on +[#915](https://github.com/mudler/vllm.cpp/issues/915) and #915 does not block on +it, although closing this row removes the outlier that +[`qwen38-27b-bf16-gate.md`](qwen38-27b-bf16-gate.md) records. + +## Work breakdown + +Non-overlapping. W1 is the whole correctness argument and can be reviewed +before any algorithm changes. + +| W | Deliverable | Reviewable on its own because | +|---|---|---| +| **W1** | The equivalence corpus and harness of `## Tests to port` item 1, plus the cost and growth assertions of items 2 and 3, all against UNCHANGED code | items 1 and 6 pass, items 2 and 3 are red. That red is the defect, recorded before anything moves | +| **W2** | The identifier-keyed merge table and the load-time refusal, with `BpeMerge` still doing the O(n^2) scan over identifiers | W1 stays exactly as green and as red as it was. Only the representation moved | +| **W3** | The heap merge, the linked symbol list, the comparator, the staleness checks | W1's items 2 and 3 turn green and nothing else changes | +| **W4** | The idle-host re-measure, the `## Outcome` section, `docs/STATUS.md` and `docs/BENCHMARKS.md` | the code is frozen; this is the record | + +W2 before W3 is deliberate. Interning and the heap are separable in the tree +even though they are one design, and splitting them means a reviewer never has +to hold a representation change and an algorithm change in mind at once. + +## Risks + +| Risk | Why it is real | Control | +|---|---|---| +| A tie is resolved differently and one token identifier changes | The heap's comparator is where leftmost-wins lives, and a reversed tie is invisible on most text | Port `Ord for Merge` (`word.rs:28-35`) with its inversion; a direct tie case in `test_bpe.cpp`, which already pins `BpeSplit("ĠĠĠ", r2) == {"ĠĠ", "Ġ"}` | +| A stale heap entry is applied | Positions and neighbours change under the heap | Mirror all three validations (`word.rs:187`, `:191`, `:198-205`); a red-first case that reaches a stale entry | +| Interning refuses a checkpoint that loads today | The load-time vocabulary rule is new for us | Verify every committed golden still loads; record which ones in `## Outcome` | +| The `kUnk` sentinel collides with a real identifier | The SentencePiece path merges a value that is not a vocabulary entry | Reserve an identifier outside the vocabulary range and assert it | +| The equivalence corpus is too small to be evidence | The existing 64-entry corpus missed this defect at 54 bytes | The corpus below is required to contain the long-pretoken regime | + +## Tests to port + +Red first, in this order. Each states what it fails on today and what it must +prove after. + +1. **Equivalence over a corpus, token identifiers only.** A CPU-tier binary that + encodes a corpus through the committed goldens and compares against recorded + identifiers. The corpus is the existing 64 entries, plus the six #1365 + prompts, plus real multilingual prose in Chinese, Japanese, Thai, Lao and + Khmer, plus the single-class runs above at a size the suite can afford. This + test **passes on today's code** and must keep passing. It is the correctness + gate, and its value is that it cannot go green on a faster wrong answer. +2. **A cost bound that fails today.** Encode one pretoken of a stated size and + assert an upper bound on wall time. This is red on `31f93787c` by three + orders of magnitude, which is the margin that makes a wall-clock assertion + honest on a contended runner. State the bound as an absolute figure with the + margin argued, not as a ratio against a same-run baseline. +3. **A growth-shape assertion.** Encode at n and at 2n and assert the ratio sits + below a bound that quadratic growth cannot satisfy and linearithmic growth + can. This is the assertion that survives a slow runner, because both halves + move together. It is red today at a ratio near 4. +4. **The leftmost tie.** Extend the existing `test_bpe.cpp` case into one that + distinguishes the two orders on a longer symbol list. +5. **A stale-entry case.** A merge sequence where a queued candidate is + invalidated by an earlier merge, constructed so that applying it produces + different identifiers. +6. **The SentencePiece arm.** The same equivalence and cost cases through + `tests/parity/goldens/tokenizer_mistral/tokenizer.json`, whose `"split": + false` makes the whole prompt one word. Ordinary English prose is the input, + because that is what fails. +7. **The load-time vocabulary refusal.** A merge table naming a token absent + from the vocabulary is refused at load, with the missing name in the message. + +The reachability mutation for the fresh reviewer: delete the call to the new +merge from `src/vllm/tokenizer/tokenizer.cpp::EncodePlain` in a scratch copy and +rerun the focused gate. A gate that stays green measured a class, not the +encode path. + +## Gates + +**This row is unusually cheap to gate, and the plan takes that seriously.** The +tokenizer is pure host code. Nothing here needs a GPU, a lease, or a checkpoint +mount, and the two goldens the gates need are already committed: +`tests/parity/goldens/tokenizer_qwen36/tokenizer.json` (248k vocabulary) and +`tests/parity/goldens/tokenizer_mistral/tokenizer.json`. + +The CPU tier can prove, with no other host: + +- Token-identifier equivalence before and after, on both tokenizer families, + over a corpus that contains the failing regime. +- The tie-break order, the stale-entry handling, and the load-time refusal. +- That the cost bound and the growth shape are red before the change and green + after. +- Through `test_tokenizer_parity`, `test_tokenizer_parity_mistral`, + `test_tokenizer_parity_deepseek` and `test_tokenizer_parity_gpt4o`, that all + four already-gated families still produce their recorded identifiers. + +What the CPU tier cannot prove: the end-to-end TTFT of #1365's legs on GB10. +That needs the bf16 27B server on `dgx:gpu0` under an `rc` lease, and it is the +row's closing evidence rather than its correctness gate. Run it only after the +CPU gates are green. + +Required before any speed figure in this spec is accepted: an idle-host, +same-binary A/B re-measure with the load average recorded, because every figure +above was taken between load average 130 and 190. + +## Owed + +Nothing yet. This row files no issue it does not fix. + +## Stop conditions + +- Stop and return `NEEDS_DECISION` if token-identifier equivalence fails on any + corpus entry. A single changed identifier is a correctness regression, not a + tuning question, and no speed result excuses it. +- Stop and return `NEEDS_DECISION` if the load-time vocabulary refusal rejects + any committed golden. That would mean the mirror costs us a checkpoint that + works today, and the trade is a developer decision. +- Stop if the fix needs a change to `src/vllm/tokenizer/pretokenizer.cpp`. The + pretokenizer mirrors an upstream regex and is out of scope; a design that + needs it is the wrong design. +- Do not take a GPU or an `rc` lease for anything in `## Tests`. If a gate seems + to need one, it is measuring the wrong thing. diff --git a/scripts/check-agent-record.py b/scripts/check-agent-record.py index cd4066244..feb388335 100644 --- a/scripts/check-agent-record.py +++ b/scripts/check-agent-record.py @@ -610,8 +610,22 @@ # implementation, `model_loader.cpp:279-303`, reads an existing cache for the # DFlash draft alone and never downloads. `READY`, spec # `specs/hf-model-download.md`, issue #1280. +# 168 since 2026-08-19: +`SPEC-BPE-QUADRATIC-MERGE` (the BPE merge loop is O(n^2) +# in pretoken length, on the request path, before `ValidatePromptLen`). Genuinely +# new and not expressible by the two tokenizer rows beside it: `LOAD-HF-BPE` and +# `LOAD-SENTENCEPIECE` both own a FORMAT -- which `tokenizer.json` shapes parse and +# which token identifiers come out -- and both are token-exact against HF goldens +# today and stay that way. This row changes no identifier at all. It replaces the +# algorithm underneath both of them, and its gate is a COST bound, which is the one +# thing a token gate provably cannot see. It is also not a benchmark row: the encode +# runs synchronously on the HTTP worker five lines before the only length check, so +# `max_model_len` bounds none of it and `/tokenize` reaches it with no engine. +# MEASURED at `31f93787c`: 65,535 bytes of one repeated character costs 28.1-48.1 s +# of one core, and 64 KB of ordinary English prose costs 24.1 s through the committed +# Mistral golden, growing n^2.0. `READY`, spec `specs/bpe-quadratic-merge.md`, +# issue #1365. # Bumped for a real new row, never to make a failing state transition pass. -ENGINE_ROWS = 167 +ENGINE_ROWS = 168 ENGINE_SUMMARY_SECTIONS = ( ("Engine and scheduling", "Engine core and scheduling"), From 9564f7dd5fbf52d165e5b478b96c5e560b26f25b Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 19 Aug 2026 18:12:32 +0000 Subject: [PATCH 2/4] spec(SPEC-BPE-QUADRATIC-MERGE): the growth gate was the flaky shape, and one measurement did not reproduce (#1365) A fresh review returned FAIL on the first revision of this spec with two must-fix findings. Both are repaired here, and every number in the document is re-derived on this box rather than relayed. The growth-ratio gate had the robustness ranking of the two cost gates backwards. Item 3 specified a 2x step, one shot, and called it "the assertion that survives a slow runner, because both halves move together". They do not move together: eight independent single-shot pairs of the heap prototype at 131,072 and 262,144 bytes gave 1.862 to 2.508, a 35% spread, against a usable band of (2.51, 4.0). That is 1.6x of margin on an assertion the spec certified as the robust one. Item 3 now specifies a step of at least 4x and min-of-k: four independent min-of-8 sweeps at 65,536 and 262,144 gave 5.51, 5.74, 6.01 and 6.03, a 9% spread against a band of (6.03, 16). The "survives a slow runner" claim is deleted, and it is item 2, the absolute bound with three orders of magnitude of headroom, that carries that property. The claim that removing the string key alone was measured and is slower is deleted from the spec and from the engine-matrix row. A same-binary A/B did not reproduce the direction, and the original figures were taken at load average 130 to 190. The conclusion is unchanged and now rests on the design argument the spec already made: upstream's staleness test compares new_id (word.rs:197-205) against an identifier-keyed table, which a step that rebuilds a std::string to name a pair cannot express, and removing a constant from a quadratic leaves a quadratic. The spec now says plainly that the key alone is not separately measured. Four smaller findings. FromGguf is a second, unnamed surface for the load-time refusal: it builds the same merge_ranks_ through the same InsertMerge at tokenizer.cpp:864, and tests/vllm/test_bpe.cpp already loads through it, so scope, risks and test item 7 now name it. The recorded identifiers of item 1 now come from HF tokenizers 0.22.2 with the capture method stated, because a baseline taken from our own output is a change detector and cannot support "it cannot go green on a faster wrong answer". The landing shape is stated: one pull request, which is the AGENTS.md default with no recorded preference, and which matters here because W1 lands two assertions red. The unbounded-rule count was wrong: rule 5, MatchWsNewlines, is uncapped too, so it is five of seven and not four, corrected in the spec, in the engine-matrix row and in the issue. Re-derived, on an AMD Ryzen 9 9950X3D at load average 4 to 90, min-of-k throughout, through the committed goldens rather than a private merge table: 64 KB of English costs 25,345.61 ms on the Mistral arm where HF tokenizers 0.22.2 reading the identical file costs 10.11 ms, a factor of 2,507, and the two id sequences are byte-identical at 8,000 bytes. The fit over 1 KB to 64 KB has exponent 2.01. 8,192 newlines are one pretoken at 580.44 ms. The heap prototype now matches today's identifiers on 20 comparisons across both committed goldens, all 20 of which also match HF, which closes the gap that every prototype input was byte-level and none exercised EncodePlainSp. One repair is not a review finding. `check-pr-size.py` was red from the spec commit, which bumped `ENGINE_ROWS` 167 to 168 with no case in `tests/scripts/test_agent_record.py`, and only continuous integration enforces that rule so preflight never saw it. `BpeQuadraticMergeRowIsCounted` is added in the shape its two precedents use, before the `__main__` guard so it actually runs. The four cases are green at 168 and two of them are red at 167, applied and restored byte for byte. The class asserts nothing about `.agents/issue-index.md`, where its siblings assert an issue number, because #1365's row landed before the issue was re-scoped and still describes the symptom, so that assertion would pass while measuring nothing about this row. No issue-index row is appended, and that is the finished state rather than a deferral. #1365's row already landed in 9e1a5e573 (PR #1369) at 13:05:11Z on 19 August 2026, and a second row for one issue number is what check-agent-record.py reports as "issue #1365 listed twice". The issue was renamed and re-scoped in place at 17:05:51Z the same day, so the landed row's text describes the symptom while the issue now describes the cause. The index is append-only, so the row is not edited, and ## Dependencies is where the reader is told the row reads as the original symptom report. `origin/main` moved twice during this work and the branch is rebased onto it rather than merged, so no merge commit without a body reaches the trailer gates. The second rebase conflicted on the engine matrix summary, where both sides had made a scoped edit to one keyed record: `origin/main` moved a speculative-decode row from READY to ACTIVE, and this branch adds a READY row to loading and tokenizer. The `origin/main` version is taken complete, this row's edit is re-applied on top, and every column of the Total is re-derived as the column sum of the ten section rows rather than taken from either side. Every other keyed row in the matrix is byte-for-byte equal to `origin/main` except `ENG-RECORD-ANCHOR-RATCHET`, whose five citations into `scripts/check-agent-record.py` moved by the comment the spec commit added, and each of those five is re-derived by locating its symbol in the merged file rather than by arithmetic. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [Claude Code] --- .agents/engine-matrix.md | 2 +- .agents/specs/bpe-quadratic-merge.md | 467 +++++++++++++++++++++------ scripts/check-agent-record.py | 8 +- tests/scripts/test_agent_record.py | 87 +++++ 4 files changed, 455 insertions(+), 109 deletions(-) diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index 7a42aa77b..c2051c008 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -277,7 +277,7 @@ claims it. | `LOAD-GGUF-MMPROJ` | A SECOND, `clip`-architecture GGUF projector file beside the language file, and the Qwen3-VL vision tower loaded out of it. The single-file assumption is structural, not incidental: `ModelSource` carries a VECTOR of safetensors shards and exactly ONE `GgufFile*`, and `EngineParams` has no projector field at all, so an mmproj has nowhere to arrive. `GgufFile::Open` does handle SHARDS (`DetectSplit` parses `-NNNNN-of-MMMMM.gguf`), but shards of one split are not a second, differently-architected file. Nothing in this tree loads a `clip` projector today and the two apparent counterexamples are neither: MiniMax-H3 reads `visual.*` out of the SAME encoder GGUF and only from a test, and MuseGlimmer's mmproj path is a REFUSAL whose only caller is a test — there is no production call site because no production path accepts a second GGUF path. First consumer is `Qwen3.8-27B` ([#821](https://github.com/mudler/vllm.cpp/issues/821)), whose `mmproj-BF16.gguf` (334 tensors, `clip.projector_type = qwen3vl_merger`, header-verified 2026-08-18: data end == file size 931,146,432) ships BOTH halves of the temporal patch embedding as `v.patch_embd.weight` + `v.patch_embd.weight.1` — precisely what MuseGlimmer's lacks, so the check that distinguishes them is this row's, not a reason to refuse. Second consumer is MuseGlimmer, whose refusal becomes REACHABLE production code the moment this lands | T0 | Pinned vLLM has no GGUF loader at all (`555967922`, `model_loader/__init__.py:33-49`), so the compatibility reference is llama.cpp `b10451` = `10bf611e5` (`PROJECTOR_TYPE_QWEN3VL`; the previously recorded `tools/mtmd/clip-impl.h:330` was read at the SUPERSEDED local fork `237ad9b96` and is owed re-anchoring, [#1003](https://github.com/mudler/vllm.cpp/issues/1003)) | the one open site [model_loader.cpp:1679](../src/vllm/entrypoints/model_loader.cpp#L1679), the single pointer [model_registry.h:98](../include/vllm/model_executor/models/model_registry.h#L98), the arch dispatch [model_loader.cpp:794](../src/vllm/entrypoints/model_loader.cpp#L794), the refusal precedent [muse_glimmer_gguf_weights.cpp:695](../src/vllm/model_executor/models/muse_glimmer_gguf_weights.cpp#L695) | none yet — this row is spec-only. The port target for the vision config mapping is [minimax_h3_vision_gguf.cpp:32](../src/vllm/model_executor/models/minimax_h3_vision_gguf.cpp#L32), which builds the same `Qwen3VLVisionConfig` from `visual.*` | [quantized arms of Qwen3.8-27B](specs/qwen38-27b-quant-arms.md) | `READY` | - | | `LOAD-HF-BPE` | HF tokenizer.json byte-level BPE and incremental detokenization | T0 | `vllm/tokenizers/registry.py:176`; `vllm/tokenizers/hf.py:163`; `tests/tokenizers_/test_hf.py:18`; `tests/tokenizers_/test_detokenize.py:148` | `src/vllm/tokenizer/tokenizer.cpp:240,484,512`; `src/vllm/v1/engine/detokenizer.cpp:409` | `tests/vllm/test_bpe.cpp:226`; `tests/vllm/test_tokenizer_parity.cpp:66,74,82,90`; `tests/vllm/test_pretokenizer.cpp:130` | `planned: specs/hf-tokenizer.md` | `ANCHOR-BACKFILL` | - | | `LOAD-SENTENCEPIECE` | SentencePiece (Metaspace + byte-fallback) BPE tokenizer family — the gate to Mistral/Gemma/SentencePiece `tokenizer.json`. `Tokenizer::FromHfJson` now dispatches on the pre_tokenizer FAMILY: a bare `Metaspace` node selects the SP path (space→▁ U+2581; `prepend_scheme` first/always/never with the starts-with-▁ guard; `split=false`; out-of-vocab char → `<0xNN>` byte-fallback tokens; merge-ranked BPE over the raw-UTF-8 string), else the BYTE-IDENTICAL byte-level path (`DetectPattern` still fails loudly on Metaspace, so families never overlap). Decode + the incremental detokenizer mirror HF's Sequence decoder (Replace ▁→space, ByteFallback, Fuse, Strip 1 leading space). **UNBLOCKS the Mistral (`MODEL-TEXT-mistral-mistral-for-causal-lm`) paged-engine SACRED gate.** `split=true` fails loudly (no golden in scope). Mirrors HF `tokenizers` 0.22 | T1 | `vllm/tokenizers/hf.py:163`; `vllm/tokenizers/mistral.py:235,467`; HF `tokenizers` 0.22 `pre_tokenizers/metaspace.rs`, `models/bpe/model.rs::merge_word`, `decoders/{replace,byte_fallback,fuse,strip}.rs` | dispatch `src/vllm/tokenizer/tokenizer.cpp:184` (`DetectMetaspace`), `:732` (`EncodePlainSp`), `:878` (`SpDecodeTokens`), `:927` (`Decode` SP branch); merge-loop factor `src/vllm/tokenizer/bpe.cpp:91` (`BpeMerge`); family/params/accessors `include/vllm/tokenizer/tokenizer.h:88,98`; incremental dispatch `src/vllm/v1/engine/detokenizer.cpp:357`; generator `tools/parity/dump_tokenizer_mistral.py` | `tests/vllm/test_tokenizer_parity_mistral.cpp:78` **6/6, 421 assertions byte-exact** vs HF `tokenizers` 0.22.2 (= vLLM 0.25.0 backend) over a 45-entry Metaspace/byte-fallback/special-token corpus (goldens `tests/parity/goldens/tokenizer_mistral/{tokenizer.json,encodings.json}`); SACRED cross-check vLLM `AutoTokenizer` (transformers 5.13.1) 0/45 mismatch (±BOS); byte-level suites byte-identical `tests/vllm/test_bpe.cpp` 852, `tests/vllm/test_detokenizer.cpp` 221, `tests/vllm/test_tokenizer_parity.cpp` 1175, `tests/vllm/test_tokenizer_parity_deepseek.cpp` 2461 | [specs/sentencepiece.md](specs/sentencepiece.md) | `ANCHOR-BACKFILL` | `CLAIM-LOAD-SENTENCEPIECE` | -| `SPEC-BPE-QUADRATIC-MERGE` | **The BPE merge loop is O(n^2) in pretoken length, on the request path, before the length check.** `src/vllm/tokenizer/bpe.cpp::BpeMerge` rescans every adjacent pair per merge and builds one `std::string` key per probe through `src/vllm/tokenizer/bpe.cpp::MergeKey`, on the premise its own comment states: "pretokens are tiny". Four of the seven pretokenizer rules return an unbounded run (`src/vllm/tokenizer/pretokenizer.cpp::MatchLetterRun`, `src/vllm/tokenizer/pretokenizer.cpp::MatchPunctRun`, `src/vllm/tokenizer/pretokenizer.cpp::MatchWsNotBeforeNonSpace`, `src/vllm/tokenizer/pretokenizer.cpp::MatchWs`), and on the SentencePiece family `src/vllm/tokenizer/tokenizer.cpp::EncodePlainSp` merges the WHOLE prompt as one word because Mistral and Gemma declare `split: false`. MEASURED at `31f93787c`: 65,535 bytes of one repeated character costs 28.1-48.1 s of one core; 64 KB of ordinary English prose through the committed Mistral golden costs 24.1 s, growing n^2.0. `src/vllm/v1/engine/input_processor.cpp::ValidatePromptLen` runs FIVE LINES AFTER the encode, so `max_model_len` bounds none of it, and `/tokenize` reaches the same path with no engine. This is a remote denial of service, not a latency curiosity. Fix mirrors HF `tokenizers` 0.22.2 `Word::merge_all` (heap over candidate merges, identifier-keyed merge table); a prototype is bit-identical on ten inputs and 3,459x faster on the 65,535-byte case. A pretoken cap is REFUSED: it changes the token identifiers. Removing the string key alone was MEASURED and is slower, so it is not a separable win | T0 | HF `tokenizers` 0.22.2 `tokenizers/src/models/bpe/word.rs:162-250` (`Word::merge_all`), `:28-35` (`Ord for Merge`), `tokenizers/src/models/bpe/mod.rs:9` + `model.rs:19,174-192` (identifier-keyed merge table and its load-time refusal) | `src/vllm/tokenizer/bpe.cpp::BpeMerge` (the defect); callers `src/vllm/tokenizer/tokenizer.cpp::EncodePlain` and `src/vllm/tokenizer/tokenizer.cpp::EncodePlainSp` | none yet — the corpus at `tests/parity/goldens/tokenizer_qwen36/` has a longest pretoken of 54 bytes, which is why every existing gate is green | [specs/bpe-quadratic-merge.md](specs/bpe-quadratic-merge.md) | `READY` | - | +| `SPEC-BPE-QUADRATIC-MERGE` | **The BPE merge loop is O(n^2) in pretoken length, on the request path, before the length check.** `src/vllm/tokenizer/bpe.cpp::BpeMerge` rescans every adjacent pair per merge and builds one `std::string` key per probe through `src/vllm/tokenizer/bpe.cpp::MergeKey`, on the premise its own comment states: "pretokens are tiny". FIVE of the seven pretokenizer rules return an unbounded run (`src/vllm/tokenizer/pretokenizer.cpp::MatchLetterRun` rule 2, `src/vllm/tokenizer/pretokenizer.cpp::MatchPunctRun` rule 4, `src/vllm/tokenizer/pretokenizer.cpp::MatchWsNewlines` rule 5, `src/vllm/tokenizer/pretokenizer.cpp::MatchWsNotBeforeNonSpace` rule 6, `src/vllm/tokenizer/pretokenizer.cpp::MatchWs` rule 7); only rule 3 `MatchNumbers` is capped and only rule 1 `MatchContraction` is bounded by its own alternation, and on the SentencePiece family `src/vllm/tokenizer/tokenizer.cpp::EncodePlainSp` merges the WHOLE prompt as one word because Mistral and Gemma declare `split: false`. MEASURED at `31f93787c`: 65,535 bytes of one repeated character costs 28.1-48.1 s of one core; 64 KB of ordinary English prose through the committed Mistral golden costs 24.1 s, growing n^2.0. `src/vllm/v1/engine/input_processor.cpp::ValidatePromptLen` runs FIVE LINES AFTER the encode, so `max_model_len` bounds none of it, and `/tokenize` reaches the same path with no engine. This is a remote denial of service, not a latency curiosity. Fix mirrors HF `tokenizers` 0.22.2 `Word::merge_all` (heap over candidate merges, identifier-keyed merge table); a prototype is bit-identical to today's identifiers on 20 comparisons across BOTH committed goldens, all 20 of which also match HF `tokenizers` 0.22.2 reading the same files, and 8,192 newlines fall from 580.44 ms to 2.60 ms. A pretoken cap is REFUSED: it changes the token identifiers. The string key is not a separable win either, on the DESIGN: upstream's staleness test compares `new_id` (`word.rs:197-205`) and its table is keyed on an identifier pair, so a step that rebuilds a `std::string` to name a pair cannot express the test; and removing a constant from a quadratic leaves a quadratic. How much the key alone is worth is NOT measured on an idle host and this row claims nothing about it | T0 | HF `tokenizers` 0.22.2 `tokenizers/src/models/bpe/word.rs:162-250` (`Word::merge_all`), `:28-35` (`Ord for Merge`), `tokenizers/src/models/bpe/mod.rs:9` + `model.rs:19,174-192` (identifier-keyed merge table and its load-time refusal) | `src/vllm/tokenizer/bpe.cpp::BpeMerge` (the defect); callers `src/vllm/tokenizer/tokenizer.cpp::EncodePlain` and `src/vllm/tokenizer/tokenizer.cpp::EncodePlainSp` | none yet — the corpus at `tests/parity/goldens/tokenizer_qwen36/` has a longest pretoken of 54 bytes, which is why every existing gate is green | [specs/bpe-quadratic-merge.md](specs/bpe-quadratic-merge.md) | `READY` | - | | `LOAD-CONFIG-SURFACE` | Dataclass-for-dataclass config and serve-compatible flags | T0/T1 | `vllm/config/scheduler.py:26`; `vllm/config/cache.py:25`; `vllm/config/compilation.py:378` | `include/vllm/config/scheduler.h:67`; `src/vllm/config/scheduler.cpp:11`; `src/vllm/transformers_utils/hf_config.cpp:83`; limited flags incl. `max_num_seqs`/`max_num_batched_tokens` `examples/server/main.cpp:63,116,170` | `tests/vllm/test_scheduler_config.cpp:10`; `tests/vllm/test_hf_config.cpp:131,224,265`; `examples/CMakeLists.txt:34` | `planned: specs/config-surface.md` | `PARTIAL` | - | | `ENG-HF-MODEL-DOWNLOAD` | Fetch a checkpoint from HuggingFace so `--model` accepts a repository identifier and not a local path only. Two forms behind one flag, with the local path probed first: `org/repo` mirrors vLLM's full-snapshot download, and `org/repo:Q4_K_M` fetches one GGUF file, a form vLLM does not implement and which llama.cpp supplies as the secondary oracle. Mirrors vLLM's two-phase fetch (config JSON first, so a bad repository fails after 200 KB and not 60 GB) and its index-driven file selection, which reads `model.safetensors.index.json` and fetches the exact names in `weight_map` instead of every match for `*.safetensors`. Reference is resolved to a commit before any byte is fetched, so a moving `main` cannot change what a second run loads. Cache layout is HuggingFace's documented local cache, so a host holding a Python `huggingface_hub` cache gets a hit and `model_loader.cpp:279-303` reads it unchanged. Transport is the already-vendored cpp-httplib with OpenSSL, matching llama.cpp, which retired libcurl at `CMakeLists.txt:195` at stock tag `b10451`. NOT a verbatim port on one point: llama.cpp's `is_valid_oid` accepts any 40- or 64-character hexadecimal string, and on 17 August 2026 this project measured the tree API answering an unauthenticated caller on gated `Lightricks/LTX-2.5` with an `lfs.oid` of one character repeated 64 times, identical for all 14 large-file-storage files, so this row treats an untokenized `lfs.oid` as absent and proves completeness structurally instead | T1 | `vllm/model_executor/model_loader/weight_utils.py:345,349-357,459,472-490,493-496,506`; `vllm/model_executor/model_loader/default_loader.py:167-184`; `vllm/engine/arg_utils.py:839`; `vllm/config/model.py:183` at pin `5559679229`. Secondary oracle llama.cpp for the `:QUANT` form only, stock tag `b10451`, anchors re-verified by W1 at commit `10bf611e533d81f739128304991c5e133c6aebd8` | W1 and W2 only: `src/vllm/transformers_utils/hf_hub.cpp:1`, `src/vllm/transformers_utils/hf_cache.cpp:1`, reached from `src/vllm/entrypoints/model_loader.cpp:280`. No downloader, no `--model` grammar, no TLS option | `tests/vllm/transformers_utils/test_hf_cache.cpp:1`, `tests/vllm/transformers_utils/test_hf_hub.cpp:1` (in-process fake hub), `tests/vllm/entrypoints/test_dflash_draft_hf_cache.cpp:1` (the loader reach, red when the call site is deleted) | [hf-model-download.md](specs/hf-model-download.md) | `READY` | - | | `LOAD-LONGTAIL` | Sharded-state, tensorizer, RunAI, BitsAndBytes loaders | T3 | `vllm/model_executor/model_loader/__init__.py:33-65` | - | - | `planned: specs/loader-longtail.md` | `INVENTORIED` | - | diff --git a/.agents/specs/bpe-quadratic-merge.md b/.agents/specs/bpe-quadratic-merge.md index 9457e9059..22d4c0a7d 100644 --- a/.agents/specs/bpe-quadratic-merge.md +++ b/.agents/specs/bpe-quadratic-merge.md @@ -8,7 +8,18 @@ every leg" to the cause measured below. surface. Every gate in this spec runs on a CPU host. **Row:** `SPEC-BPE-QUADRATIC-MERGE` in [`engine-matrix.md`](../engine-matrix.md), section "Loading, tokenizer, and config". -**Base:** `31f93787c`. Every local line number in this document is read there. +**Base:** `31f93787c`, merged with `origin/main` at `c9724b5ee`. Every local +line number in this document is read at that merge. `git diff 31f93787c HEAD -- +src/vllm/tokenizer include/vllm/tokenizer src/vllm/v1/engine/input_processor.cpp +src/vllm/entrypoints/openai/api_server.cpp` is empty, so the merge moves none of +them. +**Pull request shape:** ONE pull request carrying W1 through W4. No answer is +recorded for this row under `## Git integration` in +`.agents/developer-preferences.md`. The only recorded answer there is +`SPEC-DFLASH2`'s, and AGENTS.md makes one pull request the default when no +answer is recorded. The shape is not cosmetic here: W1 lands the cost and growth +assertions of `## Tests to port` items 2 and 3 RED against unchanged code, so a +split would leave `main` red between W1 and W3. See `## Work breakdown`. ## Now @@ -24,11 +35,13 @@ void BpeMerge(std::vector& symbols, const MergeRanks& ranks) { // (strict < keeps the first best). O(n^2) scan; pretokens are tiny. ``` -Nothing bounds a pretoken. Four of the seven pretokenizer rules return a run of +Nothing bounds a pretoken. Five of the seven pretokenizer rules return a run of unbounded length, and on the SentencePiece family the pretoken is the entire -prompt. Measured on this tree, 64 KB of the letter `a` costs 28.1 s of one core -and 64 KB of ASCII spaces costs 48.1 s, both on a path that runs before any -length check. The row is `READY`: this spec is committed before any product +prompt. Measured on this tree, 64 KB of the letter `a` costs 24.4 s of one core +and 64 KB of ASCII spaces costs 45.8 s, both on a path that runs before any +length check. HF `tokenizers` 0.22.2, reading the same committed tokenizer file, +returns the identical token identifiers for 64 KB of English in 10.1 ms against +our 25.3 s. The row is `READY`: this spec is committed before any product code, because the fix replaces the merge algorithm on a token-exact path and the design has to be agreed before anyone edits the tokenizer. @@ -38,12 +51,20 @@ In scope: - `src/vllm/tokenizer/bpe.cpp::BpeMerge`, replaced by a heap-driven merge that mirrors HF `tokenizers` `Word::merge_all`. -- Whatever `src/vllm/tokenizer/bpe.h` has to expose so that +- Whatever `include/vllm/tokenizer/bpe.h` has to expose so that `src/vllm/tokenizer/tokenizer.cpp::EncodePlain` and `src/vllm/tokenizer/tokenizer.cpp::EncodePlainSp`, the only two callers, reach the new form. - The interning decision described under `## Design`, and the load-time refusal - it needs. + it needs, on **both** load surfaces. `merge_ranks_` is built from + `tokenizer.json` by `src/vllm/tokenizer/tokenizer.cpp::FromHfJson` + (`InsertMerge` at `src/vllm/tokenizer/tokenizer.cpp:680`) and from a GGUF's + `tokenizer.ggml.merges` by `src/vllm/tokenizer/tokenizer.cpp::FromGguf` + (`src/vllm/tokenizer/tokenizer.cpp:741`, the same `InsertMerge` at `:864`). + Both reach the same table and the same `BpeMerge`, so the refusal has to be + written once where the table is built and gated on both. `FromGguf` is already + gated: `tests/vllm/test_bpe.cpp` builds a zero-tensor GGUF and loads it + through `Tokenizer::FromGguf`. - Red-first equivalence and cost cases in `tests/vllm/test_bpe.cpp` and one new CPU-tier binary described under `## Tests`. - `docs/STATUS.md` and `docs/BENCHMARKS.md` when the row moves to `DONE`. @@ -64,20 +85,29 @@ Out of scope, each with a stated reason: ## Our baseline -Every figure below was reproduced in this row, on an AMD Ryzen 9 9950X3D at -`-O2`, against a driver built from this tree's own `bpe.cpp`, -`pretokenizer.cpp` and `unicode_data.cpp`. The box carried a load average -between 130 and 190, so these are contended numbers. The **shape** is the -result. A same-binary idle re-measure belongs to the implementing row, and +Every figure below was measured on an AMD Ryzen 9 9950X3D (20 cores) at `-O2`, +against a driver built from this tree's own tokenizer sources. **Two harnesses +produced them and the tables say which.** + +- **Harness A**, the one that found the defect: `BpeSplit` alone, against the + `merges.txt` of `qwen3.8-27b-hf`, 247,586 merges, sha256 + `a9d356d7bdf1ef4949e3e748e95b8e10ad9d4e2e838eddc38a0a7b6b94d1db8d`, at a load + average between 130 and 190. Those are heavily contended single-shot numbers. +- **Harness B**, the re-derivation: full `Tokenizer::Encode` through the two + committed goldens, min-of-k, at a load average between 4 and 90 on the same + box, 2026-08-19. Every figure carries its `k` and the load. The box was not + idle: another session was compiling throughout, which is why the minimum of a + repetition set is reported rather than a mean. The minimum is the least + contended estimate available, not a claim of an idle host. + +The **shape** is what both harnesses agree on, and it is the result. A +same-binary idle-host re-measure still belongs to the implementing row, and `## Gates` requires it before any speed claim is accepted. -Merge table: `merges.txt` of `qwen3.8-27b-hf`, 247,586 merges, sha256 -`a9d356d7bdf1ef4949e3e748e95b8e10ad9d4e2e838eddc38a0a7b6b94d1db8d`. - ### The benchmark prompts that found it -The six reconstructed `vllm bench serve` prompts of the #1365 legs, timing -`BpeSplit` only: +Harness A. The six reconstructed `vllm bench serve` prompts of the #1365 legs, +timing `BpeSplit` only: | prompt | bytes | pretokens | longest pretoken | current | heap prototype | |---|---:|---:|---:|---:|---:| @@ -92,7 +122,7 @@ Index 2 is the outlier request of #1365's four legs. It is one pretoken. ### The growth is quadratic -Prompt 2 truncated and tripled, one pretoken throughout: +Harness A. Prompt 2 truncated and tripled, one pretoken throughout: | bytes | BPE merge | |---:|---:| @@ -104,35 +134,67 @@ Prompt 2 truncated and tripled, one pretoken throughout: Three times the bytes costs 7.65 times the time. -### Four rules are unbounded, not one - -`src/vllm/tokenizer/pretokenizer.cpp::MatchLetterRun` (rule 2), -`src/vllm/tokenizer/pretokenizer.cpp::MatchPunctRun` (rule 4), -`src/vllm/tokenizer/pretokenizer.cpp::MatchWsNotBeforeNonSpace` (rule 6) and -`src/vllm/tokenizer/pretokenizer.cpp::MatchWs` (rule 7) each scan `while (p < -t.size())` with no cap. Only -`src/vllm/tokenizer/pretokenizer.cpp::MatchNumbers` is capped, at one codepoint -for Qwen and three for Llama-3. Each input below is 8,192 bytes returned as one -pretoken: - -| input | rule | current | heap prototype | -|---|---|---:|---:| -| `a` x 8,192 | 2, letter run | 514.19 ms | 2.33 ms | -| `中` x 2,731 | 2, letter run | 438.58 ms | 0.85 ms | -| `ก` x 2,731 | 2, letter run | 505.81 ms | not run | -| `~` x 8,192 | 4, punct run | 451.81 ms | 2.20 ms | -| space x 8,192 | 7, whitespace | 813.02 ms | 3.16 ms | +### Five rules are unbounded, not one + +The Qwen and Llama-3 alternation has exactly seven rules, dispatched at +`src/vllm/tokenizer/pretokenizer.cpp:810-821`. **Five of them scan +`while (p < t.size())` with no cap:** + +| rule | function | bound | +|---:|---|---| +| 1 | `MatchContraction` (`:82`) | **bounded**: a fixed alternation, at most three bytes | +| 2 | `MatchLetterRun` (`:106`); `MatchTekkenLetterRun` (`:193`) on Tekken | unbounded | +| 3 | `MatchNumbers` (`:219`) | **capped** at `max_digits`: one codepoint for Qwen, three for Llama-3 | +| 4 | `MatchPunctRun` (`:241`) | unbounded | +| 5 | `MatchWsNewlines` (`:269`) | unbounded | +| 6 | `MatchWsNotBeforeNonSpace` (`:287`) | unbounded | +| 7 | `MatchWs` (`:303`) | unbounded | + +Rule 5 was missed by the first revision of this spec and by #1365 as first +re-scoped. It matches `\s*[\r\n]+`, so a block of newlines is one pretoken and +pays the same cost. A pasted log and a pasted diff both have that shape. Two +rules are bounded, and only one of those two is bounded by a *cap*: saying +"only `MatchNumbers` is capped" is true of rule 3 and says nothing about rule 1, +which is bounded by its own alternation rather than by a counter. + +Harness B, 8,192 bytes of one character, `Tokenizer::Encode` through the +committed `tests/parity/goldens/tokenizer_qwen36/tokenizer.json`, min-of-5, load +average 33 to 37: + +| input | rule | pretokens | current | heap prototype | +|---|---|---:|---:|---:| +| `a` x 8,192 | 2, letter run | 1 | 467.59 ms | 1.96 ms | +| `中` x 2,730 (+2 B) | 2, letter run | 2, longest 8,190 | 433.12 ms | 0.81 ms | +| `ก` x 2,730 (+2 B) | 2, letter run | 2, longest 8,190 | 532.33 ms | 1.14 ms | +| `的` x 2,730 (+2 B) | 2, letter run | 2, longest 8,190 | 389.74 ms | 0.82 ms | +| `~` x 8,192 | 4, punct run | 1 | 414.21 ms | 1.91 ms | +| **newline x 8,192** | **5, `\s*[\r\n]+`** | **1** | **580.44 ms** | **2.60 ms** | +| space x 8,192 | 7, whitespace | 1 | 618.56 ms | 2.84 ms | + +The pretoken counts are read from `vllm::tok::Pretokenize` itself, not inferred: +the CJK and Thai inputs are 2,730 codepoints plus a 2-byte remainder, which is a +second pretoken, so their longest run is 8,190 rather than 8,192. The trigger is a long run of one character class, not one script. ASCII reaches -it. +it, and so does an empty-looking block of newlines. ### At 65,535 bytes -| input | current | heap prototype | -|---|---:|---:| -| `的` x 21,845 | 28,746.70 ms | 8.31 ms | -| `a` x 65,535 | 28,130.35 ms | not run | -| space x 65,535 | 48,098.11 ms | not run | +Harness B, `Tokenizer::Encode` through the committed Qwen3.6 golden. Every input +is one pretoken of 65,535 bytes, read from `vllm::tok::Pretokenize`. Current +min-of-2, prototype min-of-5, load average 25 to 90. The current column is +min-of-2 rather than min-of-5 because each repetition costs about 30 s of one +core, which is itself the finding: + +| input | rule | current | heap prototype | ratio | +|---|---|---:|---:|---:| +| `a` x 65,535 | 2 | 24,358.86 ms | 20.71 ms | 1,176x | +| `中` x 21,845 | 2 | 26,823.67 ms | 8.00 ms | 3,353x | +| `ก` x 21,845 | 2 | 39,122.71 ms | 10.72 ms | 3,649x | +| `的` x 21,845 | 2 | 30,309.46 ms | 7.40 ms | 4,096x | +| `~` x 65,535 | 4 | 32,468.48 ms | 18.85 ms | 1,723x | +| newline x 65,535 | 5 | 40,538.57 ms | 24.14 ms | 1,679x | +| space x 65,535 | 7 | 45,780.21 ms | 28.07 ms | 1,631x | ### The SentencePiece family pays it on ordinary English @@ -146,22 +208,38 @@ metaspace layout, and the committed Mistral golden `"split": false` in its own `pre_tokenizer`. Full `Tokenizer::Encode` through that committed golden, on "The quick brown fox -jumps over the lazy dog. " repeated to length. The Qwen3.6 golden is the -byte-level control on the identical text: - -| prompt bytes | Mistral golden | Qwen3.6 golden | -|---:|---:|---:| -| 1,000 | 5.59 ms | 0.15 ms | -| 2,000 | 21.73 ms | 0.11 ms | -| 4,000 | 87.15 ms | 0.23 ms | -| 8,000 | 357.01 ms | 0.47 ms | -| 16,000 | 1,410.42 ms | 0.92 ms | -| 32,000 | 5,804.42 ms | not run | -| 65,536 | 24,144.72 ms | not run | - -Sixteen times the bytes costs 252 times the time. That exponent is 2.0. A -Mistral or Gemma server spends 24 s of one core to tokenize a 64 KB English -document, and a user who pastes a document sends exactly that. +jumps over the lazy dog. " repeated to length. Two controls run the identical +text: the committed Qwen3.6 golden, which is the byte-level arm of our own +tokenizer, and **HF `tokenizers` 0.22.2 reading the same Mistral +`tokenizer.json` file**, which is the code the pinned `transformers` executes +and therefore the oracle for both the identifiers and the cost. Harness B, +min-of-3 for the two Mistral columns, min-of-5 for the Qwen3.6 control, load +average 4 to 12: + +| prompt bytes | ids | ours, Mistral golden | HF `tokenizers` 0.22.2, same file | ours / HF | ours, Qwen3.6 control | +|---:|---:|---:|---:|---:|---:| +| 1,000 | 267 | 5.61 ms | 0.14 ms | 40x | 0.058 ms | +| 8,000 | 2,134 | 358.83 ms | 1.09 ms | 329x | 0.457 ms | +| 32,000 | 8,534 | 6,005.56 ms | 4.92 ms | 1,221x | 1.883 ms | +| 65,536 | 17,476 | 25,345.61 ms | 10.11 ms | **2,507x** | 3.877 ms | + +**The identifiers are the same on both sides**: the `ids` column is one number +because ours and HF's agree at every size, and at 8,000 bytes the two id +sequences were compared element by element and are byte-identical (2,134 of +2,134). So the 2,507x is a pure cost difference on identical output, not a +different answer computed faster. + +Fitting our Mistral column, 65.536x the bytes costs 4,518x the time, an exponent +of **2.01**. Read at a single 4x step it is plainer still: 8,000 to 32,000 bytes +costs 16.74x, against the 16x that `n^2` predicts and the ~4.6x that +`n log n` does (`4 * log2(32000) / log2(8000)`, the word being the whole prompt +on this arm, so `n` is the prompt length in codepoints). The Qwen3.6 control +over the same 65.536x range costs 66.8x, which is linear. HF over the same range +costs 72x, also linear. + +A Mistral or Gemma server spends 25 s of one core to tokenize a 64 KB English +document that HF tokenizes in 10 ms, and a user who pastes a document sends +exactly that. ### Why no gate saw it @@ -207,7 +285,8 @@ that crate holds the merge loop. So the executing chain for a vLLM encode is: | `transformers` 5.14.1 | the `tokenizer.json` contract | the pin resolved inside the pinned vLLM environment, [`../oracles/transformers.md`](../oracles/transformers.md) | | `tokenizers` 0.22.2 | **the merge algorithm** | `tokenizers/src/models/bpe/word.rs:162-250` (`Word::merge_all`) | | `tokenizers` 0.22.2 | the merge-table representation | `tokenizers/src/models/bpe/mod.rs:9` (`type Pair = (u32, u32)`), `model.rs:19` (`MergeMap`), `model.rs:174-192` (built at load, refuses an out-of-vocabulary merge) | -| `tokenizers` 0.22.2 | the tie-break order | `tokenizers/src/models/bpe/word.rs:28-35` (`Ord for Merge`) | +| `tokenizers` 0.22.2 | the tie-break order | `tokenizers/src/models/bpe/word.rs:28-36` (`Ord for Merge`) | +| `tokenizers` 0.22.2 | what a stale heap entry is | `tokenizers/src/models/bpe/word.rs:197-205`, which compares `new_id`, NOT the pair | | `tokenizers` 0.22.2 | the symbol list | `tokenizers/src/models/bpe/word.rs:37-53` (`Symbol`, `merge_with`) | | `tokenizers` 0.22.2 | word construction, byte fallback, unk fusing | `tokenizers/src/models/bpe/model.rs:382-460` (`merge_word`) | | `tokenizers` 0.22.2 | the word cache, out of scope here | `model.rs:475-496`, `tokenizers/src/utils/cache.rs:7,10` | @@ -226,9 +305,10 @@ selects this version. The implementing row records the version it read. | `word.rs:162-250` `Word::merge_all` | `src/vllm/tokenizer/bpe.cpp::BpeMerge`, an O(n^2) rescan | the same signature, heap-driven | | `word.rs:37-43` `Symbol {c, prev, next, len}` | `std::vector` with `erase` per merge | a linked symbol list with `len = 0` tombstones | | `word.rs:28-35` `Ord for Merge` | `strict <` inside the scan | an explicit comparator, with the tie pinned by a test | -| `mod.rs:9`, `model.rs:19` identifier-keyed `MergeMap` | `MergeRanks = unordered_map` in `src/vllm/tokenizer/bpe.h` | an identifier-pair-keyed table built at load | +| `mod.rs:9`, `model.rs:19` identifier-keyed `MergeMap` | `MergeRanks = unordered_map` in `include/vllm/tokenizer/bpe.h` | an identifier-pair-keyed table built at load | | `src/vllm/tokenizer/bpe.cpp::MergeKey` | one `std::string` per probe | deleted; there is no key to build | -| `model.rs:180-189` `MergeTokenOutOfVocabulary` | no rule; the failure appears per request in `src/vllm/tokenizer/tokenizer.cpp::EncodePlain` | refused at load, naming the missing token | +| `model.rs:180-189` `MergeTokenOutOfVocabulary` | no rule; the failure appears per request in `src/vllm/tokenizer/tokenizer.cpp::EncodePlain` | refused at load, naming the missing token, on `FromHfJson` AND `FromGguf` | +| `model.rs:169-173`, `:186` `new_token = format!("{}{}", a, &b[prefix_len..])` | no counterpart: `src/vllm/tokenizer/tokenizer.cpp::MergeKey` concatenates `a` and `b` whole | still whole. `prefix_len` is `continuing_subword_prefix.len()`, and `src/vllm/tokenizer/tokenizer.cpp:624-631` already REFUSES a non-empty `continuing_subword_prefix` at load, so `prefix_len` is 0 on every checkpoint we accept and the term is inert for us. Port the concatenation without it, and do not silently drop the refusal that makes that legal | | `model.rs:382-460` `merge_word` | `src/vllm/tokenizer/tokenizer.cpp::EncodePlainSp`'s symbol builder | unchanged in behaviour, emitting identifiers | | `model.rs:475-496` word cache | absent | still absent, and out of scope | @@ -251,13 +331,20 @@ selects this version. The implementing row records the version it read. (`word.rs:217-244`). 4. A popped candidate is validated before use rather than removed when it goes stale: skip it when its left symbol was removed (`word.rs:187`), when it has - no right neighbour (`word.rs:191`), or when the pair it names is no longer - the pair at that position (`word.rs:198-205`). + no right neighbour (`word.rs:191`), or when the merge table's `new_id` for + the pair now at that position differs from the entry's own `new_id` + (`word.rs:197-205`). **The third test compares `new_id`, not the pair.** + Upstream's line is + `merges.get(&target_new_pair).is_none_or(|(_, new_id)| *new_id != top.new_id)` + (`word.rs:200-203`), so a table with two distinct pairs mapping to one + `new_id` accepts the entry where a pair comparison would reject it. Port the + `new_id` comparison, and note that this is only expressible once the table is + identifier-keyed, which is the second reason W2 comes before W3. That is O(n log n) merges of O(1) work each, against our O(n) rescan per merge. **The order must stay identical, and upstream makes it explicit.** `Ord for -Merge` (`word.rs:28-35`) inverts both comparisons, so the max-heap yields the +Merge` (`word.rs:28-36`) inverts both comparisons, so the max-heap yields the lowest `rank` first and the lowest `pos` on a tie. That is the same leftmost rule our loop gets from `strict <`. Our implementation states this in a comment and a test pins it, because a heap silently reversing a tie is exactly the @@ -298,29 +385,93 @@ emit, so it breaks the mirror on exactly the inputs it claims to protect. Truncation is worse, because it changes the output silently. Nobody should reach for this as a shortcut, which is why it is written here. -**Rejected on measurement: removing the per-probe `std::string` on its own.** -This was the low-risk option and it does not work. The same loop with one -reused key buffer, measured in the same harness on the same inputs, scored -31,076.04 ms on the 65,535-byte case against the current 28,746.70 ms, and -639.39 ms against 813.02 ms on 8,192 spaces. It is within noise, and slower on -the largest case. The allocation is not the cost. The rescan is. The separable -question therefore has a measured answer of no, and the two changes are one -change: interning removes the key because the algorithm no longer names one. +**Removing the per-probe `std::string` on its own is not a separate option, and +that is an argument about the design rather than a measurement.** Interning is +forced by the algorithm: upstream's heap entry carries `new_id` +(`word.rs:8-12`), its staleness test compares `new_id` (`word.rs:197-205`), and +its table is keyed on an identifier pair (`mod.rs:9`, `model.rs:19`). A merge +step that has to rebuild a `std::string` to name a pair cannot express that +test. So the key disappears as a consequence of mirroring `merge_all`, not as a +tuning step taken beside it. The rescan is the `O(n^2)` term and the key is a +constant on top of it. Removing a constant from a quadratic leaves a quadratic, +which is why nobody should reach for the key alone as the cheap half. + +**How much the key alone is worth has NOT been measured on an idle host, and +this spec makes no claim about it.** A first revision of this spec asserted that +a reused key buffer had been measured and was *slower*, and a same-binary A/B +did not reproduce that direction: the sign reversed on the largest case. The +original figures were taken at load average 130-190, which is where a reversed +sign comes from. The claim is withdrawn rather than restated with a new sign, +because neither reading has an idle-host A/B behind it and the design argument +above does not need one. What IS measured, in `## Prototype evidence`, is the +heap against the current loop, and that ratio is three orders of magnitude, not +a margin any allocation constant could account for. ## Prototype evidence -A prototype of the design, built against this tree's sources, produced symbol -output **bit-identical** to `BpeMerge` on all ten inputs it was run against: the -six benchmark prompts, 8,192 spaces, 8,192 `~`, 8,192 `a`, 2,731 `中`, and the -65,535-byte `的` case. It is a prototype, not the implementation. It validates -staleness by comparing the merged string, where the implementation must mirror -upstream's identifier comparison (`word.rs:198-205`), and it uses -`std::priority_queue` where upstream uses a 4-ary heap. Neither difference -changes the order, and both are named so the implementer does not inherit them -by accident. +A prototype of the design, built against this tree's sources by substituting one +`bpe.cpp` in the link line, produced output **bit-identical** to `BpeMerge` +everywhere it has been run. + +**Harness A**, symbol output only, against the `qwen3.8-27b-hf` merge table: +ten inputs: the six benchmark prompts, 8,192 spaces, 8,192 `~`, 8,192 `a`, +2,731 `中`, and the 65,535-byte `的` case. **Every one of those is byte-level.** +None of them exercises `EncodePlainSp`, which is the arm this row exists for, +so on its own that set does not cover the SentencePiece path at all. + +**Harness B** closes exactly that gap, comparing token identifiers out of full +`Tokenizer::Encode` through **both** committed goldens, the SentencePiece +`tokenizer_mistral` and the byte-level `tokenizer_qwen36`, over ten inputs each: +2,000 and 8,000 bytes of English prose, 4,096 bytes each of spaces, `a`, `~` and +newlines, 2,001 bytes of `的`, 3,000 bytes each of Chinese and Thai prose, and +4,000 bytes of mixed script with punctuation, digits and whitespace. All 20 +comparisons are identical, and **all 20 also match HF `tokenizers` 0.22.2 +reading the same two files**, which is what makes them an oracle result rather +than a self-consistency check. + +It is a prototype, not the implementation. Three differences are named so the +implementer does not inherit them by accident: + +- It validates staleness by comparing the merged **string**, where upstream + compares `new_id` (`word.rs:197-205`) and the implementation must too. On a + merge table with two distinct pairs sharing one `new_id`, the two tests are + not the same test. +- It uses `std::priority_queue` where upstream uses a 4-ary heap. The order is + the same because the comparator is the same. The constant is not. +- It still builds a `MergeKey` string per candidate and a merged string per + heap entry, because it reuses the string-keyed `MergeRanks`. The + implementation interns, so its constant is lower and its growth ratio is + closer to `n log n` than the prototype's. **A bound derived from the + prototype's ratio is therefore not a bound for the implementation**, and + `## Tests to port` item 3 requires the implementing row to re-derive it on + its own code. ## Defence in depth +**Nothing else in the stack bounds this, and that is checked rather than +assumed.** Two facts, both read at this base: + +- **There is no authentication anywhere in `src/vllm/entrypoints/`.** A + case-insensitive grep for `api_key`, `api-key`, `bearer`, `authorization` and + `authenticat` over all 143 files of `src/vllm/entrypoints/` and + `include/vllm/entrypoints/` returns nothing, exit status 1. So there is no + credential between an unauthenticated caller and `/tokenize`, which needs no + engine and no model. +- **The only size bound in the stack is httplib's default**, + `CPPHTTPLIB_PAYLOAD_MAX_LENGTH` at `third_party/httplib/httplib.h:129-130`, + which is `100 * 1024 * 1024`. Nothing in `src/`, `include/`, `cmake/` or + `CMakeLists.txt` overrides it and nothing calls `set_payload_max_length`, so + 100 MB is what a request body may be. + +Extrapolate the measured `n^2.01` fit, taking `t = c n^2` with `c` from the +25,345.61 ms at 65,536 bytes. A 100 MB body of a single character class is then +on the order of `6e7` s, several hundred CPU-days of one core, for one request. +That extrapolation spans three decades beyond the largest measured point, so +read it as an order of magnitude and not as a measurement. What is measured is +that 64 KB already costs 25 s. Either way the conclusion does not depend on the +constant: the cost is unbounded in the request body, unauthenticated, and paid +on the HTTP worker before any check. + A length guard at the API boundary is worth having **after** the algorithmic fix, not instead of it, and it is not in this row's scope. Two constraints bind it if a later row adds one: @@ -333,19 +484,42 @@ it if a later row adds one: wrong place, because it needs the token count that the expensive step produces. -After the fix, 64 KB encodes in single-digit milliseconds, so the guard protects -against a future regression rather than against today's cost. File it as its own +After the fix, 64 KB encodes in tens of milliseconds. The prototype takes +15.485 ms on the English case and 7.40 ms to 28.07 ms on the single-class cases. +The guard therefore protects against a future regression rather than against +today's cost. File it as its own issue when the row lands. ## Dependencies -None. The row is self-contained host code on base `31f93787c`. It needs no -oracle run, no lease, no checkpoint mount, and no other row. The two goldens its -gates read are already committed. It does not block on +**Code:** none. The row is self-contained host code. It needs no oracle run, no +lease, no checkpoint mount, and no other row's code. The two goldens its gates +read are already committed. It does not block on [#915](https://github.com/mudler/vllm.cpp/issues/915) and #915 does not block on it, although closing this row removes the outlier that [`qwen38-27b-bf16-gate.md`](qwen38-27b-bf16-gate.md) records. +**Record:** none, and this row appends no +[`issue-index.md`](../issue-index.md) row of its own. + +#1365 already has its row. `9e1a5e573` +([PR #1369](https://github.com/mudler/vllm.cpp/pull/1369)) appended it at +13:05:11Z on 19 August 2026, and it was correctly anchored when it landed. A second row for the same issue number is +what `scripts/check-agent-record.py` reports as `issue #1365 listed twice`, +because under `merge=union` a duplicate is exactly what two branches appending +one issue look like. So no row is appended here, and that is the finished state +rather than a deferral. + +**The landed row's text is stale, and it is not repaired by editing it.** #1365 +was renamed and re-scoped in place at 17:05:51Z on 19 August 2026, from the +symptom, a reproducible ~4 s TTFT outlier on request 3 of every leg, onto the +cause this spec addresses. The index row was written before that and still reads +as the original symptom report. The issue now legitimately covers both, and its +body keeps the original observation verbatim under +`## Original report, kept verbatim`. The index is append-only, so the row stays +as it landed and this paragraph is where the reader is told that the row +describes the symptom while the issue describes the cause. + ## Work breakdown Non-overlapping. W1 is the whole correctness argument and can be reviewed @@ -360,15 +534,31 @@ before any algorithm changes. W2 before W3 is deliberate. Interning and the heap are separable in the tree even though they are one design, and splitting them means a reviewer never has -to hold a representation change and an algorithm change in mind at once. +to hold a representation change and an algorithm change in mind at once. It is +also the order the `new_id` staleness test needs: that comparison cannot be +written against a string-keyed table. + +**These four waves are four commits in ONE pull request, and the split is a +review aid rather than a landing plan.** W1 lands items 2 and 3 red against +unchanged code, so between W1 and W3 the gate is failing by construction. A +separate W1 pull request would put that red on `main` and leave every other +row's gate unable to tell its own failure from this one. AGENTS.md's default +with no recorded preference is one pull request, none of its three split cases +applies here, because no helper needs a base-reachable spec, the scope is one +function, and W1 through W3 all write product code. The commit order still +proves the spec came first. If the developer prefers a split, the reds must land +`SKIP`-ped with the reason and the owning wave named, and W3 must remove the +skip in the same change that turns them green. A permanently skipped assertion +is not an acceptable resting state. ## Risks | Risk | Why it is real | Control | |---|---|---| | A tie is resolved differently and one token identifier changes | The heap's comparator is where leftmost-wins lives, and a reversed tie is invisible on most text | Port `Ord for Merge` (`word.rs:28-35`) with its inversion; a direct tie case in `test_bpe.cpp`, which already pins `BpeSplit("ĠĠĠ", r2) == {"ĠĠ", "Ġ"}` | -| A stale heap entry is applied | Positions and neighbours change under the heap | Mirror all three validations (`word.rs:187`, `:191`, `:198-205`); a red-first case that reaches a stale entry | +| A stale heap entry is applied | Positions and neighbours change under the heap | Mirror all three validations (`word.rs:187`, `:191`, `:197-205`), the third of them comparing `new_id` rather than the pair; a red-first case that reaches a stale entry | | Interning refuses a checkpoint that loads today | The load-time vocabulary rule is new for us | Verify every committed golden still loads; record which ones in `## Outcome` | +| A llama.cpp-converted GGUF is refused where `tokenizer.json` was not | `src/vllm/tokenizer/tokenizer.cpp::FromGguf` builds the same `merge_ranks_` at `:864`, and `tokenizer.ggml.merges` is a legacy `"left right"` string list produced by a CONVERTER rather than copied from the original checkpoint. A converter that drops or renames a vocabulary entry names a token the vocabulary does not carry, and after W2 that is a load-time refusal on a file that loads today. Neither we nor upstream have a GGUF case for this: HF `tokenizers` never reads GGUF, so upstream cannot be the oracle for the GGUF arm | The refusal message names the missing token, so the failure is actionable at load rather than per request. `tests/vllm/test_bpe.cpp` already loads a GGUF through `FromGguf`; the implementing row adds one case that a GGUF merge naming an absent token is refused at load with the name in the message, and one that the existing well-formed GGUF still loads. If a real converted checkpoint is refused, that is `NEEDS_DECISION` under `## Stop conditions`, not a quiet widening | | The `kUnk` sentinel collides with a real identifier | The SentencePiece path merges a value that is not a vocabulary entry | Reserve an identifier outside the vocabulary range and assert it | | The equivalence corpus is too small to be evidence | The existing 64-entry corpus missed this defect at 54 bytes | The corpus below is required to contain the long-pretoken regime | @@ -382,17 +572,59 @@ prove after. identifiers. The corpus is the existing 64 entries, plus the six #1365 prompts, plus real multilingual prose in Chinese, Japanese, Thai, Lao and Khmer, plus the single-class runs above at a size the suite can afford. This - test **passes on today's code** and must keep passing. It is the correctness - gate, and its value is that it cannot go green on a faster wrong answer. -2. **A cost bound that fails today.** Encode one pretoken of a stated size and - assert an upper bound on wall time. This is red on `31f93787c` by three - orders of magnitude, which is the margin that makes a wall-clock assertion - honest on a contended runner. State the bound as an absolute figure with the - margin argued, not as a ratio against a same-run baseline. -3. **A growth-shape assertion.** Encode at n and at 2n and assert the ratio sits - below a bound that quadratic growth cannot satisfy and linearithmic growth - can. This is the assertion that survives a slow runner, because both halves - move together. It is red today at a ratio near 4. + test **passes on today's code** and must keep passing. + **The recorded identifiers come from HF `tokenizers` 0.22.2, not from our own + output.** A baseline captured from the code under test is a change detector: + it can only prove that the answer did not move, never that the answer is + right, so recording our own output would make the sentence "it cannot go + green on a faster wrong answer" false. The capture is cheap and has already + been done for the 20 comparisons of `## Prototype evidence`: `Tokenizer.from_file` + on the same committed `tokenizer.json` file the C++ side reads, then + `encode(text, add_special_tokens=False)` against `Tokenizer::Encode` (the + `add_special_tokens=True` arm pairs with `EncodeWithSpecialTokens`), comparing + `Encoding.ids` element by element. The generator script, the `tokenizers` + version, and the golden file's sha256 are committed beside the recorded + identifiers so the capture is reproducible. Where an entry cannot be captured + from HF, the test records that entry as self-referential in its own comment + rather than letting the whole corpus inherit an oracle it does not have. +2. **A cost bound that fails today, and it is the ROBUST of the two.** Encode + one pretoken of a stated size and assert an upper bound on wall time. State + the bound as an absolute figure with the margin argued, not as a ratio + against a same-run baseline. At 65,536 bytes of English through the Mistral + golden the two sides are 25,345.61 ms and, for the prototype, 15.485 ms, so a + bound placed anywhere in the middle of that has **three orders of magnitude** + of headroom. A runner would have to be a thousand times slower than this box + to cross it. That margin, not the fact that it is a single measurement, is + what makes this assertion survive a contended runner. +3. **A growth-shape assertion, which is the TIGHTER of the two and needs care.** + Encode at n and at 4n and assert the ratio sits below a bound that quadratic + growth cannot satisfy and the implemented algorithm can. **Use a step of at + least 4x, and take the minimum of k repetitions rather than one shot.** Both + requirements were measured, because a first revision of this spec specified a + 2x single-shot ratio and called it "the assertion that survives a slow + runner", which is backwards: + + - **A 2x single shot has about 1.6x of usable band.** Eight independent + single-shot pairs of the prototype at 131,072 and 262,144 bytes, load + average 20 to 30, gave ratios 1.862, 2.291, 2.303, 2.219, 2.508, 2.331, + 2.395 + and 2.336, a 35% spread over the minimum. Linearithmic predicts ~2.1 and + quadratic 4.0, so the band a bound may sit in is (2.51, 4.0). The two + halves are separate, independently preemptible measurement windows. They do + **not** move together, and one deschedule in the 2n half crosses 4.0. + - **A 4x step with min-of-8 has about 2.7x, and is four times more stable.** + Four independent min-of-8 sweeps at 65,536 and 262,144 bytes on the same + box gave 5.51, 5.74, 6.01 and 6.03, a 9% spread. Quadratic predicts 16, + so the band is (6.03, 16). + + The red side is measured on the current code at the same shape: 8,000 to + 32,000 bytes costs 16.74x, against the 16x `n^2` predicts. The prototype's + 5.5 to 6.0 is above the ~4.6 that `n log n` predicts because the prototype + still builds a string key and a merged string per candidate. The interned + implementation removes both, so **the implementing row re-derives the bound + from min-of-k sweeps on its own code** and records the sweep, the `k`, and + the host load beside it. A bound copied from the prototype is not a bound for + the implementation. 4. **The leftmost tie.** Extend the existing `test_bpe.cpp` case into one that distinguishes the two orders on a longer symbol list. 5. **A stale-entry case.** A merge sequence where a queued candidate is @@ -402,8 +634,17 @@ prove after. `tests/parity/goldens/tokenizer_mistral/tokenizer.json`, whose `"split": false` makes the whole prompt one word. Ordinary English prose is the input, because that is what fails. -7. **The load-time vocabulary refusal.** A merge table naming a token absent - from the vocabulary is refused at load, with the missing name in the message. +7. **The load-time vocabulary refusal, on BOTH load surfaces.** A merge table + naming a token absent from the vocabulary is refused at load, with the + missing name in the message, once through + `src/vllm/tokenizer/tokenizer.cpp::FromHfJson` and once through + `src/vllm/tokenizer/tokenizer.cpp::FromGguf`, whose + `tokenizer.ggml.merges` reaches the same `InsertMerge` at + `src/vllm/tokenizer/tokenizer.cpp:864`. `tests/vllm/test_bpe.cpp` already + builds a zero-tensor GGUF and loads it through `FromGguf`, so the GGUF case + is one more kv block in an existing fixture, not new machinery. Also assert + that the existing well-formed GGUF still loads, because that is the + regression the refusal can cause. The reachability mutation for the fresh reviewer: delete the call to the new merge from `src/vllm/tokenizer/tokenizer.cpp::EncodePlain` in a scratch copy and @@ -428,6 +669,15 @@ The CPU tier can prove, with no other host: - Through `test_tokenizer_parity`, `test_tokenizer_parity_mistral`, `test_tokenizer_parity_deepseek` and `test_tokenizer_parity_gpt4o`, that all four already-gated families still produce their recorded identifiers. +- Through `test_bpe`, `test_tokenizer_metaspace_split` and `test_detokenizer`, + that the units this change actually edits still hold. These three are the + suites nearest the diff and the first revision of this spec omitted all of + them. `test_bpe` owns `BpeSplit`, `BpeMerge`, `MergeKey` and the `FromGguf` + load path. `test_tokenizer_metaspace_split` owns the `metaspace_split_` flag + that decides whether the SentencePiece arm makes the whole prompt one word. + `test_detokenizer` is the round trip that a changed symbol boundary would + break. All three are declared in `tests/CMakeLists.txt` and run in the CPU + tier today. What the CPU tier cannot prove: the end-to-end TTFT of #1365's legs on GB10. That needs the bf16 27B server on `dgx:gpu0` under an `rc` lease, and it is the @@ -435,12 +685,21 @@ row's closing evidence rather than its correctness gate. Run it only after the CPU gates are green. Required before any speed figure in this spec is accepted: an idle-host, -same-binary A/B re-measure with the load average recorded, because every figure -above was taken between load average 130 and 190. +same-binary A/B re-measure with the load average recorded. No figure above was +taken on an idle host. Harness A ran at load average 130 to 190 and Harness B at +4 to 90, on a 20-core box with another session compiling and testing throughout. +Harness B reports the minimum of a repetition set, which is the least contended +estimate the box could give, and that is not the same thing as an idle +measurement. The two harnesses agree on the shape and on every absolute figure +to within about 25%, which is what makes the shape safe to reason from and the +constants not yet quotable. ## Owed -Nothing yet. This row files no issue it does not fix. +Nothing. This row files no issue it does not fix, and it owes no +[`issue-index.md`](../issue-index.md) row: #1365's row already exists and a +second one for the same number is a checker failure, argued under +`## Dependencies`. ## Stop conditions diff --git a/scripts/check-agent-record.py b/scripts/check-agent-record.py index feb388335..e2e9d2aa4 100644 --- a/scripts/check-agent-record.py +++ b/scripts/check-agent-record.py @@ -620,10 +620,10 @@ # thing a token gate provably cannot see. It is also not a benchmark row: the encode # runs synchronously on the HTTP worker five lines before the only length check, so # `max_model_len` bounds none of it and `/tokenize` reaches it with no engine. -# MEASURED at `31f93787c`: 65,535 bytes of one repeated character costs 28.1-48.1 s -# of one core, and 64 KB of ordinary English prose costs 24.1 s through the committed -# Mistral golden, growing n^2.0. `READY`, spec `specs/bpe-quadratic-merge.md`, -# issue #1365. +# MEASURED: 64 KB of ordinary English prose costs 25.3 s through the committed +# Mistral golden against HF `tokenizers` 0.22.2's 10.1 ms on the same file for +# byte-identical identifiers, and the fit over 1 KB to 64 KB has exponent 2.01. +# `READY`, spec `specs/bpe-quadratic-merge.md`, issue #1365. # Bumped for a real new row, never to make a failing state transition pass. ENGINE_ROWS = 168 diff --git a/tests/scripts/test_agent_record.py b/tests/scripts/test_agent_record.py index 1e03b00ec..417130cfc 100644 --- a/tests/scripts/test_agent_record.py +++ b/tests/scripts/test_agent_record.py @@ -1730,5 +1730,92 @@ def test_the_pin_agrees_with_the_matrix_it_counts(self) -> None: self.assertIn(self.ROW, {r.item_id for r in engine}) +class BpeQuadraticMergeRowIsCounted(unittest.TestCase): + """The ENGINE ratchet bump 167 -> 168 is backed by a real row (#1365). + + Same shape and the same reason as `HfModelDownloadRowIsCounted`, applied to + the pin this change moves. `ENGINE_ROWS` is re-pinned by hand, so a bump + with nothing behind it looks exactly like a bump for a row that landed. + `test_engine_row_ratchet_is_load_bearing` proves the pin BINDS by moving it, + which holds for any value and cannot say whether 168 is the right one. This + class says that, by tying the pin to the row the matrix carries. + + This class asserts nothing about `.agents/issue-index.md`, where the sibling + classes assert their issue number, and this change appends no row there. + #1365's row already landed in `9e1a5e573` and a second row for one issue + number is what `check-agent-record.py` reports as `issue #1365 listed + twice`. The row's TEXT is stale, because #1365 was re-scoped in place from + the symptom onto the cause after the row landed, so `assertIn("issues/1365)", + index)` would pass here against a row describing the symptom and would + measure nothing about this row's work. The staleness is recorded in the + spec's `## Dependencies` instead, where prose can say it. + """ + + ROW = "SPEC-BPE-QUADRATIC-MERGE" + + def test_the_row_exists_in_the_engine_matrix(self) -> None: + text = (ROOT / ".agents/engine-matrix.md").read_text(encoding="utf-8") + matching = [ + line for line in text.splitlines() if line.startswith(f"| `{self.ROW}` |") + ] + self.assertEqual(len(matching), 1, f"{self.ROW} must appear exactly once") + + def test_the_row_names_its_spec(self) -> None: + """A row whose spec lives only in the PR body is untraceable.""" + text = (ROOT / ".agents/engine-matrix.md").read_text(encoding="utf-8") + row = next(l for l in text.splitlines() if l.startswith(f"| `{self.ROW}` |")) + self.assertIn("bpe-quadratic-merge.md", row) + self.assertTrue( + (ROOT / ".agents/specs/bpe-quadratic-merge.md").is_file(), + "the spec the row cites must exist", + ) + + def test_the_engine_pin_is_load_bearing_for_this_row(self) -> None: + """MUTATION: with this row removed, the pinned count must disagree. + + Redirects only the ENGINE matrix at a mutated copy on disk, for the + reason `TenstorrentMistralRowIsCounted` records: patching `read_text` + globally would feed engine content to every matrix, and this case would + then pass on errors that have nothing to do with the removal. + """ + clean: list[str] = [] + agent_record.check_matrices(clean) + self.assertEqual([e for e in clean if "engine rows" in e], []) + + path = agent_record.ENGINE_MATRIX + text = path.read_text(encoding="utf-8") + without = "\n".join( + l for l in text.splitlines() if not l.startswith(f"| `{self.ROW}` |") + ) + self.assertNotEqual(without, text, "the row must be present to remove") + + with tempfile.TemporaryDirectory(dir=agent_record.ROOT) as tmp: + mutated = Path(tmp) / "engine-matrix.md" + mutated.write_text(without, encoding="utf-8") + paths = [mutated if q == path else q for q in agent_record.MATRIX_PATHS] + errors: list[str] = [] + with mock.patch.object(agent_record, "MATRIX_PATHS", paths), \ + mock.patch.object(agent_record, "ENGINE_MATRIX", mutated): + agent_record.check_matrices(errors) + self.assertTrue( + any("engine rows" in e for e in errors), + f"removing {self.ROW} must break the engine count; got {errors}", + ) + + def test_the_pin_agrees_with_the_matrix_it_counts(self) -> None: + """MUTATION TARGET: `ENGINE_ROWS` back at 167 must be an error. + + The pin and the matrix are two hand-maintained records of one number. + This asserts they agree at the value this change lands, so lowering the + constant to the previous 167 while the row is present reds here. + """ + errors: list[str] = [] + rows, _ = agent_record.check_matrices(errors) + self.assertEqual([e for e in errors if "engine rows" in e], []) + engine = [r for r in rows if r.path == agent_record.ENGINE_MATRIX] + self.assertEqual(len(engine), agent_record.ENGINE_ROWS) + self.assertIn(self.ROW, {r.item_id for r in engine}) + + if __name__ == "__main__": unittest.main() From 29a54071a998b38b981f75b02417c559b119f0e4 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 19 Aug 2026 22:17:34 +0000 Subject: [PATCH 3/4] spec(SPEC-BPE-QUADRATIC-MERGE): the growth gate's own stability figures did not reproduce either (#1365) The previous revision repaired the growth assertion by replacing a 2x single-shot ratio with a 4x min-of-8 sweep, and certified that sweep as having a 9% spread and a (6.03, 16) band. Those figures were themselves taken at a lighter and unrecorded load, and a re-derivation at load 50 to 57 does not reproduce them: eight min-of-8 sweeps at 65,536 and 262,144 bytes spread 3.2x, from 2.800 to 8.920, against the certified 5.51 to 6.03. The direction of the earlier repair survives and is now better supported. A 2x step is unusable, because its observed maximum of 4.414 is already above the 4.0 that quadratic growth predicts, so the assertion would fire on the correct algorithm before it fired on the defect. A 4x step still separates, with the observed maximum 8.920 against a predicted 16. What does not survive is the claim that min-of-k makes the assertion stable. The red side re-derived on the current code reads 12.680 to 16.468, so the window between the prototype's slowest ratio and the current code's fastest is 1.42x. That is the whole budget the assertion has on a loaded host, and the spec now says the stability is UNESTABLISHED rather than certifying a number that moved the moment it was measured again. Three records disagreed about the absolute constants, because the engine-matrix row still carried first-revision figures the spec had already superseded. The row said 28.1-48.1 s and 24.1 s where the spec's re-derivation says 24.4-45.8 s and 25.3 s. A confirmation run at load 23 to 51 read 37.6 s for the input the record has at 24.4 s, a 54% move on load alone, so the row, the spec and the checker comment now carry the same figures and all three name them as contended minima rather than constants. The issue's title quoted the most contended harness of the three as fact and now states the shape instead. The record edits ride here because this change is what made them stale. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [Claude Code] --- .agents/engine-matrix.md | 4 +- .agents/specs/bpe-quadratic-merge.md | 91 +++++++++++++++++++--------- scripts/check-agent-record.py | 3 + 3 files changed, 66 insertions(+), 32 deletions(-) diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index c2051c008..cd5aa97d4 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -224,7 +224,7 @@ claims it. | `ENG-RELEASE-WINDOWS` | Native Windows x86_64 pre-alpha release extension: one adaptive MSVC/UCRT CPU bundle with AVX2 executed in CI and one Vulkan preview bundle, both deterministic ZIPs and authenticated by the existing release handoff | T0 | vLLM has no Windows release path; runtime behavior remains pinned to vLLM `555967922`. Platform substrate reference: llama.cpp `src/llama-mmap.cpp:520-590` @ `237ad9b961f009ae19ac29dbce4cd0c1251f94b3`; Win32 API is the OS authority | W14 Win32 portability/MSVC CPU, W15 deterministic ZIP/PE packaging + Vulkan, and W16 ten-tuple prerelease workflow/version/docs implemented for one PR | Linux portability/release mutation gates are local evidence only. Native `windows-2022` MSVC `/W4 /WX`, extracted runtime/ISA smokes, merged-SHA ten-tuple dry run, `v0.0.3-pre.1` publication, attestations, and exact 32-asset audit remain pending; no Windows ZIP exists yet | [windows-binary-release.md](specs/windows-binary-release.md); [#117](https://github.com/mudler/vllm.cpp/issues/117) | `ACTIVE` | `CLAIM-ENG-RELEASE-WINDOWS` | | `ENG-RELEASE-CONTAINERS` | Published OCI container images on GHCR, built by GitHub Actions: the same staged server bundle as `ENG-RELEASE-BINARIES`, shipped from one package `ghcr.io/mudler/vllm.cpp` with the lane in the tag — `:-cuda` / `-vulkan` / `-cpu`, the moving `:latest-cuda` / `:latest-vulkan` / `:latest-cpu`, and a bare `:latest` aliasing the cpu lane, with `ENTRYPOINT vllm-server`. Lanes `cuda` (one fat image covering every supported SM), `vulkan`, `cpu` (adaptive baseline); `rocm` blocked-preview, tracking its binary channel. Version tags are immutable; every `latest-` moves. Each lane is a `linux/amd64` + `linux/arm64` multi-arch manifest built on native runners — aarch64 is first-class here because GB10 (sm_121a), Thor (sm_110) and Orin (sm_87) are all arm64. The image contains the bundle and nothing else: no weights, no Python, no PyTorch, no compiler, no build tree. BOUNDARY: the GPU driver and container runtime stay on the host and are never bundled; Metal and MLX are NOT-CONTAINERIZABLE (no macOS container runtime and no Metal passthrough exists) and remain static-binary-only lanes, recorded as a permanent boundary rather than pending work. No image, workflow, registry package or pull is claimed to exist. | T0 | release image lanes `.buildkite/release-pipeline.yaml:34-170` and the published-image dependency boundary `docker/Dockerfile.cpu:262-290` @ `555967922` | `docker/Dockerfile` (cpu/vulkan/cuda targets calling the release scripts); `docker/healthcheck.sh`; `release/container-matrix.json`; `scripts/check-container-matrix.py`; `scripts/check-container-workflow.py`; `scripts/validate-container-image.py`; `scripts/container_tags.py`; `.github/workflows/containers.yml`; SIGTERM handler `src/vllm/entrypoints/openai/server_main.cpp` (`SignalShutdown`, all three `listen()` sites); the pre-existing `docker/Dockerfile.arm64` is an unrelated CPU bench cross-check | issues `#170`, `#312`, `#394`; `tests/scripts/test_check_container_matrix.py` 31/31; `test_check_container_workflow.py` 29/29; `test_check_cuda_fat_gencode.py` 7+4 subtests. **GB10 2026-08-11 (`promaxgb10-4ad8`, `sm_121a`, CUDA 13.3): arm64 cuda image 1.71 GB, 673/673 objects, ten-SM gencode audit PASS, and a REAL GPU boot -- `/health` 200, `/version` 200, in-container healthcheck, clean SIGTERM, `--gpus all`, host driver 580.159.03 injected.** cpu amd64 783 MB gated locally; cpu+vulkan amd64 green on hosted CI **arm64 cuda lane RUNTIME-VERIFIED on GB10 2026-08-11** -- the first accelerator-hardware evidence for any lane. Four defects were removed to get there, each found by building rather than reading: the CUDA 12.9 base could not compile `sm_110`, the BuildKit cache mount outlived its toolchain (both #366), Marlin gencode had drifted from the feature table and failed the audit on 14 correctly-compiled TUs (#394, blocking BOTH cuda tuples project-wide), and the validator could only ever produce build evidence because its boot smoke never passed `--gpus`. **NOT established: nothing is published to GHCR; amd64 cuda is unbuilt; the published arm64 image is SBSA (`targets/sbsa-linux`), so Tegra -- Thor `sm_110`, Orin `sm_87` -- is untested and NOT covered** **ORIN (Tegra) 2026-08-11: the SBSA image RUNS on Jetson AGX Orin `sm_87` (L4T R36.4.3, Docker 27.5.1) -- Qwen3-0.6B (rev `c1899de2`) loads and GENERATES via `/v1/completions`, tegrastats GR3D 95-97% during decode vs 14-15% idle.** Tegra needs `--runtime nvidia --gpus all`: `--gpus` alone is refused by the hook and `--runtime` alone mounts no driver | [container-images.md](specs/container-images.md); issues [#170](https://github.com/mudler/vllm.cpp/issues/170), [#312](https://github.com/mudler/vllm.cpp/issues/312), [#394](https://github.com/mudler/vllm.cpp/issues/394) | `ACTIVE` | `CLAIM-ENG-RELEASE-CONTAINERS-W1-W7` | | `ENG-DOCS-SITE` | Publish the 11 `docs/*.md` as a browsable GitHub Pages site at `https://mudler.github.io/vllm.cpp/` WITHOUT a second copy of the prose. A Hugo site at `website/` mounts `../docs` READ-ONLY and derives everything else from what is already in the files: each page title from the file's first `# H1`, the sidebar order from `website/data/nav.yaml`, and links through a Goldmark render hook (internal `.md` → site URL; the 139 `../.agents/**` and `../AGENTS.md` escapes → GitHub blob URLs, since the protocol tree is deliberately NOT published). **No file under `docs/` is modified, moved, renamed, or given front matter**, so `check-doc-checkpoint.py` and every protocol path reference keep working and there is no second surface that can drift — the whole point of the row. Custom lean layouts, NO theme and NO submodule: off-the-shelf docs themes read titles, weights and menus out of front matter this design deliberately does not have, so each would need its title partial, menu and link hook overridden anyway, and hugo-book additionally floors at Hugo 0.158 against the 0.146.3 pin CI and the local toolchain share. Hard prerequisite inside the repo: `classify_path` in `scripts/check-pr-size.py` FAILS CLOSED on `website/**` (verified: raises `ValueError: unclassified repository path`), so the classifier must learn the path or the PR cannot pass the project's own size gate. Hard prerequisite outside it: GitHub Pages must be enabled with the source set to GitHub Actions — the workflow is inert otherwise. A marketing landing page is explicitly OUT of scope (`README.md` stays the front door), as is any restructuring of `docs/`; the custom domain is parked behind the pending vLLM trademark question | T1 | NO vLLM analogue — upstream's docs are a separate mkdocs site and nothing in this row mirrors upstream *behavior*, so it carries no parity obligation. The STRUCTURAL reference is LocalAI's `.github/workflows/gh-pages.yml` (two Hugo sites merged into one Pages artifact), reduced to the docs half | read-only mount `website/hugo.toml:29`; title-from-H1 `website/layouts/partials/title.html:10`; link rewriting `website/layouts/_default/_markup/render-link.html:27`; guard `scripts/check-site.py:70`; deploy `.github/workflows/gh-pages.yml` | `tests/scripts/test_check_site.py:51,56,66,80,89,97` (6 mutation cases: clean tree, H1 stripped, doc absent from nav, nav entry with no file, duplicated entry, missing nav file); build evidence 14 pages with `docs/bench-evidence` + `docs/superpowers` absent from `public/` and no `href` ending in `.md`; 48 protocol links rewritten in `docs/status/`. NO published page is claimed: GitHub Pages is not yet enabled on the repository, which is the recorded stop condition holding this row at `GATING` | [gh-pages-docs-site.md](specs/gh-pages-docs-site.md); issue [#224](https://github.com/mudler/vllm.cpp/issues/224) | `READY` | `CLAIM-ENG-DOCS-SITE` | -| `ENG-RECORD-ANCHOR-RATCHET` | **The record's `path:line` citations were range-checked and never reported.** `check-agent-record.py` parsed BOTH forms: markdown links, and bare `` `file.cpp:123` `` through `RAW_LOCAL_ANCHOR_RE` since `ee511ca8a`. On a missing file or an out-of-range line `local_line_anchors` runs `continue`, so the bad anchor never reaches the caller, and `is_code_anchor` then answers with **any**, so one good sibling covers the rest. There was no symbol test and no report, and **32 of the 38** offenders are IN RANGE, so range-checking could not have found them. Measured at `8daa67b39`: **832 of 867** in-scope citations (**96.0%**) were already parsed and range-checked, and the **35** new to parsing sit under `.agents/`, `docs/` and `website/`; `EVIDENCED_STATES` omits `ACTIVE`/`READY` entirely and is deliberately NOT widened, because requiring an anchor there raises 85 errors across 53 rows. Even the fraction it saw was only range-checked, never checked to CONTAIN the symbol named beside it — every stale anchor found in the 2026-08-13/14 campaign was in range. LANDED as a device-leakage-shaped ratchet over a recorded baseline, never a bulk cleanup: the backlog is fixed by whoever next touches each row | T1 | none — this is our own record surface; the discipline mirrors AGENTS.md §Records ("cite the `file:line` you ported from") | parser + classifier + ratchet in `check-agent-record.py`: `BARE_CITATION_RE` `scripts/check-agent-record.py:1251` (the bare form), `cell_citations` `scripts/check-agent-record.py:1305` (both forms, with the adjacent-symbol rule), `classify_citation` `scripts/check-agent-record.py:1362` (OK / STALE / BROKEN), `RECORD_ANCHOR_STATES` `scripts/check-agent-record.py:1249` (gap 3: `ACTIVE` and `READY` join the count), `check_record_anchors` `scripts/check-agent-record.py:1485` (the two-way gate); budget in `scripts/record-anchor-baseline.json` | `RecordAnchorRatchet` `tests/scripts/test_agent_record.py:1397` — 10 cases, RED-first, including `test_one_good_link_does_not_cover_a_rotted_bare_citation` `tests/scripts/test_agent_record.py:1465`, the `any()` shape the rot hid in. Five mutants red it: report-only, `EVIDENCED_STATES` restored, links-only, first-citation-only, range-only. Measured baseline **38** (32 STALE + 6 BROKEN); gate wired in `scripts/agent-preflight.sh` and the `agent-record` CI job (`--report`) | [record-anchor-ratchet.md](specs/record-anchor-ratchet.md) | `ACTIVE` | `CLAIM-ENG-RECORD-ANCHOR-RATCHET` | +| `ENG-RECORD-ANCHOR-RATCHET` | **The record's `path:line` citations were range-checked and never reported.** `check-agent-record.py` parsed BOTH forms: markdown links, and bare `` `file.cpp:123` `` through `RAW_LOCAL_ANCHOR_RE` since `ee511ca8a`. On a missing file or an out-of-range line `local_line_anchors` runs `continue`, so the bad anchor never reaches the caller, and `is_code_anchor` then answers with **any**, so one good sibling covers the rest. There was no symbol test and no report, and **32 of the 38** offenders are IN RANGE, so range-checking could not have found them. Measured at `8daa67b39`: **832 of 867** in-scope citations (**96.0%**) were already parsed and range-checked, and the **35** new to parsing sit under `.agents/`, `docs/` and `website/`; `EVIDENCED_STATES` omits `ACTIVE`/`READY` entirely and is deliberately NOT widened, because requiring an anchor there raises 85 errors across 53 rows. Even the fraction it saw was only range-checked, never checked to CONTAIN the symbol named beside it — every stale anchor found in the 2026-08-13/14 campaign was in range. LANDED as a device-leakage-shaped ratchet over a recorded baseline, never a bulk cleanup: the backlog is fixed by whoever next touches each row | T1 | none — this is our own record surface; the discipline mirrors AGENTS.md §Records ("cite the `file:line` you ported from") | parser + classifier + ratchet in `check-agent-record.py`: `BARE_CITATION_RE` `scripts/check-agent-record.py:1254` (the bare form), `cell_citations` `scripts/check-agent-record.py:1308` (both forms, with the adjacent-symbol rule), `classify_citation` `scripts/check-agent-record.py:1365` (OK / STALE / BROKEN), `RECORD_ANCHOR_STATES` `scripts/check-agent-record.py:1252` (gap 3: `ACTIVE` and `READY` join the count), `check_record_anchors` `scripts/check-agent-record.py:1488` (the two-way gate); budget in `scripts/record-anchor-baseline.json` | `RecordAnchorRatchet` `tests/scripts/test_agent_record.py:1397` — 10 cases, RED-first, including `test_one_good_link_does_not_cover_a_rotted_bare_citation` `tests/scripts/test_agent_record.py:1465`, the `any()` shape the rot hid in. Five mutants red it: report-only, `EVIDENCED_STATES` restored, links-only, first-citation-only, range-only. Measured baseline **38** (32 STALE + 6 BROKEN); gate wired in `scripts/agent-preflight.sh` and the `agent-record` CI job (`--report`) | [record-anchor-ratchet.md](specs/record-anchor-ratchet.md) | `ACTIVE` | `CLAIM-ENG-RECORD-ANCHOR-RATCHET` | | `ENG-RECORD-CONFLICT-SURFACES` | Retire the shared record surfaces that make concurrent PRs conflict by construction. MEASURED at `origin/main` `d928e2c3` with `git merge-tree --write-tree` over every open PR: **16 of 29 conflict (55%), and 13 of the 16 conflict in bookkeeping files ONLY**, with no product code involved — `.agents/coordination.md` in 8, `.agents/NOW.md` in 5, `.agents/roadmap_v1.md` in 4, `scripts/check-public-doc-tables.py` in 4, `docs/STATUS.md` in 4, and any `src/`/`tests/` path in just 3. Three defects, each of which GUARANTEES rather than risks a collision. (1) `.agents/NOW.md` is a fixed-size shared buffer at EXACTLY 6000/6000 chars (`check-now-current.py:31`), so adding a row requires evicting another and every PR is a read-modify-write of one global — and the conflict is the LUCKY outcome, since a clean three-way merge would apply both evictions and both additions, silently dropping live rows and blowing the very budget the checker defends. (2) `STATUS_RATCHET = {"chars": 243245}` (`check-public-doc-tables.py:557`) is a hardcoded byte count of a DIFFERENT file that may only fall, so a PR owing `docs/STATUS.md` one lifecycle line must delete unrelated prose from another row to pay for it and edit the checker too; the checker's own comment at `:331` already records the failure (*"a ratchet pinned to the byte turns every concurrently merged row's one-line status edit into a spurious failure"*) and answered it with slack instead of removing the coupling. (3) `.agents/coordination.md`'s active-claims table is insert-at-one-anchor: the six ROCm GDN PRs (#334 #336 #341 #343 #345 #348) are ONE author's sequential stack that conflicts on nothing else, each appending a ~1,500-char row — the PR description, transcribed into a file every other claim also writes. It also contradicts the protocol it serves: `AGENTS.md` holds that *"History is git"* and *"There is no state log"*, yet both claims tables ARE state logs duplicating `gh pr list`, `row/` branch names and issue state; the argument that refuses a waiver registry applies unchanged to a claims registry. Precedent twice over — `policy.csv` retired in `0f3e44ee`, per-class line budgets retired 2026-08-10 because the gate fired on ordinary work. The exonerated surfaces share ONE property, one writer per file: `.agents/specs/.md` (one file per row, **zero conflicts** in the sample), the `*-matrix.md` inventories, and the append-only `.agents/benchmark-record.md`. SCOPE: remove `STATUS_RATCHET` and the doc-gating global counters while KEEPING the per-cell/per-paragraph caps (local, so they couple nothing); remove the active-claims table and derive claims from open PRs and branch names; drop `NOW.md`'s byte budget; order the roadmap's keyed tables by ID so distinct keys stop colliding at one anchor; and record the invariant — **no surface that every PR must write** — in `AGENTS.md`. No product source, kernel or gate semantic moves | T0 | NO vLLM analogue — this is local protocol machinery, so the mirror rule does not apply and no upstream `file:line` exists to port from. Governed instead by `AGENTS.md` §"Changing the rules or a checker", which requires a spec, a red-before test or mutation, and green-after evidence | - | - (spec-before-code: the red-before suites are named in the spec's Tests section — `tests/scripts/test_check_public_doc_tables.py`, `tests/scripts/test_check_now_current.py`, a mutation case per removed rule proving the obligation survives in the retained caps and `check-doc-checkpoint.py`, and a `git merge-tree` merge-shape regression that must be RED before the `NOW.md`/roadmap work and GREEN after) | [retire-shared-record-surfaces.md](specs/retire-shared-record-surfaces.md); issue [#364](https://github.com/mudler/vllm.cpp/issues/364) | `READY` | `CLAIM-ENG-RECORD-CONFLICT-SURFACES` | | `ENG-TRAILER-MERGE-ARTIFACTS` | The trailer gate rejects CORRECT commits because of paragraph placement, and that is why `main` is red on `agent-record`. `check-commit-trailers.py` reads trailers via `git interpret-trailers --parse`, which treats ONLY the final paragraph as the block; GitHub appends `Co-authored-by:` as a SEPARATE trailing paragraph on a squash merge, so a complete correct block becomes invisible and the gate reports it missing. MEASURED: piping `git show -s --format=%B dbd0d51c` into `git interpret-trailers --parse` prints nothing but the co-author line, and 13 of the last 30 commits on `main` fail the check -- unnoticed only because those runs were cancelled (#274), which HID the defect rather than causing it. FIX: fuse consecutive trailing TRAILER-SHAPED paragraphs before parsing. Nothing is relaxed -- the block must still exist, the marker must still sit above it, each declaration must still appear exactly once, and an AI co-author is still forbidden; the block is merely FOUND where the merge tool left it. A prose paragraph still terminates it. REJECTED IN FLIGHT and recorded because it is the more instructive half: a first attempt also collapsed identical duplicate trailers to fix the multi-commit-squash shape, which relaxes the uniqueness rule an existing test already pins. Rewriting that assertion to suit the change is what AGENTS.md forbids, and the distinction is real -- a doubled block is genuinely malformed and fixable at source, whereas the co-author case is a correct commit defeated by the parser. Reverted in full. SCOPE LIMIT, stated rather than implied: this fixes ONE of five observed shapes. `f64f2b71` (bot co-author) is a REAL violation the parse had been hiding and now correctly fails; `87308dea` (GitHub's `---------` separator), `b8293c88` (squash doubled the block) and `b580452d` (merge button, no trailers) stay red by design. Closing those is a merge-method change, not a checker change | T0 | NO vLLM analogue -- local protocol machinery, so the mirror rule does not apply and there is no upstream `file:line` to port from. Governed by `AGENTS.md` §"Changing the rules or a checker" | `scripts/check-commit-trailers.py:60` (`join_trailing_trailer_paragraphs`, `_is_trailer_paragraph`, and the fused `parsed_trailers`) | `tests/scripts/test_check_commit_trailers.py:1` 21 cases -- the RED-BEFORE appended-co-author case plus four GUARDS that keep the fusion bounded (doubled block still fails, contradictory declarations still fail, a no-trailer merge message still fails, prose after the block still fails), all four green before and after; closure [parity-ledger.md#L941](parity-ledger.md#L941) | [trailer-merge-artifacts.md](specs/trailer-merge-artifacts.md); issue [#406](https://github.com/mudler/vllm.cpp/issues/406) | `DONE` | `157080c8` | | `ENG-FORGE-COAUTHOR` | The forbidden-AI-trailer rule was catching ATTRIBUTION rather than an authorship claim, which is why bot-opened PRs red `main` on merge. GitHub composes the squash message itself and appends the account that opened the PR — `Co-authored-by: localai-org-maint-bot <...@users.noreply.github.com>` — and most PRs here are opened by a bot, so nearly every squash trips the AI-identity check. Real instance `f64f2b71`, invisible until #406 repaired the parse, which is why it reads as a new failure and is not one. The rule exists so an AI cannot claim it WROTE the code, and that stays; GitHub is recording who pressed the button, and the AI-involvement claim is already carried separately by `AI-Assisted` and `Assisted-by` in the same block. FIX: accept a `Co-authored-by` at a GitHub account noreply address even when the name matches an AI identity token, keyed on the FORGE'S OWN DOMAIN rather than the name so the exemption cannot be borrowed. A hand-written `Co-authored-by: Claude ` still fails; `Signed-off-by` is excluded from the exemption entirely, because a sign-off is a legal assertion about provenance rather than attribution. `AGENTS.md` records the same distinction in the same change so prose and checker cannot drift | T0 | NO vLLM analogue -- local protocol machinery, so the mirror rule does not apply and there is no upstream `file:line` to port from. Governed by `AGENTS.md` §"Changing the rules or a checker" | `scripts/check-commit-trailers.py:38` (`FORGE_ACCOUNT_EMAIL` and the forbidden-trailer skip) | `tests/scripts/test_check_commit_trailers.py:1` 25 cases -- the RED-BEFORE forge-bot case plus THREE guards that matter more than the relaxation because this LOOSENS a rule: a hand-written AI co-author still fails, `Signed-off-by` at the same noreply address still fails, and a human co-author still passes; all three green before and after. Real commit `f64f2b71` re-verified per commit | [forge-coauthor-attribution.md](specs/forge-coauthor-attribution.md); issue [#418](https://github.com/mudler/vllm.cpp/issues/418) | `ACTIVE` | `CLAIM-ENG-FORGE-COAUTHOR` | @@ -277,7 +277,7 @@ claims it. | `LOAD-GGUF-MMPROJ` | A SECOND, `clip`-architecture GGUF projector file beside the language file, and the Qwen3-VL vision tower loaded out of it. The single-file assumption is structural, not incidental: `ModelSource` carries a VECTOR of safetensors shards and exactly ONE `GgufFile*`, and `EngineParams` has no projector field at all, so an mmproj has nowhere to arrive. `GgufFile::Open` does handle SHARDS (`DetectSplit` parses `-NNNNN-of-MMMMM.gguf`), but shards of one split are not a second, differently-architected file. Nothing in this tree loads a `clip` projector today and the two apparent counterexamples are neither: MiniMax-H3 reads `visual.*` out of the SAME encoder GGUF and only from a test, and MuseGlimmer's mmproj path is a REFUSAL whose only caller is a test — there is no production call site because no production path accepts a second GGUF path. First consumer is `Qwen3.8-27B` ([#821](https://github.com/mudler/vllm.cpp/issues/821)), whose `mmproj-BF16.gguf` (334 tensors, `clip.projector_type = qwen3vl_merger`, header-verified 2026-08-18: data end == file size 931,146,432) ships BOTH halves of the temporal patch embedding as `v.patch_embd.weight` + `v.patch_embd.weight.1` — precisely what MuseGlimmer's lacks, so the check that distinguishes them is this row's, not a reason to refuse. Second consumer is MuseGlimmer, whose refusal becomes REACHABLE production code the moment this lands | T0 | Pinned vLLM has no GGUF loader at all (`555967922`, `model_loader/__init__.py:33-49`), so the compatibility reference is llama.cpp `b10451` = `10bf611e5` (`PROJECTOR_TYPE_QWEN3VL`; the previously recorded `tools/mtmd/clip-impl.h:330` was read at the SUPERSEDED local fork `237ad9b96` and is owed re-anchoring, [#1003](https://github.com/mudler/vllm.cpp/issues/1003)) | the one open site [model_loader.cpp:1679](../src/vllm/entrypoints/model_loader.cpp#L1679), the single pointer [model_registry.h:98](../include/vllm/model_executor/models/model_registry.h#L98), the arch dispatch [model_loader.cpp:794](../src/vllm/entrypoints/model_loader.cpp#L794), the refusal precedent [muse_glimmer_gguf_weights.cpp:695](../src/vllm/model_executor/models/muse_glimmer_gguf_weights.cpp#L695) | none yet — this row is spec-only. The port target for the vision config mapping is [minimax_h3_vision_gguf.cpp:32](../src/vllm/model_executor/models/minimax_h3_vision_gguf.cpp#L32), which builds the same `Qwen3VLVisionConfig` from `visual.*` | [quantized arms of Qwen3.8-27B](specs/qwen38-27b-quant-arms.md) | `READY` | - | | `LOAD-HF-BPE` | HF tokenizer.json byte-level BPE and incremental detokenization | T0 | `vllm/tokenizers/registry.py:176`; `vllm/tokenizers/hf.py:163`; `tests/tokenizers_/test_hf.py:18`; `tests/tokenizers_/test_detokenize.py:148` | `src/vllm/tokenizer/tokenizer.cpp:240,484,512`; `src/vllm/v1/engine/detokenizer.cpp:409` | `tests/vllm/test_bpe.cpp:226`; `tests/vllm/test_tokenizer_parity.cpp:66,74,82,90`; `tests/vllm/test_pretokenizer.cpp:130` | `planned: specs/hf-tokenizer.md` | `ANCHOR-BACKFILL` | - | | `LOAD-SENTENCEPIECE` | SentencePiece (Metaspace + byte-fallback) BPE tokenizer family — the gate to Mistral/Gemma/SentencePiece `tokenizer.json`. `Tokenizer::FromHfJson` now dispatches on the pre_tokenizer FAMILY: a bare `Metaspace` node selects the SP path (space→▁ U+2581; `prepend_scheme` first/always/never with the starts-with-▁ guard; `split=false`; out-of-vocab char → `<0xNN>` byte-fallback tokens; merge-ranked BPE over the raw-UTF-8 string), else the BYTE-IDENTICAL byte-level path (`DetectPattern` still fails loudly on Metaspace, so families never overlap). Decode + the incremental detokenizer mirror HF's Sequence decoder (Replace ▁→space, ByteFallback, Fuse, Strip 1 leading space). **UNBLOCKS the Mistral (`MODEL-TEXT-mistral-mistral-for-causal-lm`) paged-engine SACRED gate.** `split=true` fails loudly (no golden in scope). Mirrors HF `tokenizers` 0.22 | T1 | `vllm/tokenizers/hf.py:163`; `vllm/tokenizers/mistral.py:235,467`; HF `tokenizers` 0.22 `pre_tokenizers/metaspace.rs`, `models/bpe/model.rs::merge_word`, `decoders/{replace,byte_fallback,fuse,strip}.rs` | dispatch `src/vllm/tokenizer/tokenizer.cpp:184` (`DetectMetaspace`), `:732` (`EncodePlainSp`), `:878` (`SpDecodeTokens`), `:927` (`Decode` SP branch); merge-loop factor `src/vllm/tokenizer/bpe.cpp:91` (`BpeMerge`); family/params/accessors `include/vllm/tokenizer/tokenizer.h:88,98`; incremental dispatch `src/vllm/v1/engine/detokenizer.cpp:357`; generator `tools/parity/dump_tokenizer_mistral.py` | `tests/vllm/test_tokenizer_parity_mistral.cpp:78` **6/6, 421 assertions byte-exact** vs HF `tokenizers` 0.22.2 (= vLLM 0.25.0 backend) over a 45-entry Metaspace/byte-fallback/special-token corpus (goldens `tests/parity/goldens/tokenizer_mistral/{tokenizer.json,encodings.json}`); SACRED cross-check vLLM `AutoTokenizer` (transformers 5.13.1) 0/45 mismatch (±BOS); byte-level suites byte-identical `tests/vllm/test_bpe.cpp` 852, `tests/vllm/test_detokenizer.cpp` 221, `tests/vllm/test_tokenizer_parity.cpp` 1175, `tests/vllm/test_tokenizer_parity_deepseek.cpp` 2461 | [specs/sentencepiece.md](specs/sentencepiece.md) | `ANCHOR-BACKFILL` | `CLAIM-LOAD-SENTENCEPIECE` | -| `SPEC-BPE-QUADRATIC-MERGE` | **The BPE merge loop is O(n^2) in pretoken length, on the request path, before the length check.** `src/vllm/tokenizer/bpe.cpp::BpeMerge` rescans every adjacent pair per merge and builds one `std::string` key per probe through `src/vllm/tokenizer/bpe.cpp::MergeKey`, on the premise its own comment states: "pretokens are tiny". FIVE of the seven pretokenizer rules return an unbounded run (`src/vllm/tokenizer/pretokenizer.cpp::MatchLetterRun` rule 2, `src/vllm/tokenizer/pretokenizer.cpp::MatchPunctRun` rule 4, `src/vllm/tokenizer/pretokenizer.cpp::MatchWsNewlines` rule 5, `src/vllm/tokenizer/pretokenizer.cpp::MatchWsNotBeforeNonSpace` rule 6, `src/vllm/tokenizer/pretokenizer.cpp::MatchWs` rule 7); only rule 3 `MatchNumbers` is capped and only rule 1 `MatchContraction` is bounded by its own alternation, and on the SentencePiece family `src/vllm/tokenizer/tokenizer.cpp::EncodePlainSp` merges the WHOLE prompt as one word because Mistral and Gemma declare `split: false`. MEASURED at `31f93787c`: 65,535 bytes of one repeated character costs 28.1-48.1 s of one core; 64 KB of ordinary English prose through the committed Mistral golden costs 24.1 s, growing n^2.0. `src/vllm/v1/engine/input_processor.cpp::ValidatePromptLen` runs FIVE LINES AFTER the encode, so `max_model_len` bounds none of it, and `/tokenize` reaches the same path with no engine. This is a remote denial of service, not a latency curiosity. Fix mirrors HF `tokenizers` 0.22.2 `Word::merge_all` (heap over candidate merges, identifier-keyed merge table); a prototype is bit-identical to today's identifiers on 20 comparisons across BOTH committed goldens, all 20 of which also match HF `tokenizers` 0.22.2 reading the same files, and 8,192 newlines fall from 580.44 ms to 2.60 ms. A pretoken cap is REFUSED: it changes the token identifiers. The string key is not a separable win either, on the DESIGN: upstream's staleness test compares `new_id` (`word.rs:197-205`) and its table is keyed on an identifier pair, so a step that rebuilds a `std::string` to name a pair cannot express the test; and removing a constant from a quadratic leaves a quadratic. How much the key alone is worth is NOT measured on an idle host and this row claims nothing about it | T0 | HF `tokenizers` 0.22.2 `tokenizers/src/models/bpe/word.rs:162-250` (`Word::merge_all`), `:28-35` (`Ord for Merge`), `tokenizers/src/models/bpe/mod.rs:9` + `model.rs:19,174-192` (identifier-keyed merge table and its load-time refusal) | `src/vllm/tokenizer/bpe.cpp::BpeMerge` (the defect); callers `src/vllm/tokenizer/tokenizer.cpp::EncodePlain` and `src/vllm/tokenizer/tokenizer.cpp::EncodePlainSp` | none yet — the corpus at `tests/parity/goldens/tokenizer_qwen36/` has a longest pretoken of 54 bytes, which is why every existing gate is green | [specs/bpe-quadratic-merge.md](specs/bpe-quadratic-merge.md) | `READY` | - | +| `SPEC-BPE-QUADRATIC-MERGE` | **The BPE merge loop is O(n^2) in pretoken length, on the request path, before the length check.** `src/vllm/tokenizer/bpe.cpp::BpeMerge` rescans every adjacent pair per merge and builds one `std::string` key per probe through `src/vllm/tokenizer/bpe.cpp::MergeKey`, on the premise its own comment states: "pretokens are tiny". FIVE of the seven pretokenizer rules return an unbounded run (`src/vllm/tokenizer/pretokenizer.cpp::MatchLetterRun` rule 2, `src/vllm/tokenizer/pretokenizer.cpp::MatchPunctRun` rule 4, `src/vllm/tokenizer/pretokenizer.cpp::MatchWsNewlines` rule 5, `src/vllm/tokenizer/pretokenizer.cpp::MatchWsNotBeforeNonSpace` rule 6, `src/vllm/tokenizer/pretokenizer.cpp::MatchWs` rule 7); only rule 3 `MatchNumbers` is capped and only rule 1 `MatchContraction` is bounded by its own alternation, and on the SentencePiece family `src/vllm/tokenizer/tokenizer.cpp::EncodePlainSp` merges the WHOLE prompt as one word because Mistral and Gemma declare `split: false`. MEASURED at `31f93787c`, min-of-k on a CONTENDED 20-core box at load average 25-90, so these are contended minima and not idle-host constants: 65,535 bytes of one repeated character costs 24.4-45.8 s of one core; 64 KB of ordinary English prose through the committed Mistral golden costs 25.3 s against HF `tokenizers` 0.22.2's 10.1 ms on the same file for byte-identical identifiers, growing n^2.0. A confirmation run at load 23-51 read 37.6 s where the recorded figure reads 24.4 s, so the SHAPE is the result and the constants move with the box. `src/vllm/v1/engine/input_processor.cpp::ValidatePromptLen` runs FIVE LINES AFTER the encode, so `max_model_len` bounds none of it, and `/tokenize` reaches the same path with no engine. This is a remote denial of service, not a latency curiosity. Fix mirrors HF `tokenizers` 0.22.2 `Word::merge_all` (heap over candidate merges, identifier-keyed merge table); a prototype is bit-identical to today's identifiers on 20 comparisons across BOTH committed goldens, all 20 of which also match HF `tokenizers` 0.22.2 reading the same files, and 8,192 newlines fall from 580.44 ms to 2.60 ms. A pretoken cap is REFUSED: it changes the token identifiers. The string key is not a separable win either, on the DESIGN: upstream's staleness test compares `new_id` (`word.rs:197-205`) and its table is keyed on an identifier pair, so a step that rebuilds a `std::string` to name a pair cannot express the test; and removing a constant from a quadratic leaves a quadratic. How much the key alone is worth is NOT measured on an idle host and this row claims nothing about it | T0 | HF `tokenizers` 0.22.2 `tokenizers/src/models/bpe/word.rs:162-250` (`Word::merge_all`), `:28-35` (`Ord for Merge`), `tokenizers/src/models/bpe/mod.rs:9` + `model.rs:19,174-192` (identifier-keyed merge table and its load-time refusal) | `src/vllm/tokenizer/bpe.cpp::BpeMerge` (the defect); callers `src/vllm/tokenizer/tokenizer.cpp::EncodePlain` and `src/vllm/tokenizer/tokenizer.cpp::EncodePlainSp` | none yet — the corpus at `tests/parity/goldens/tokenizer_qwen36/` has a longest pretoken of 54 bytes, which is why every existing gate is green | [specs/bpe-quadratic-merge.md](specs/bpe-quadratic-merge.md) | `READY` | - | | `LOAD-CONFIG-SURFACE` | Dataclass-for-dataclass config and serve-compatible flags | T0/T1 | `vllm/config/scheduler.py:26`; `vllm/config/cache.py:25`; `vllm/config/compilation.py:378` | `include/vllm/config/scheduler.h:67`; `src/vllm/config/scheduler.cpp:11`; `src/vllm/transformers_utils/hf_config.cpp:83`; limited flags incl. `max_num_seqs`/`max_num_batched_tokens` `examples/server/main.cpp:63,116,170` | `tests/vllm/test_scheduler_config.cpp:10`; `tests/vllm/test_hf_config.cpp:131,224,265`; `examples/CMakeLists.txt:34` | `planned: specs/config-surface.md` | `PARTIAL` | - | | `ENG-HF-MODEL-DOWNLOAD` | Fetch a checkpoint from HuggingFace so `--model` accepts a repository identifier and not a local path only. Two forms behind one flag, with the local path probed first: `org/repo` mirrors vLLM's full-snapshot download, and `org/repo:Q4_K_M` fetches one GGUF file, a form vLLM does not implement and which llama.cpp supplies as the secondary oracle. Mirrors vLLM's two-phase fetch (config JSON first, so a bad repository fails after 200 KB and not 60 GB) and its index-driven file selection, which reads `model.safetensors.index.json` and fetches the exact names in `weight_map` instead of every match for `*.safetensors`. Reference is resolved to a commit before any byte is fetched, so a moving `main` cannot change what a second run loads. Cache layout is HuggingFace's documented local cache, so a host holding a Python `huggingface_hub` cache gets a hit and `model_loader.cpp:279-303` reads it unchanged. Transport is the already-vendored cpp-httplib with OpenSSL, matching llama.cpp, which retired libcurl at `CMakeLists.txt:195` at stock tag `b10451`. NOT a verbatim port on one point: llama.cpp's `is_valid_oid` accepts any 40- or 64-character hexadecimal string, and on 17 August 2026 this project measured the tree API answering an unauthenticated caller on gated `Lightricks/LTX-2.5` with an `lfs.oid` of one character repeated 64 times, identical for all 14 large-file-storage files, so this row treats an untokenized `lfs.oid` as absent and proves completeness structurally instead | T1 | `vllm/model_executor/model_loader/weight_utils.py:345,349-357,459,472-490,493-496,506`; `vllm/model_executor/model_loader/default_loader.py:167-184`; `vllm/engine/arg_utils.py:839`; `vllm/config/model.py:183` at pin `5559679229`. Secondary oracle llama.cpp for the `:QUANT` form only, stock tag `b10451`, anchors re-verified by W1 at commit `10bf611e533d81f739128304991c5e133c6aebd8` | W1 and W2 only: `src/vllm/transformers_utils/hf_hub.cpp:1`, `src/vllm/transformers_utils/hf_cache.cpp:1`, reached from `src/vllm/entrypoints/model_loader.cpp:280`. No downloader, no `--model` grammar, no TLS option | `tests/vllm/transformers_utils/test_hf_cache.cpp:1`, `tests/vllm/transformers_utils/test_hf_hub.cpp:1` (in-process fake hub), `tests/vllm/entrypoints/test_dflash_draft_hf_cache.cpp:1` (the loader reach, red when the call site is deleted) | [hf-model-download.md](specs/hf-model-download.md) | `READY` | - | | `LOAD-LONGTAIL` | Sharded-state, tensorizer, RunAI, BitsAndBytes loaders | T3 | `vllm/model_executor/model_loader/__init__.py:33-65` | - | - | `planned: specs/loader-longtail.md` | `INVENTORIED` | - | diff --git a/.agents/specs/bpe-quadratic-merge.md b/.agents/specs/bpe-quadratic-merge.md index 22d4c0a7d..d1e4fe536 100644 --- a/.agents/specs/bpe-quadratic-merge.md +++ b/.agents/specs/bpe-quadratic-merge.md @@ -86,7 +86,7 @@ Out of scope, each with a stated reason: ## Our baseline Every figure below was measured on an AMD Ryzen 9 9950X3D (20 cores) at `-O2`, -against a driver built from this tree's own tokenizer sources. **Two harnesses +against a driver built from this tree's own tokenizer sources. **Three harnesses produced them and the tables say which.** - **Harness A**, the one that found the defect: `BpeSplit` alone, against the @@ -99,10 +99,19 @@ produced them and the tables say which.** idle: another session was compiling throughout, which is why the minimum of a repetition set is reported rather than a mean. The minimum is the least contended estimate available, not a claim of an idle host. - -The **shape** is what both harnesses agree on, and it is the result. A -same-binary idle-host re-measure still belongs to the implementing row, and -`## Gates` requires it before any speed claim is accepted. +- **Harness C**, the confirmation: the same Harness B binaries re-run on + 2026-08-19 at load average 23 to 57, which is BUSIER than either earlier run. + It exists to answer one question, whether the recorded constants reproduce. + They do not, and the direction is the one contention predicts: `a` x 65,535 + read 37,564.80 ms against Harness B's 24,358.86 ms, a 54% inflation at roughly + double the load. The growth ratios it re-derived are in `## Tests to port` + item 3 and they replace the figures a previous revision recorded there. + +The **shape** is what all three harnesses agree on, and it is the result. Every +absolute figure below is a contended minimum, and Harness C is the evidence that +none of them is quotable as a constant: the same input on the same binary moved +54% on load alone. A same-binary idle-host re-measure still belongs to the +implementing row, and `## Gates` requires it before any speed claim is accepted. ### The benchmark prompts that found it @@ -600,31 +609,53 @@ prove after. Encode at n and at 4n and assert the ratio sits below a bound that quadratic growth cannot satisfy and the implemented algorithm can. **Use a step of at least 4x, and take the minimum of k repetitions rather than one shot.** Both - requirements were measured, because a first revision of this spec specified a - 2x single-shot ratio and called it "the assertion that survives a slow - runner", which is backwards: - - - **A 2x single shot has about 1.6x of usable band.** Eight independent - single-shot pairs of the prototype at 131,072 and 262,144 bytes, load - average 20 to 30, gave ratios 1.862, 2.291, 2.303, 2.219, 2.508, 2.331, - 2.395 - and 2.336, a 35% spread over the minimum. Linearithmic predicts ~2.1 and - quadratic 4.0, so the band a bound may sit in is (2.51, 4.0). The two - halves are separate, independently preemptible measurement windows. They do - **not** move together, and one deschedule in the 2n half crosses 4.0. - - **A 4x step with min-of-8 has about 2.7x, and is four times more stable.** - Four independent min-of-8 sweeps at 65,536 and 262,144 bytes on the same - box gave 5.51, 5.74, 6.01 and 6.03, a 9% spread. Quadratic predicts 16, - so the band is (6.03, 16). - - The red side is measured on the current code at the same shape: 8,000 to - 32,000 bytes costs 16.74x, against the 16x `n^2` predicts. The prototype's - 5.5 to 6.0 is above the ~4.6 that `n log n` predicts because the prototype - still builds a string key and a merged string per candidate. The interned - implementation removes both, so **the implementing row re-derives the bound - from min-of-k sweeps on its own code** and records the sweep, the `k`, and - the host load beside it. A bound copied from the prototype is not a bound for - the implementation. + requirements are measured below, and so is the limit of what they buy: + + - **A 2x step cannot separate the two algorithms at all under load.** Eight + independent min-of-8 sweeps of the prototype at 131,072 and 262,144 bytes, + load average 50 to 57 on the 20-core box, 2026-08-19, gave 3.328, 2.525, + 1.573, 4.414, 1.485, 3.193, 1.916 and 1.372. Quadratic growth predicts + 4.0, and **the observed maximum 4.414 is already above it**. A 2x + assertion therefore fires on the correct algorithm before it ever fires on + the defect, which is why the step has to be at least 4x. + - **A 4x step still separates, with about 1.8x of margin.** Eight + independent min-of-8 sweeps at 65,536 and 262,144 bytes, same binary, same + session, same load, gave 7.202, 8.920, 4.182, 2.800, 6.626, 4.198, 4.507 + and 8.586. Quadratic predicts 16 and the observed maximum is 8.920, so a + bound sits in (8.92, 16). + + **Min-of-k is necessary and it is not sufficient, and this spec no longer + claims otherwise.** A first revision specified a 2x single-shot ratio and + called it "the assertion that survives a slow runner"; a second replaced that + with a 4x min-of-8 sweep and certified a 9% spread and a (6.03, 16) band from + four sweeps. Neither figure reproduced. The eight-sweep re-derivation above, + taken at load 50 to 57 rather than the earlier revision's lighter and + unrecorded load, spreads 3.2x at both step sizes. The two halves of a ratio + are separate, independently preemptible measurement windows, and taking a + minimum inside each half does not make them move together. **The stability of + this assertion on a contended runner is therefore UNESTABLISHED**, and the + claim that a 4x step is "four times more stable" is withdrawn rather than + restated with a new number. + + The red side, re-derived on the current code at the same shape and load, + min-of-3, four sweeps: 8,000 to 32,000 bytes costs 12.680, 14.740, 16.468 and + 15.276, against the 16x `n^2` predicts. Placing that beside the prototype's + worst gives the real picture: **the window between the prototype's slowest + ratio, 8.920, and the current code's fastest, 12.680, is 1.42x.** That is the + whole budget this assertion has on a loaded host, and it is the reason item 2 + and not this item is the robust gate. + + Two obligations follow for the implementing row. It **re-derives the bound + from min-of-k sweeps on its own code**, because the interned implementation + removes the string key and the merged string the prototype still builds and + should sit nearer the ~4.6 that `n log n` predicts than the prototype's 2.8 + to 8.9. And it records the sweep, the `k`, the load average, and the host + beside the bound, so that the next reader can tell a real regression from a + busy box. A bound copied from the prototype is not a bound for the + implementation, and a bound derived on a loaded host is not a bound at all. + If the assertion cannot be made to hold with margin on the CI runner, it + lands `SKIP`-ped with this paragraph as the reason rather than tuned until it + passes. 4. **The leftmost tie.** Extend the existing `test_bpe.cpp` case into one that distinguishes the two orders on a longer symbol list. 5. **A stale-entry case.** A merge sequence where a queued candidate is diff --git a/scripts/check-agent-record.py b/scripts/check-agent-record.py index e2e9d2aa4..4ffd2e018 100644 --- a/scripts/check-agent-record.py +++ b/scripts/check-agent-record.py @@ -623,6 +623,9 @@ # MEASURED: 64 KB of ordinary English prose costs 25.3 s through the committed # Mistral golden against HF `tokenizers` 0.22.2's 10.1 ms on the same file for # byte-identical identifiers, and the fit over 1 KB to 64 KB has exponent 2.01. +# Those are contended minima on a 20-core box at load 4-90, not idle-host +# constants; the spec's `## Gates` owes the idle-host re-measure. The exponent, +# not the constant, is what makes this a row. # `READY`, spec `specs/bpe-quadratic-merge.md`, issue #1365. # Bumped for a real new row, never to make a failing state transition pass. ENGINE_ROWS = 168 From e9ba42182956694e3ebc2034b2371f6d301f8822 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 19 Aug 2026 23:25:48 +0000 Subject: [PATCH 4/4] spec(SPEC-BPE-QUADRATIC-MERGE): the growth gate is dropped, and the harness three rounds of numbers never had is committed (#1365) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A fresh reviewer could not reproduce this row's certified growth window, for the third consecutive revision. The operator ruled the assertion out rather than deferring it again, and this change carries that ruling, the evidence behind it, and the root-cause fix underneath all three failures. ## The ruling `## Tests to port` item 3 was a growth-shape assertion: encode at n and 4n and bound the ratio. It is dropped. Not deferred to the implementing row, not landed `SKIP`-ped. Ten sweeps of 8,000 to 32,000 bytes on the CURRENT, defective code at load 176-268 gave 4.654 to 17.896, so five of ten fall below the 12.680 a previous revision certified as the defective MINIMUM and one falls below the 8.920 it certified as the prototype's MAXIMUM. The two distributions overlap and no bound fits between them. The mechanism is this row's own diagnosis turned against its own gate. In the sweep that produced 4.654 the 8,000-byte leg read 4,383 ms against the other nine's 650-1,878 ms: a preempted DENOMINATOR alone handed the defective code a ratio the correct algorithm is supposed to own. Min-of-k inside each half cannot help, because the two halves are independently preemptible windows. The margin argument for the 2x step is falsified too. It rested on one observed 4.414, above the 4.0 quadratic predicts. Twenty-four min-of-8 sweeps of HF `tokenizers` 0.22.2 at the pinned version, load 198-228, ranged 0.756 to 3.349 and never exceeded 4.0. It is kept as a recorded negative result, not restated as evidence. Item 2, the absolute cost bound, carries the timing gate alone on about three orders of magnitude of headroom. ## The harness, which is why any of this happened No harness was committed and no build or run recipe was recorded, against AGENTS.md §Gates. Three harnesses existed in prose only, so nothing in the tree let a second reader re-derive one figure — which is exactly how three rounds of constants each contradicted the last. `tools/bench/bpe_encode_cost.cpp` is committed with its `g++` build line and its run lines, in its own header and again in the spec's `## Gates`. It times `Tokenizer::Encode` at stated sizes through either committed golden, min-of-k, and prints the 1/5/15-minute load average beside every row. Its banner says its output is a session reading and never a bound. CI compiles it as the never-linked OBJECT library `vllm_bpe_encode_cost` and runs it never; the row just ruled that a growth gate is not viable on a shared runner, so there is nothing here for CI to decide. Run for this change on the 20-core box at load 194/209/196, min-of-3: English through the Mistral golden read 12.802 ms at 1,000 bytes and 606.424 ms at 8,000, against the spec table's 5.61 ms and 358.83 ms at load 4-12. About 1.7x on load alone, while the identifier counts came back exactly at 267 and 2,134. The identifiers reproduce and the times do not, which is this row in two numbers, and 1.7x against a thousandfold margin is why item 2 is the gate. ## What else was wrong `## Gates` contradicted the spec twice in its binding section. It claimed the harnesses agreed "on every absolute figure to within about 25%" — a sentence that named A and B and was never updated when Harness C recorded 54% on the same binary and the same input. And it required the growth shape red-then-green while item 3 permitted it to land skipped; dropping item 3 removes that. `## Port map` named `tokenizer.cpp::MergeKey`. `MergeKey` is DEFINED at `src/vllm/tokenizer/bpe.cpp:82` and only called from `tokenizer.cpp:64`, and the row two lines above in the same table already said so. The engine-matrix row labelled the 25.3 s / 10.1 ms English figures "load average 25-90". That load belongs to the different 65,535-byte table; the English table records load 4 to 12, so the label made a near-idle figure look contended. Each figure now carries its own load. The confirmation run reads 23-57, which is what the spec records; the row said 23-51 and the two landed in one commit, so the projection is made to agree with the record rather than inventing a third number. The `check-agent-record.py` comment kept the exponent 2.01 and the 2,507x ratio, both ratios taken inside one session, and dropped the absolute milliseconds. Those had become a fourth place for a number that moved 54%. `docs/USAGE.md` gains the paragraph that owes it. The `CMakeLists.txt` target is a user-facing build target, so the doc checkpoint asks for it, and a reader who finds `vllm_bpe_encode_cost` in a build log deserves to be told what it is, that nothing runs it, and how to read its output. ## Two pins moved, and both are load-bearing Adding the recipe to `## Gates` puts the row in `check-gate-commands.py`'s runnable population, so `RUNNABLE_BASELINE` is re-pinned in this change with the row and the reason named, and `test_bpe_quadratic_merge_earns_its_runnable_baseline_entry` proves the entry is earned: removing it reds the equality. Verified red at rc=1 with the entry removed and green at rc=0 restored, with the file sha256-identical afterwards. The comment edit to `check-agent-record.py` shifted five `ENG-RECORD-ANCHOR-RATCHET` citations by four lines, which the anchor ratchet caught at stale=37 against baseline 32. The five anchors are repaired to the lines their symbols now hold and the report is back at stale=32, broken=6. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [Claude Code] --- .agents/engine-matrix.md | 4 +- .agents/specs/bpe-quadratic-merge.md | 245 ++++++++++++++-------- CMakeLists.txt | 28 +++ docs/USAGE.md | 23 ++ scripts/check-agent-record.py | 16 +- scripts/check-gate-commands.py | 10 + tests/scripts/test_check_gate_commands.py | 21 ++ tools/bench/bpe_encode_cost.cpp | 243 +++++++++++++++++++++ 8 files changed, 498 insertions(+), 92 deletions(-) create mode 100644 tools/bench/bpe_encode_cost.cpp diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index 954de5bcf..cea88cb18 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -224,7 +224,7 @@ claims it. | `ENG-RELEASE-WINDOWS` | Native Windows x86_64 pre-alpha release extension: one adaptive MSVC/UCRT CPU bundle with AVX2 executed in CI and one Vulkan preview bundle, both deterministic ZIPs and authenticated by the existing release handoff | T0 | vLLM has no Windows release path; runtime behavior remains pinned to vLLM `555967922`. Platform substrate reference: llama.cpp `src/llama-mmap.cpp:520-590` @ `237ad9b961f009ae19ac29dbce4cd0c1251f94b3`; Win32 API is the OS authority | W14 Win32 portability/MSVC CPU, W15 deterministic ZIP/PE packaging + Vulkan, and W16 ten-tuple prerelease workflow/version/docs implemented for one PR | Linux portability/release mutation gates are local evidence only. Native `windows-2022` MSVC `/W4 /WX`, extracted runtime/ISA smokes, merged-SHA ten-tuple dry run, `v0.0.3-pre.1` publication, attestations, and exact 32-asset audit remain pending; no Windows ZIP exists yet | [windows-binary-release.md](specs/windows-binary-release.md); [#117](https://github.com/mudler/vllm.cpp/issues/117) | `ACTIVE` | `CLAIM-ENG-RELEASE-WINDOWS` | | `ENG-RELEASE-CONTAINERS` | Published OCI container images on GHCR, built by GitHub Actions: the same staged server bundle as `ENG-RELEASE-BINARIES`, shipped from one package `ghcr.io/mudler/vllm.cpp` with the lane in the tag — `:-cuda` / `-vulkan` / `-cpu`, the moving `:latest-cuda` / `:latest-vulkan` / `:latest-cpu`, and a bare `:latest` aliasing the cpu lane, with `ENTRYPOINT vllm-server`. Lanes `cuda` (one fat image covering every supported SM), `vulkan`, `cpu` (adaptive baseline); `rocm` blocked-preview, tracking its binary channel. Version tags are immutable; every `latest-` moves. Each lane is a `linux/amd64` + `linux/arm64` multi-arch manifest built on native runners — aarch64 is first-class here because GB10 (sm_121a), Thor (sm_110) and Orin (sm_87) are all arm64. The image contains the bundle and nothing else: no weights, no Python, no PyTorch, no compiler, no build tree. BOUNDARY: the GPU driver and container runtime stay on the host and are never bundled; Metal and MLX are NOT-CONTAINERIZABLE (no macOS container runtime and no Metal passthrough exists) and remain static-binary-only lanes, recorded as a permanent boundary rather than pending work. No image, workflow, registry package or pull is claimed to exist. | T0 | release image lanes `.buildkite/release-pipeline.yaml:34-170` and the published-image dependency boundary `docker/Dockerfile.cpu:262-290` @ `555967922` | `docker/Dockerfile` (cpu/vulkan/cuda targets calling the release scripts); `docker/healthcheck.sh`; `release/container-matrix.json`; `scripts/check-container-matrix.py`; `scripts/check-container-workflow.py`; `scripts/validate-container-image.py`; `scripts/container_tags.py`; `.github/workflows/containers.yml`; SIGTERM handler `src/vllm/entrypoints/openai/server_main.cpp` (`SignalShutdown`, all three `listen()` sites); the pre-existing `docker/Dockerfile.arm64` is an unrelated CPU bench cross-check | issues `#170`, `#312`, `#394`; `tests/scripts/test_check_container_matrix.py` 31/31; `test_check_container_workflow.py` 29/29; `test_check_cuda_fat_gencode.py` 7+4 subtests. **GB10 2026-08-11 (`promaxgb10-4ad8`, `sm_121a`, CUDA 13.3): arm64 cuda image 1.71 GB, 673/673 objects, ten-SM gencode audit PASS, and a REAL GPU boot -- `/health` 200, `/version` 200, in-container healthcheck, clean SIGTERM, `--gpus all`, host driver 580.159.03 injected.** cpu amd64 783 MB gated locally; cpu+vulkan amd64 green on hosted CI **arm64 cuda lane RUNTIME-VERIFIED on GB10 2026-08-11** -- the first accelerator-hardware evidence for any lane. Four defects were removed to get there, each found by building rather than reading: the CUDA 12.9 base could not compile `sm_110`, the BuildKit cache mount outlived its toolchain (both #366), Marlin gencode had drifted from the feature table and failed the audit on 14 correctly-compiled TUs (#394, blocking BOTH cuda tuples project-wide), and the validator could only ever produce build evidence because its boot smoke never passed `--gpus`. **NOT established: nothing is published to GHCR; amd64 cuda is unbuilt; the published arm64 image is SBSA (`targets/sbsa-linux`), so Tegra -- Thor `sm_110`, Orin `sm_87` -- is untested and NOT covered** **ORIN (Tegra) 2026-08-11: the SBSA image RUNS on Jetson AGX Orin `sm_87` (L4T R36.4.3, Docker 27.5.1) -- Qwen3-0.6B (rev `c1899de2`) loads and GENERATES via `/v1/completions`, tegrastats GR3D 95-97% during decode vs 14-15% idle.** Tegra needs `--runtime nvidia --gpus all`: `--gpus` alone is refused by the hook and `--runtime` alone mounts no driver | [container-images.md](specs/container-images.md); issues [#170](https://github.com/mudler/vllm.cpp/issues/170), [#312](https://github.com/mudler/vllm.cpp/issues/312), [#394](https://github.com/mudler/vllm.cpp/issues/394) | `ACTIVE` | `CLAIM-ENG-RELEASE-CONTAINERS-W1-W7` | | `ENG-DOCS-SITE` | Publish the 11 `docs/*.md` as a browsable GitHub Pages site at `https://mudler.github.io/vllm.cpp/` WITHOUT a second copy of the prose. A Hugo site at `website/` mounts `../docs` READ-ONLY and derives everything else from what is already in the files: each page title from the file's first `# H1`, the sidebar order from `website/data/nav.yaml`, and links through a Goldmark render hook (internal `.md` → site URL; the 139 `../.agents/**` and `../AGENTS.md` escapes → GitHub blob URLs, since the protocol tree is deliberately NOT published). **No file under `docs/` is modified, moved, renamed, or given front matter**, so `check-doc-checkpoint.py` and every protocol path reference keep working and there is no second surface that can drift — the whole point of the row. Custom lean layouts, NO theme and NO submodule: off-the-shelf docs themes read titles, weights and menus out of front matter this design deliberately does not have, so each would need its title partial, menu and link hook overridden anyway, and hugo-book additionally floors at Hugo 0.158 against the 0.146.3 pin CI and the local toolchain share. Hard prerequisite inside the repo: `classify_path` in `scripts/check-pr-size.py` FAILS CLOSED on `website/**` (verified: raises `ValueError: unclassified repository path`), so the classifier must learn the path or the PR cannot pass the project's own size gate. Hard prerequisite outside it: GitHub Pages must be enabled with the source set to GitHub Actions — the workflow is inert otherwise. A marketing landing page is explicitly OUT of scope (`README.md` stays the front door), as is any restructuring of `docs/`; the custom domain is parked behind the pending vLLM trademark question | T1 | NO vLLM analogue — upstream's docs are a separate mkdocs site and nothing in this row mirrors upstream *behavior*, so it carries no parity obligation. The STRUCTURAL reference is LocalAI's `.github/workflows/gh-pages.yml` (two Hugo sites merged into one Pages artifact), reduced to the docs half | read-only mount `website/hugo.toml:29`; title-from-H1 `website/layouts/partials/title.html:10`; link rewriting `website/layouts/_default/_markup/render-link.html:27`; guard `scripts/check-site.py:70`; deploy `.github/workflows/gh-pages.yml` | `tests/scripts/test_check_site.py:51,56,66,80,89,97` (6 mutation cases: clean tree, H1 stripped, doc absent from nav, nav entry with no file, duplicated entry, missing nav file); build evidence 14 pages with `docs/bench-evidence` + `docs/superpowers` absent from `public/` and no `href` ending in `.md`; 48 protocol links rewritten in `docs/status/`. NO published page is claimed: GitHub Pages is not yet enabled on the repository, which is the recorded stop condition holding this row at `GATING` | [gh-pages-docs-site.md](specs/gh-pages-docs-site.md); issue [#224](https://github.com/mudler/vllm.cpp/issues/224) | `READY` | `CLAIM-ENG-DOCS-SITE` | -| `ENG-RECORD-ANCHOR-RATCHET` | **The record's `path:line` citations were range-checked and never reported.** `check-agent-record.py` parsed BOTH forms: markdown links, and bare `` `file.cpp:123` `` through `RAW_LOCAL_ANCHOR_RE` since `ee511ca8a`. On a missing file or an out-of-range line `local_line_anchors` runs `continue`, so the bad anchor never reaches the caller, and `is_code_anchor` then answers with **any**, so one good sibling covers the rest. There was no symbol test and no report, and **32 of the 38** offenders are IN RANGE, so range-checking could not have found them. Measured at `8daa67b39`: **832 of 867** in-scope citations (**96.0%**) were already parsed and range-checked, and the **35** new to parsing sit under `.agents/`, `docs/` and `website/`; `EVIDENCED_STATES` omits `ACTIVE`/`READY` entirely and is deliberately NOT widened, because requiring an anchor there raises 85 errors across 53 rows. Even the fraction it saw was only range-checked, never checked to CONTAIN the symbol named beside it — every stale anchor found in the 2026-08-13/14 campaign was in range. LANDED as a device-leakage-shaped ratchet over a recorded baseline, never a bulk cleanup: the backlog is fixed by whoever next touches each row | T1 | none — this is our own record surface; the discipline mirrors AGENTS.md §Records ("cite the `file:line` you ported from") | parser + classifier + ratchet in `check-agent-record.py`: `BARE_CITATION_RE` `scripts/check-agent-record.py:1254` (the bare form), `cell_citations` `scripts/check-agent-record.py:1308` (both forms, with the adjacent-symbol rule), `classify_citation` `scripts/check-agent-record.py:1365` (OK / STALE / BROKEN), `RECORD_ANCHOR_STATES` `scripts/check-agent-record.py:1252` (gap 3: `ACTIVE` and `READY` join the count), `check_record_anchors` `scripts/check-agent-record.py:1488` (the two-way gate); budget in `scripts/record-anchor-baseline.json` | `RecordAnchorRatchet` `tests/scripts/test_agent_record.py:1397` — 10 cases, RED-first, including `test_one_good_link_does_not_cover_a_rotted_bare_citation` `tests/scripts/test_agent_record.py:1465`, the `any()` shape the rot hid in. Five mutants red it: report-only, `EVIDENCED_STATES` restored, links-only, first-citation-only, range-only. Measured baseline **38** (32 STALE + 6 BROKEN); gate wired in `scripts/agent-preflight.sh` and the `agent-record` CI job (`--report`) | [record-anchor-ratchet.md](specs/record-anchor-ratchet.md) | `ACTIVE` | `CLAIM-ENG-RECORD-ANCHOR-RATCHET` | +| `ENG-RECORD-ANCHOR-RATCHET` | **The record's `path:line` citations were range-checked and never reported.** `check-agent-record.py` parsed BOTH forms: markdown links, and bare `` `file.cpp:123` `` through `RAW_LOCAL_ANCHOR_RE` since `ee511ca8a`. On a missing file or an out-of-range line `local_line_anchors` runs `continue`, so the bad anchor never reaches the caller, and `is_code_anchor` then answers with **any**, so one good sibling covers the rest. There was no symbol test and no report, and **32 of the 38** offenders are IN RANGE, so range-checking could not have found them. Measured at `8daa67b39`: **832 of 867** in-scope citations (**96.0%**) were already parsed and range-checked, and the **35** new to parsing sit under `.agents/`, `docs/` and `website/`; `EVIDENCED_STATES` omits `ACTIVE`/`READY` entirely and is deliberately NOT widened, because requiring an anchor there raises 85 errors across 53 rows. Even the fraction it saw was only range-checked, never checked to CONTAIN the symbol named beside it — every stale anchor found in the 2026-08-13/14 campaign was in range. LANDED as a device-leakage-shaped ratchet over a recorded baseline, never a bulk cleanup: the backlog is fixed by whoever next touches each row | T1 | none — this is our own record surface; the discipline mirrors AGENTS.md §Records ("cite the `file:line` you ported from") | parser + classifier + ratchet in `check-agent-record.py`: `BARE_CITATION_RE` `scripts/check-agent-record.py:1258` (the bare form), `cell_citations` `scripts/check-agent-record.py:1312` (both forms, with the adjacent-symbol rule), `classify_citation` `scripts/check-agent-record.py:1369` (OK / STALE / BROKEN), `RECORD_ANCHOR_STATES` `scripts/check-agent-record.py:1256` (gap 3: `ACTIVE` and `READY` join the count), `check_record_anchors` `scripts/check-agent-record.py:1492` (the two-way gate); budget in `scripts/record-anchor-baseline.json` | `RecordAnchorRatchet` `tests/scripts/test_agent_record.py:1397` — 10 cases, RED-first, including `test_one_good_link_does_not_cover_a_rotted_bare_citation` `tests/scripts/test_agent_record.py:1465`, the `any()` shape the rot hid in. Five mutants red it: report-only, `EVIDENCED_STATES` restored, links-only, first-citation-only, range-only. Measured baseline **38** (32 STALE + 6 BROKEN); gate wired in `scripts/agent-preflight.sh` and the `agent-record` CI job (`--report`) | [record-anchor-ratchet.md](specs/record-anchor-ratchet.md) | `ACTIVE` | `CLAIM-ENG-RECORD-ANCHOR-RATCHET` | | `ENG-RECORD-CONFLICT-SURFACES` | Retire the shared record surfaces that make concurrent PRs conflict by construction. MEASURED at `origin/main` `d928e2c3` with `git merge-tree --write-tree` over every open PR: **16 of 29 conflict (55%), and 13 of the 16 conflict in bookkeeping files ONLY**, with no product code involved — `.agents/coordination.md` in 8, `.agents/NOW.md` in 5, `.agents/roadmap_v1.md` in 4, `scripts/check-public-doc-tables.py` in 4, `docs/STATUS.md` in 4, and any `src/`/`tests/` path in just 3. Three defects, each of which GUARANTEES rather than risks a collision. (1) `.agents/NOW.md` is a fixed-size shared buffer at EXACTLY 6000/6000 chars (`check-now-current.py:31`), so adding a row requires evicting another and every PR is a read-modify-write of one global — and the conflict is the LUCKY outcome, since a clean three-way merge would apply both evictions and both additions, silently dropping live rows and blowing the very budget the checker defends. (2) `STATUS_RATCHET = {"chars": 243245}` (`check-public-doc-tables.py:557`) is a hardcoded byte count of a DIFFERENT file that may only fall, so a PR owing `docs/STATUS.md` one lifecycle line must delete unrelated prose from another row to pay for it and edit the checker too; the checker's own comment at `:331` already records the failure (*"a ratchet pinned to the byte turns every concurrently merged row's one-line status edit into a spurious failure"*) and answered it with slack instead of removing the coupling. (3) `.agents/coordination.md`'s active-claims table is insert-at-one-anchor: the six ROCm GDN PRs (#334 #336 #341 #343 #345 #348) are ONE author's sequential stack that conflicts on nothing else, each appending a ~1,500-char row — the PR description, transcribed into a file every other claim also writes. It also contradicts the protocol it serves: `AGENTS.md` holds that *"History is git"* and *"There is no state log"*, yet both claims tables ARE state logs duplicating `gh pr list`, `row/` branch names and issue state; the argument that refuses a waiver registry applies unchanged to a claims registry. Precedent twice over — `policy.csv` retired in `0f3e44ee`, per-class line budgets retired 2026-08-10 because the gate fired on ordinary work. The exonerated surfaces share ONE property, one writer per file: `.agents/specs/.md` (one file per row, **zero conflicts** in the sample), the `*-matrix.md` inventories, and the append-only `.agents/benchmark-record.md`. SCOPE: remove `STATUS_RATCHET` and the doc-gating global counters while KEEPING the per-cell/per-paragraph caps (local, so they couple nothing); remove the active-claims table and derive claims from open PRs and branch names; drop `NOW.md`'s byte budget; order the roadmap's keyed tables by ID so distinct keys stop colliding at one anchor; and record the invariant — **no surface that every PR must write** — in `AGENTS.md`. No product source, kernel or gate semantic moves | T0 | NO vLLM analogue — this is local protocol machinery, so the mirror rule does not apply and no upstream `file:line` exists to port from. Governed instead by `AGENTS.md` §"Changing the rules or a checker", which requires a spec, a red-before test or mutation, and green-after evidence | - | - (spec-before-code: the red-before suites are named in the spec's Tests section — `tests/scripts/test_check_public_doc_tables.py`, `tests/scripts/test_check_now_current.py`, a mutation case per removed rule proving the obligation survives in the retained caps and `check-doc-checkpoint.py`, and a `git merge-tree` merge-shape regression that must be RED before the `NOW.md`/roadmap work and GREEN after) | [retire-shared-record-surfaces.md](specs/retire-shared-record-surfaces.md); issue [#364](https://github.com/mudler/vllm.cpp/issues/364) | `READY` | `CLAIM-ENG-RECORD-CONFLICT-SURFACES` | | `ENG-TRAILER-MERGE-ARTIFACTS` | The trailer gate rejects CORRECT commits because of paragraph placement, and that is why `main` is red on `agent-record`. `check-commit-trailers.py` reads trailers via `git interpret-trailers --parse`, which treats ONLY the final paragraph as the block; GitHub appends `Co-authored-by:` as a SEPARATE trailing paragraph on a squash merge, so a complete correct block becomes invisible and the gate reports it missing. MEASURED: piping `git show -s --format=%B dbd0d51c` into `git interpret-trailers --parse` prints nothing but the co-author line, and 13 of the last 30 commits on `main` fail the check -- unnoticed only because those runs were cancelled (#274), which HID the defect rather than causing it. FIX: fuse consecutive trailing TRAILER-SHAPED paragraphs before parsing. Nothing is relaxed -- the block must still exist, the marker must still sit above it, each declaration must still appear exactly once, and an AI co-author is still forbidden; the block is merely FOUND where the merge tool left it. A prose paragraph still terminates it. REJECTED IN FLIGHT and recorded because it is the more instructive half: a first attempt also collapsed identical duplicate trailers to fix the multi-commit-squash shape, which relaxes the uniqueness rule an existing test already pins. Rewriting that assertion to suit the change is what AGENTS.md forbids, and the distinction is real -- a doubled block is genuinely malformed and fixable at source, whereas the co-author case is a correct commit defeated by the parser. Reverted in full. SCOPE LIMIT, stated rather than implied: this fixes ONE of five observed shapes. `f64f2b71` (bot co-author) is a REAL violation the parse had been hiding and now correctly fails; `87308dea` (GitHub's `---------` separator), `b8293c88` (squash doubled the block) and `b580452d` (merge button, no trailers) stay red by design. Closing those is a merge-method change, not a checker change | T0 | NO vLLM analogue -- local protocol machinery, so the mirror rule does not apply and there is no upstream `file:line` to port from. Governed by `AGENTS.md` §"Changing the rules or a checker" | `scripts/check-commit-trailers.py:60` (`join_trailing_trailer_paragraphs`, `_is_trailer_paragraph`, and the fused `parsed_trailers`) | `tests/scripts/test_check_commit_trailers.py:1` 21 cases -- the RED-BEFORE appended-co-author case plus four GUARDS that keep the fusion bounded (doubled block still fails, contradictory declarations still fail, a no-trailer merge message still fails, prose after the block still fails), all four green before and after; closure [parity-ledger.md#L941](parity-ledger.md#L941) | [trailer-merge-artifacts.md](specs/trailer-merge-artifacts.md); issue [#406](https://github.com/mudler/vllm.cpp/issues/406) | `DONE` | `157080c8` | | `ENG-FORGE-COAUTHOR` | The forbidden-AI-trailer rule was catching ATTRIBUTION rather than an authorship claim, which is why bot-opened PRs red `main` on merge. GitHub composes the squash message itself and appends the account that opened the PR — `Co-authored-by: localai-org-maint-bot <...@users.noreply.github.com>` — and most PRs here are opened by a bot, so nearly every squash trips the AI-identity check. Real instance `f64f2b71`, invisible until #406 repaired the parse, which is why it reads as a new failure and is not one. The rule exists so an AI cannot claim it WROTE the code, and that stays; GitHub is recording who pressed the button, and the AI-involvement claim is already carried separately by `AI-Assisted` and `Assisted-by` in the same block. FIX: accept a `Co-authored-by` at a GitHub account noreply address even when the name matches an AI identity token, keyed on the FORGE'S OWN DOMAIN rather than the name so the exemption cannot be borrowed. A hand-written `Co-authored-by: Claude ` still fails; `Signed-off-by` is excluded from the exemption entirely, because a sign-off is a legal assertion about provenance rather than attribution. `AGENTS.md` records the same distinction in the same change so prose and checker cannot drift | T0 | NO vLLM analogue -- local protocol machinery, so the mirror rule does not apply and there is no upstream `file:line` to port from. Governed by `AGENTS.md` §"Changing the rules or a checker" | `scripts/check-commit-trailers.py:38` (`FORGE_ACCOUNT_EMAIL` and the forbidden-trailer skip) | `tests/scripts/test_check_commit_trailers.py:1` 25 cases -- the RED-BEFORE forge-bot case plus THREE guards that matter more than the relaxation because this LOOSENS a rule: a hand-written AI co-author still fails, `Signed-off-by` at the same noreply address still fails, and a human co-author still passes; all three green before and after. Real commit `f64f2b71` re-verified per commit | [forge-coauthor-attribution.md](specs/forge-coauthor-attribution.md); issue [#418](https://github.com/mudler/vllm.cpp/issues/418) | `ACTIVE` | `CLAIM-ENG-FORGE-COAUTHOR` | @@ -277,7 +277,7 @@ claims it. | `LOAD-GGUF-MMPROJ` | A SECOND, `clip`-architecture GGUF projector file beside the language file, and the Qwen3-VL vision tower loaded out of it. The single-file assumption is structural, not incidental: `ModelSource` carries a VECTOR of safetensors shards and exactly ONE `GgufFile*`, and `EngineParams` has no projector field at all, so an mmproj has nowhere to arrive. `GgufFile::Open` does handle SHARDS (`DetectSplit` parses `-NNNNN-of-MMMMM.gguf`), but shards of one split are not a second, differently-architected file. Nothing in this tree loads a `clip` projector today and the two apparent counterexamples are neither: MiniMax-H3 reads `visual.*` out of the SAME encoder GGUF and only from a test, and MuseGlimmer's mmproj path is a REFUSAL whose only caller is a test — there is no production call site because no production path accepts a second GGUF path. First consumer is `Qwen3.8-27B` ([#821](https://github.com/mudler/vllm.cpp/issues/821)), whose `mmproj-BF16.gguf` (334 tensors, `clip.projector_type = qwen3vl_merger`, header-verified 2026-08-18: data end == file size 931,146,432) ships BOTH halves of the temporal patch embedding as `v.patch_embd.weight` + `v.patch_embd.weight.1` — precisely what MuseGlimmer's lacks, so the check that distinguishes them is this row's, not a reason to refuse. Second consumer is MuseGlimmer, whose refusal becomes REACHABLE production code the moment this lands | T0 | Pinned vLLM has no GGUF loader at all (`555967922`, `model_loader/__init__.py:33-49`), so the compatibility reference is llama.cpp `b10451` = `10bf611e5` (`PROJECTOR_TYPE_QWEN3VL`; the previously recorded `tools/mtmd/clip-impl.h:330` was read at the SUPERSEDED local fork `237ad9b96` and is owed re-anchoring, [#1003](https://github.com/mudler/vllm.cpp/issues/1003)) | the one open site [model_loader.cpp:1679](../src/vllm/entrypoints/model_loader.cpp#L1679), the single pointer [model_registry.h:98](../include/vllm/model_executor/models/model_registry.h#L98), the arch dispatch [model_loader.cpp:794](../src/vllm/entrypoints/model_loader.cpp#L794), the refusal precedent [muse_glimmer_gguf_weights.cpp:695](../src/vllm/model_executor/models/muse_glimmer_gguf_weights.cpp#L695) | none yet — this row is spec-only. The port target for the vision config mapping is [minimax_h3_vision_gguf.cpp:32](../src/vllm/model_executor/models/minimax_h3_vision_gguf.cpp#L32), which builds the same `Qwen3VLVisionConfig` from `visual.*` | [quantized arms of Qwen3.8-27B](specs/qwen38-27b-quant-arms.md) | `READY` | - | | `LOAD-HF-BPE` | HF tokenizer.json byte-level BPE and incremental detokenization | T0 | `vllm/tokenizers/registry.py:176`; `vllm/tokenizers/hf.py:163`; `tests/tokenizers_/test_hf.py:18`; `tests/tokenizers_/test_detokenize.py:148` | `src/vllm/tokenizer/tokenizer.cpp:240,484,512`; `src/vllm/v1/engine/detokenizer.cpp:409` | `tests/vllm/test_bpe.cpp:226`; `tests/vllm/test_tokenizer_parity.cpp:66,74,82,90`; `tests/vllm/test_pretokenizer.cpp:130` | `planned: specs/hf-tokenizer.md` | `ANCHOR-BACKFILL` | - | | `LOAD-SENTENCEPIECE` | SentencePiece (Metaspace + byte-fallback) BPE tokenizer family — the gate to Mistral/Gemma/SentencePiece `tokenizer.json`. `Tokenizer::FromHfJson` now dispatches on the pre_tokenizer FAMILY: a bare `Metaspace` node selects the SP path (space→▁ U+2581; `prepend_scheme` first/always/never with the starts-with-▁ guard; `split=false`; out-of-vocab char → `<0xNN>` byte-fallback tokens; merge-ranked BPE over the raw-UTF-8 string), else the BYTE-IDENTICAL byte-level path (`DetectPattern` still fails loudly on Metaspace, so families never overlap). Decode + the incremental detokenizer mirror HF's Sequence decoder (Replace ▁→space, ByteFallback, Fuse, Strip 1 leading space). **UNBLOCKS the Mistral (`MODEL-TEXT-mistral-mistral-for-causal-lm`) paged-engine SACRED gate.** `split=true` fails loudly (no golden in scope). Mirrors HF `tokenizers` 0.22 | T1 | `vllm/tokenizers/hf.py:163`; `vllm/tokenizers/mistral.py:235,467`; HF `tokenizers` 0.22 `pre_tokenizers/metaspace.rs`, `models/bpe/model.rs::merge_word`, `decoders/{replace,byte_fallback,fuse,strip}.rs` | dispatch `src/vllm/tokenizer/tokenizer.cpp:184` (`DetectMetaspace`), `:732` (`EncodePlainSp`), `:878` (`SpDecodeTokens`), `:927` (`Decode` SP branch); merge-loop factor `src/vllm/tokenizer/bpe.cpp:91` (`BpeMerge`); family/params/accessors `include/vllm/tokenizer/tokenizer.h:88,98`; incremental dispatch `src/vllm/v1/engine/detokenizer.cpp:357`; generator `tools/parity/dump_tokenizer_mistral.py` | `tests/vllm/test_tokenizer_parity_mistral.cpp:78` **6/6, 421 assertions byte-exact** vs HF `tokenizers` 0.22.2 (= vLLM 0.25.0 backend) over a 45-entry Metaspace/byte-fallback/special-token corpus (goldens `tests/parity/goldens/tokenizer_mistral/{tokenizer.json,encodings.json}`); SACRED cross-check vLLM `AutoTokenizer` (transformers 5.13.1) 0/45 mismatch (±BOS); byte-level suites byte-identical `tests/vllm/test_bpe.cpp` 852, `tests/vllm/test_detokenizer.cpp` 221, `tests/vllm/test_tokenizer_parity.cpp` 1175, `tests/vllm/test_tokenizer_parity_deepseek.cpp` 2461 | [specs/sentencepiece.md](specs/sentencepiece.md) | `ANCHOR-BACKFILL` | `CLAIM-LOAD-SENTENCEPIECE` | -| `SPEC-BPE-QUADRATIC-MERGE` | **The BPE merge loop is O(n^2) in pretoken length, on the request path, before the length check.** `src/vllm/tokenizer/bpe.cpp::BpeMerge` rescans every adjacent pair per merge and builds one `std::string` key per probe through `src/vllm/tokenizer/bpe.cpp::MergeKey`, on the premise its own comment states: "pretokens are tiny". FIVE of the seven pretokenizer rules return an unbounded run (`src/vllm/tokenizer/pretokenizer.cpp::MatchLetterRun` rule 2, `src/vllm/tokenizer/pretokenizer.cpp::MatchPunctRun` rule 4, `src/vllm/tokenizer/pretokenizer.cpp::MatchWsNewlines` rule 5, `src/vllm/tokenizer/pretokenizer.cpp::MatchWsNotBeforeNonSpace` rule 6, `src/vllm/tokenizer/pretokenizer.cpp::MatchWs` rule 7); only rule 3 `MatchNumbers` is capped and only rule 1 `MatchContraction` is bounded by its own alternation, and on the SentencePiece family `src/vllm/tokenizer/tokenizer.cpp::EncodePlainSp` merges the WHOLE prompt as one word because Mistral and Gemma declare `split: false`. MEASURED at `31f93787c`, min-of-k on a CONTENDED 20-core box at load average 25-90, so these are contended minima and not idle-host constants: 65,535 bytes of one repeated character costs 24.4-45.8 s of one core; 64 KB of ordinary English prose through the committed Mistral golden costs 25.3 s against HF `tokenizers` 0.22.2's 10.1 ms on the same file for byte-identical identifiers, growing n^2.0. A confirmation run at load 23-51 read 37.6 s where the recorded figure reads 24.4 s, so the SHAPE is the result and the constants move with the box. `src/vllm/v1/engine/input_processor.cpp::ValidatePromptLen` runs FIVE LINES AFTER the encode, so `max_model_len` bounds none of it, and `/tokenize` reaches the same path with no engine. This is a remote denial of service, not a latency curiosity. Fix mirrors HF `tokenizers` 0.22.2 `Word::merge_all` (heap over candidate merges, identifier-keyed merge table); a prototype is bit-identical to today's identifiers on 20 comparisons across BOTH committed goldens, all 20 of which also match HF `tokenizers` 0.22.2 reading the same files, and 8,192 newlines fall from 580.44 ms to 2.60 ms. A pretoken cap is REFUSED: it changes the token identifiers. The string key is not a separable win either, on the DESIGN: upstream's staleness test compares `new_id` (`word.rs:197-205`) and its table is keyed on an identifier pair, so a step that rebuilds a `std::string` to name a pair cannot express the test; and removing a constant from a quadratic leaves a quadratic. How much the key alone is worth is NOT measured on an idle host and this row claims nothing about it | T0 | HF `tokenizers` 0.22.2 `tokenizers/src/models/bpe/word.rs:162-250` (`Word::merge_all`), `:28-35` (`Ord for Merge`), `tokenizers/src/models/bpe/mod.rs:9` + `model.rs:19,174-192` (identifier-keyed merge table and its load-time refusal) | `src/vllm/tokenizer/bpe.cpp::BpeMerge` (the defect); callers `src/vllm/tokenizer/tokenizer.cpp::EncodePlain` and `src/vllm/tokenizer/tokenizer.cpp::EncodePlainSp` | none yet — the corpus at `tests/parity/goldens/tokenizer_qwen36/` has a longest pretoken of 54 bytes, which is why every existing gate is green | [specs/bpe-quadratic-merge.md](specs/bpe-quadratic-merge.md) | `READY` | - | +| `SPEC-BPE-QUADRATIC-MERGE` | **The BPE merge loop is O(n^2) in pretoken length, on the request path, before the length check.** `src/vllm/tokenizer/bpe.cpp::BpeMerge` rescans every adjacent pair per merge and builds one `std::string` key per probe through `src/vllm/tokenizer/bpe.cpp::MergeKey`, on the premise its own comment states: "pretokens are tiny". FIVE of the seven pretokenizer rules return an unbounded run (`src/vllm/tokenizer/pretokenizer.cpp::MatchLetterRun` rule 2, `src/vllm/tokenizer/pretokenizer.cpp::MatchPunctRun` rule 4, `src/vllm/tokenizer/pretokenizer.cpp::MatchWsNewlines` rule 5, `src/vllm/tokenizer/pretokenizer.cpp::MatchWsNotBeforeNonSpace` rule 6, `src/vllm/tokenizer/pretokenizer.cpp::MatchWs` rule 7); only rule 3 `MatchNumbers` is capped and only rule 1 `MatchContraction` is bounded by its own alternation, and on the SentencePiece family `src/vllm/tokenizer/tokenizer.cpp::EncodePlainSp` merges the WHOLE prompt as one word because Mistral and Gemma declare `split: false`. MEASURED at `31f93787c`, min-of-k on a CONTENDED 20-core box, so these are contended minima and not idle-host constants, and EACH figure carries its own load: 65,535 bytes of one repeated character costs 24.4-45.8 s of one core at load average 25-90; 64 KB of ordinary English prose through the committed Mistral golden costs 25.3 s AT LOAD AVERAGE 4-12 against HF `tokenizers` 0.22.2's 10.1 ms on the same file for byte-identical identifiers, growing n^2.0. A confirmation run at load 23-57 read 37.6 s where the 65,535-byte figure reads 24.4 s, and a fourth reading through the committed harness `tools/bench/bpe_encode_cost.cpp` moved the 8 KB English figure about 1.7x at load 194, so the SHAPE is the result and the constants move with the box. `src/vllm/v1/engine/input_processor.cpp::ValidatePromptLen` runs FIVE LINES AFTER the encode, so `max_model_len` bounds none of it, and `/tokenize` reaches the same path with no engine. This is a remote denial of service, not a latency curiosity. Fix mirrors HF `tokenizers` 0.22.2 `Word::merge_all` (heap over candidate merges, identifier-keyed merge table); a prototype is bit-identical to today's identifiers on 20 comparisons across BOTH committed goldens, all 20 of which also match HF `tokenizers` 0.22.2 reading the same files, and 8,192 newlines fall from 580.44 ms to 2.60 ms. A pretoken cap is REFUSED: it changes the token identifiers. A GROWTH-RATIO gate is refused too, by operator ruling recorded in the spec's `## Tests to port` item 3: at load 176-268 the defective code's own 4x ratios spread 4.654 to 17.896 and OVERLAP the correct algorithm's, because the two halves of a ratio are independently preemptible. The absolute cost bound of item 2 carries the timing gate alone, on about three orders of magnitude of headroom. The string key is not a separable win either, on the DESIGN: upstream's staleness test compares `new_id` (`word.rs:197-205`) and its table is keyed on an identifier pair, so a step that rebuilds a `std::string` to name a pair cannot express the test; and removing a constant from a quadratic leaves a quadratic. How much the key alone is worth is NOT measured on an idle host and this row claims nothing about it | T0 | HF `tokenizers` 0.22.2 `tokenizers/src/models/bpe/word.rs:162-250` (`Word::merge_all`), `:28-35` (`Ord for Merge`), `tokenizers/src/models/bpe/mod.rs:9` + `model.rs:19,174-192` (identifier-keyed merge table and its load-time refusal) | `src/vllm/tokenizer/bpe.cpp::BpeMerge` (the defect); callers `src/vllm/tokenizer/tokenizer.cpp::EncodePlain` and `src/vllm/tokenizer/tokenizer.cpp::EncodePlainSp` | none yet — the corpus at `tests/parity/goldens/tokenizer_qwen36/` has a longest pretoken of 54 bytes, which is why every existing gate is green | [specs/bpe-quadratic-merge.md](specs/bpe-quadratic-merge.md) | `READY` | - | | `LOAD-CONFIG-SURFACE` | Dataclass-for-dataclass config and serve-compatible flags | T0/T1 | `vllm/config/scheduler.py:26`; `vllm/config/cache.py:25`; `vllm/config/compilation.py:378` | `include/vllm/config/scheduler.h:67`; `src/vllm/config/scheduler.cpp:11`; `src/vllm/transformers_utils/hf_config.cpp:83`; limited flags incl. `max_num_seqs`/`max_num_batched_tokens` `examples/server/main.cpp:63,116,170` | `tests/vllm/test_scheduler_config.cpp:10`; `tests/vllm/test_hf_config.cpp:131,224,265`; `examples/CMakeLists.txt:34` | `planned: specs/config-surface.md` | `PARTIAL` | - | | `ENG-HF-MODEL-DOWNLOAD` | Fetch a checkpoint from HuggingFace so `--model` accepts a repository identifier and not a local path only. Two forms behind one flag, with the local path probed first: `org/repo` mirrors vLLM's full-snapshot download, and `org/repo:Q4_K_M` fetches one GGUF file, a form vLLM does not implement and which llama.cpp supplies as the secondary oracle. Mirrors vLLM's two-phase fetch (config JSON first, so a bad repository fails after 200 KB and not 60 GB) and its index-driven file selection, which reads `model.safetensors.index.json` and fetches the exact names in `weight_map` instead of every match for `*.safetensors`. Reference is resolved to a commit before any byte is fetched, so a moving `main` cannot change what a second run loads. Cache layout is HuggingFace's documented local cache, so a host holding a Python `huggingface_hub` cache gets a hit and `model_loader.cpp:279-303` reads it unchanged. Transport is the already-vendored cpp-httplib with OpenSSL, matching llama.cpp, which retired libcurl at `CMakeLists.txt:195` at stock tag `b10451`. NOT a verbatim port on one point: llama.cpp's `is_valid_oid` accepts any 40- or 64-character hexadecimal string, and on 17 August 2026 this project measured the tree API answering an unauthenticated caller on gated `Lightricks/LTX-2.5` with an `lfs.oid` of one character repeated 64 times, identical for all 14 large-file-storage files, so this row treats an untokenized `lfs.oid` as absent and proves completeness structurally instead | T1 | `vllm/model_executor/model_loader/weight_utils.py:345,349-357,459,472-490,493-496,506`; `vllm/model_executor/model_loader/default_loader.py:167-184`; `vllm/engine/arg_utils.py:839`; `vllm/config/model.py:183` at pin `5559679229`. Secondary oracle llama.cpp for the `:QUANT` form only, stock tag `b10451`, anchors re-verified by W1 at commit `10bf611e533d81f739128304991c5e133c6aebd8` | W1 and W2 only: `src/vllm/transformers_utils/hf_hub.cpp:1`, `src/vllm/transformers_utils/hf_cache.cpp:1`, reached from `src/vllm/entrypoints/model_loader.cpp:280`. No downloader, no `--model` grammar, no TLS option | `tests/vllm/transformers_utils/test_hf_cache.cpp:1`, `tests/vllm/transformers_utils/test_hf_hub.cpp:1` (in-process fake hub), `tests/vllm/entrypoints/test_dflash_draft_hf_cache.cpp:1` (the loader reach, red when the call site is deleted) | [hf-model-download.md](specs/hf-model-download.md) | `READY` | - | | `LOAD-LONGTAIL` | Sharded-state, tensorizer, RunAI, BitsAndBytes loaders | T3 | `vllm/model_executor/model_loader/__init__.py:33-65` | - | - | `planned: specs/loader-longtail.md` | `INVENTORIED` | - | diff --git a/.agents/specs/bpe-quadratic-merge.md b/.agents/specs/bpe-quadratic-merge.md index d1e4fe536..086670b87 100644 --- a/.agents/specs/bpe-quadratic-merge.md +++ b/.agents/specs/bpe-quadratic-merge.md @@ -17,9 +17,9 @@ them. recorded for this row under `## Git integration` in `.agents/developer-preferences.md`. The only recorded answer there is `SPEC-DFLASH2`'s, and AGENTS.md makes one pull request the default when no -answer is recorded. The shape is not cosmetic here: W1 lands the cost and growth -assertions of `## Tests to port` items 2 and 3 RED against unchanged code, so a -split would leave `main` red between W1 and W3. See `## Work breakdown`. +answer is recorded. The shape is not cosmetic here: W1 lands the cost assertion +of `## Tests to port` item 2 RED against unchanged code, so a split would leave +`main` red between W1 and W3. See `## Work breakdown`. ## Now @@ -104,8 +104,16 @@ produced them and the tables say which.** It exists to answer one question, whether the recorded constants reproduce. They do not, and the direction is the one contention predicts: `a` x 65,535 read 37,564.80 ms against Harness B's 24,358.86 ms, a 54% inflation at roughly - double the load. The growth ratios it re-derived are in `## Tests to port` - item 3 and they replace the figures a previous revision recorded there. + double the load. The growth ratios it re-derived are gone with the assertion + they were derived for; `## Tests to port` item 3 records that ruling. + +**None of those three harnesses was committed, and that is the root cause of +three rounds of irreproducible constants.** `tools/bench/bpe_encode_cost.cpp` +is the committed one, added with this revision. It is not any of A, B or C: it +is a fourth driver, it re-derived the English figures within the factor that +load alone explains, and its build and run commands are in `## Gates`. Every +figure below predates it and none can be re-derived from the tree at the +revision that recorded it. The **shape** is what all three harnesses agree on, and it is the result. Every absolute figure below is a contended minimum, and Harness C is the evidence that @@ -317,7 +325,7 @@ selects this version. The implementing row records the version it read. | `mod.rs:9`, `model.rs:19` identifier-keyed `MergeMap` | `MergeRanks = unordered_map` in `include/vllm/tokenizer/bpe.h` | an identifier-pair-keyed table built at load | | `src/vllm/tokenizer/bpe.cpp::MergeKey` | one `std::string` per probe | deleted; there is no key to build | | `model.rs:180-189` `MergeTokenOutOfVocabulary` | no rule; the failure appears per request in `src/vllm/tokenizer/tokenizer.cpp::EncodePlain` | refused at load, naming the missing token, on `FromHfJson` AND `FromGguf` | -| `model.rs:169-173`, `:186` `new_token = format!("{}{}", a, &b[prefix_len..])` | no counterpart: `src/vllm/tokenizer/tokenizer.cpp::MergeKey` concatenates `a` and `b` whole | still whole. `prefix_len` is `continuing_subword_prefix.len()`, and `src/vllm/tokenizer/tokenizer.cpp:624-631` already REFUSES a non-empty `continuing_subword_prefix` at load, so `prefix_len` is 0 on every checkpoint we accept and the term is inert for us. Port the concatenation without it, and do not silently drop the refusal that makes that legal | +| `model.rs:169-173`, `:186` `new_token = format!("{}{}", a, &b[prefix_len..])` | no counterpart: `src/vllm/tokenizer/bpe.cpp::MergeKey` concatenates `a` and `b` whole | still whole. `prefix_len` is `continuing_subword_prefix.len()`, and `src/vllm/tokenizer/tokenizer.cpp:624-631` already REFUSES a non-empty `continuing_subword_prefix` at load, so `prefix_len` is 0 on every checkpoint we accept and the term is inert for us. Port the concatenation without it, and do not silently drop the refusal that makes that legal | | `model.rs:382-460` `merge_word` | `src/vllm/tokenizer/tokenizer.cpp::EncodePlainSp`'s symbol builder | unchanged in behaviour, emitting identifiers | | `model.rs:475-496` word cache | absent | still absent, and out of scope | @@ -451,9 +459,9 @@ implementer does not inherit them by accident: heap entry, because it reuses the string-keyed `MergeRanks`. The implementation interns, so its constant is lower and its growth ratio is closer to `n log n` than the prototype's. **A bound derived from the - prototype's ratio is therefore not a bound for the implementation**, and - `## Tests to port` item 3 requires the implementing row to re-derive it on - its own code. + prototype's ratio is therefore not a bound for the implementation.** No such + bound is asserted anywhere in this row: `## Tests to port` item 3 records the + ruling that dropped it. ## Defence in depth @@ -473,8 +481,10 @@ assumed.** Two facts, both read at this base: 100 MB is what a request body may be. Extrapolate the measured `n^2.01` fit, taking `t = c n^2` with `c` from the -25,345.61 ms at 65,536 bytes. A 100 MB body of a single character class is then -on the order of `6e7` s, several hundred CPU-days of one core, for one request. +25,345.61 ms at 65,536 bytes — one session's reading at load average 4 to 12, +which is the term that has since been shown to move a figure like this one by +54%. A 100 MB body of a single character class is then on the order of `6e7` s, +several hundred CPU-days of one core, for one request. That extrapolation spans three decades beyond the largest measured point, so read it as an order of magnitude and not as a measurement. What is measured is that 64 KB already costs 25 s. Either way the conclusion does not depend on the @@ -496,8 +506,7 @@ it if a later row adds one: After the fix, 64 KB encodes in tens of milliseconds. The prototype takes 15.485 ms on the English case and 7.40 ms to 28.07 ms on the single-class cases. The guard therefore protects against a future regression rather than against -today's cost. File it as its own -issue when the row lands. +today's cost. File it as its own issue when the row lands. ## Dependencies @@ -536,9 +545,9 @@ before any algorithm changes. | W | Deliverable | Reviewable on its own because | |---|---|---| -| **W1** | The equivalence corpus and harness of `## Tests to port` item 1, plus the cost and growth assertions of items 2 and 3, all against UNCHANGED code | items 1 and 6 pass, items 2 and 3 are red. That red is the defect, recorded before anything moves | +| **W1** | The equivalence corpus and harness of `## Tests to port` item 1, plus the cost assertion of item 2, all against UNCHANGED code | items 1 and 6 pass, item 2 is red. That red is the defect, recorded before anything moves | | **W2** | The identifier-keyed merge table and the load-time refusal, with `BpeMerge` still doing the O(n^2) scan over identifiers | W1 stays exactly as green and as red as it was. Only the representation moved | -| **W3** | The heap merge, the linked symbol list, the comparator, the staleness checks | W1's items 2 and 3 turn green and nothing else changes | +| **W3** | The heap merge, the linked symbol list, the comparator, the staleness checks | W1's item 2 turns green and nothing else changes | | **W4** | The idle-host re-measure, the `## Outcome` section, `docs/STATUS.md` and `docs/BENCHMARKS.md` | the code is frozen; this is the record | W2 before W3 is deliberate. Interning and the heap are separable in the tree @@ -548,17 +557,18 @@ also the order the `new_id` staleness test needs: that comparison cannot be written against a string-keyed table. **These four waves are four commits in ONE pull request, and the split is a -review aid rather than a landing plan.** W1 lands items 2 and 3 red against +review aid rather than a landing plan.** W1 lands item 2 red against unchanged code, so between W1 and W3 the gate is failing by construction. A separate W1 pull request would put that red on `main` and leave every other row's gate unable to tell its own failure from this one. AGENTS.md's default with no recorded preference is one pull request, none of its three split cases applies here, because no helper needs a base-reachable spec, the scope is one function, and W1 through W3 all write product code. The commit order still -proves the spec came first. If the developer prefers a split, the reds must land +proves the spec came first. If the developer prefers a split, the red must land `SKIP`-ped with the reason and the owning wave named, and W3 must remove the -skip in the same change that turns them green. A permanently skipped assertion -is not an acceptable resting state. +skip in the same change that turns it green. A permanently skipped assertion +is not an acceptable resting state, which is also why item 3 is dropped outright +rather than deferred. ## Risks @@ -596,66 +606,86 @@ prove after. identifiers so the capture is reproducible. Where an entry cannot be captured from HF, the test records that entry as self-referential in its own comment rather than letting the whole corpus inherit an oracle it does not have. -2. **A cost bound that fails today, and it is the ROBUST of the two.** Encode - one pretoken of a stated size and assert an upper bound on wall time. State - the bound as an absolute figure with the margin argued, not as a ratio - against a same-run baseline. At 65,536 bytes of English through the Mistral - golden the two sides are 25,345.61 ms and, for the prototype, 15.485 ms, so a - bound placed anywhere in the middle of that has **three orders of magnitude** - of headroom. A runner would have to be a thousand times slower than this box - to cross it. That margin, not the fact that it is a single measurement, is - what makes this assertion survive a contended runner. -3. **A growth-shape assertion, which is the TIGHTER of the two and needs care.** - Encode at n and at 4n and assert the ratio sits below a bound that quadratic - growth cannot satisfy and the implemented algorithm can. **Use a step of at - least 4x, and take the minimum of k repetitions rather than one shot.** Both - requirements are measured below, and so is the limit of what they buy: - - - **A 2x step cannot separate the two algorithms at all under load.** Eight - independent min-of-8 sweeps of the prototype at 131,072 and 262,144 bytes, - load average 50 to 57 on the 20-core box, 2026-08-19, gave 3.328, 2.525, - 1.573, 4.414, 1.485, 3.193, 1.916 and 1.372. Quadratic growth predicts - 4.0, and **the observed maximum 4.414 is already above it**. A 2x - assertion therefore fires on the correct algorithm before it ever fires on - the defect, which is why the step has to be at least 4x. - - **A 4x step still separates, with about 1.8x of margin.** Eight - independent min-of-8 sweeps at 65,536 and 262,144 bytes, same binary, same - session, same load, gave 7.202, 8.920, 4.182, 2.800, 6.626, 4.198, 4.507 - and 8.586. Quadratic predicts 16 and the observed maximum is 8.920, so a - bound sits in (8.92, 16). - - **Min-of-k is necessary and it is not sufficient, and this spec no longer - claims otherwise.** A first revision specified a 2x single-shot ratio and - called it "the assertion that survives a slow runner"; a second replaced that - with a 4x min-of-8 sweep and certified a 9% spread and a (6.03, 16) band from - four sweeps. Neither figure reproduced. The eight-sweep re-derivation above, - taken at load 50 to 57 rather than the earlier revision's lighter and - unrecorded load, spreads 3.2x at both step sizes. The two halves of a ratio - are separate, independently preemptible measurement windows, and taking a - minimum inside each half does not make them move together. **The stability of - this assertion on a contended runner is therefore UNESTABLISHED**, and the - claim that a 4x step is "four times more stable" is withdrawn rather than - restated with a new number. - - The red side, re-derived on the current code at the same shape and load, - min-of-3, four sweeps: 8,000 to 32,000 bytes costs 12.680, 14.740, 16.468 and - 15.276, against the 16x `n^2` predicts. Placing that beside the prototype's - worst gives the real picture: **the window between the prototype's slowest - ratio, 8.920, and the current code's fastest, 12.680, is 1.42x.** That is the - whole budget this assertion has on a loaded host, and it is the reason item 2 - and not this item is the robust gate. - - Two obligations follow for the implementing row. It **re-derives the bound - from min-of-k sweeps on its own code**, because the interned implementation - removes the string key and the merged string the prototype still builds and - should sit nearer the ~4.6 that `n log n` predicts than the prototype's 2.8 - to 8.9. And it records the sweep, the `k`, the load average, and the host - beside the bound, so that the next reader can tell a real regression from a - busy box. A bound copied from the prototype is not a bound for the - implementation, and a bound derived on a loaded host is not a bound at all. - If the assertion cannot be made to hold with margin on the CI runner, it - lands `SKIP`-ped with this paragraph as the reason rather than tuned until it - passes. +2. **A cost bound that fails today. It is the ONLY timing assertion this row + ships, and item 3 records why.** Encode one pretoken of a stated size and + assert an upper bound on wall time. State the bound as an absolute figure + with the margin argued, not as a ratio against a same-run baseline. At 65,536 + bytes of English through the Mistral golden the two sides read 25,345.61 ms + and, for the prototype, 15.485 ms — one session's readings at load average 4 + to 12, tabulated in `## The SentencePiece family pays it on ordinary + English`, not constants. What survives the load is the SEPARATION: a bound + placed between them has **three orders of magnitude** of headroom, so a + runner would have to be a thousand times slower than that box to cross it. + + That margin is the whole argument, and it is cheap to check that load does + not eat it. Re-derived for this revision with the committed harness + `tools/bench/bpe_encode_cost.cpp` on the same box on 2026-08-19, min-of-3, + at load average 194/209/196 — an order of magnitude busier than the table + above — English through the Mistral golden read 12.802 ms at 1,000 bytes and + 606.424 ms at 8,000 bytes, against that table's 5.61 ms and 358.83 ms. So + roughly 1.7x on load alone, and the identifier counts (267 and 2,134) came + back exactly. **The identifiers reproduce and the times do not**, which is + the whole shape of this row in two numbers, and 1.7x against a thousandfold + margin is why item 2 and not item 3 is the gate. +3. **A growth-shape assertion. CONSIDERED AND REJECTED — do not implement it.** + The shape would have been: encode at n and at 4n, and assert the ratio sits + below a bound that quadratic growth cannot satisfy and the implemented + algorithm can. It is not in this row's scope, it is not deferred to the + implementing row, and it must not land `SKIP`-ped. **Operator ruling, + 2026-08-19.** It is recorded here at length so that nobody proposes it again + from the same reasoning that produced it twice. + + The ruling rests on a re-derivation by an independent reviewer, on the + 20-core box at load average 176 to 268: + + - **The red and green distributions OVERLAP at that load.** Ten sweeps of + 8,000 to 32,000 bytes on the CURRENT, defective code gave 4.654, 9.213, + 9.629, 11.989, 12.115, 13.206, 15.241, 16.017, 16.376 and 17.896. Five of + the ten fall below 12.680, which a previous revision of this document + certified as the defective code's MINIMUM, and one falls below 8.920, + which the same revision certified as the prototype's MAXIMUM. A bound + cannot be placed between two distributions that overlap. + - **The mechanism is this row's own diagnosis turned against its own gate.** + In the sweep that produced 4.654, the 8,000-byte leg read 4,383 ms against + the other nine sweeps' 650 to 1,878 ms. A preempted DENOMINATOR alone + dropped the defective code to a ratio the correct algorithm is supposed to + own. Min-of-k inside each half does not help: the two halves of a ratio are + separate, independently preemptible measurement windows, and a minimum + taken inside one of them says nothing about how the other one was + scheduled. + - **The margin argument for a 2x step was FALSIFIED.** A previous revision + argued that a 2x step is unusable because a correct algorithm was observed + at 4.414, above the 4.0 that quadratic predicts. That 4.414 was a + single-session artifact. Twenty-four min-of-8 sweeps of HF `tokenizers` + 0.22.2 — the reference implementation this design mirrors, at the version + [`../oracles/transformers.md`](../oracles/transformers.md) pins — ranged + 0.756 to 3.349 at load average 198 to 228, and never once exceeded 4.0. + The crossing does not reproduce, so it is recorded here as a negative + result and is not evidence for anything. + + **Item 2 carries the timing gate alone, and that is the right split rather + than a reduction.** Item 2's separation is about three orders of magnitude + and is immune to load in the only sense that matters: a 1.7x inflation, which + is what an order-of-magnitude load increase actually cost above, does not + approach it. The best separation this assertion was ever certified at was + 1.42x, a figure that did not itself reproduce, and the separation actually + observed above is none at all. A gate whose margin is smaller than the noise of the + host it runs on does not measure the code; it measures the box, and on a red + day it reports the defect as fixed. + + Three revisions of this document each certified a growth window, and no + reviewer reproduced any of them: a 2x single-shot ratio called "the assertion + that survives a slow runner"; a 4x min-of-8 sweep with a certified 9% spread + and a (6.03, 16) band; and an eight-sweep re-derivation with a 1.42x window. + The pattern is not carelessness in any one of them. It is that this quantity + is not stable on a shared host, and the correct response to a third + irreproducible certification is to stop certifying it. + + What the implementing row owes instead is nothing here: it implements item 2, + and if it wants a growth reading for its `## Outcome`, it takes one with + `tools/bench/bpe_encode_cost.cpp`, records the sweep, the `k`, the load + average and the host beside it, and states it as a session reading. A reading + is not a gate, and this item is the record of why this one cannot become one. 4. **The leftmost tie.** Extend the existing `test_bpe.cpp` case into one that distinguishes the two orders on a longer symbol list. 5. **A stale-entry case.** A merge sequence where a queued candidate is @@ -695,8 +725,10 @@ The CPU tier can prove, with no other host: - Token-identifier equivalence before and after, on both tokenizer families, over a corpus that contains the failing regime. - The tie-break order, the stale-entry handling, and the load-time refusal. -- That the cost bound and the growth shape are red before the change and green - after. +- That the cost bound of `## Tests to port` item 2 is red before the change + and green after. It is the row's only timing assertion; item 3 records the + growth-shape assertion that was considered and rejected, and no gate here + lands `SKIP`-ped. - Through `test_tokenizer_parity`, `test_tokenizer_parity_mistral`, `test_tokenizer_parity_deepseek` and `test_tokenizer_parity_gpt4o`, that all four already-gated families still produce their recorded identifiers. @@ -715,15 +747,60 @@ That needs the bf16 27B server on `dgx:gpu0` under an `rc` lease, and it is the row's closing evidence rather than its correctness gate. Run it only after the CPU gates are green. +**The harness is committed, and this is the recipe AGENTS.md §Gates requires.** +`tools/bench/bpe_encode_cost.cpp` times `Tokenizer::Encode` on one synthetic +input at stated sizes through a stated `tokenizer.json`, min-of-k, and prints +the 1/5/15-minute load average beside every row it emits. Its own header carries +the same two commands and states that its output is a session reading and never +a bound. Build it, from the repository root: + +```sh +g++ -O2 -std=c++20 -I include -I src -isystem third_party \ + tools/bench/bpe_encode_cost.cpp \ + src/vllm/tokenizer/bpe.cpp src/vllm/tokenizer/tokenizer.cpp \ + src/vllm/tokenizer/pretokenizer.cpp src/vllm/tokenizer/unicode_data.cpp \ + src/vllm/model_executor/model_loader/gguf_reader.cpp \ + src/vllm/model_executor/model_loader/read_only_file_mapping.cpp \ + -o /tmp/bpe_encode_cost +``` + +and run it against either committed golden: + +```sh +/tmp/bpe_encode_cost tests/parity/goldens/tokenizer_mistral/tokenizer.json \ + --case english --repeats 5 --sizes 1000,8000 +/tmp/bpe_encode_cost tests/parity/goldens/tokenizer_qwen36/tokenizer.json \ + --case a --case newline --repeats 5 --sizes 1000,4096 +``` + +It is registered as no test and CI runs it never. CI does COMPILE it, as the +never-linked OBJECT library `vllm_bpe_encode_cost` in `CMakeLists.txt`, so it +cannot rot behind a `Tokenizer::Encode` or `FromHfJson` signature change while +still being the only artifact these figures can be reproduced from. **It is +deliberately not a gate**: item 3 rules that a growth gate is not viable on a +shared runner, and the cost bound of item 2 belongs in the test suite rather +than in a bench driver. This harness exists so that a human or an agent can +re-derive a figure deliberately, on a host whose load they have looked at. + Required before any speed figure in this spec is accepted: an idle-host, same-binary A/B re-measure with the load average recorded. No figure above was taken on an idle host. Harness A ran at load average 130 to 190 and Harness B at 4 to 90, on a 20-core box with another session compiling and testing throughout. Harness B reports the minimum of a repetition set, which is the least contended estimate the box could give, and that is not the same thing as an idle -measurement. The two harnesses agree on the shape and on every absolute figure -to within about 25%, which is what makes the shape safe to reason from and the -constants not yet quotable. +measurement. + +**The harnesses agree on the SHAPE and they do not agree on the constants.** An +earlier revision of this section said the harnesses agreed "on every absolute +figure to within about 25%". That sentence named Harness A and B and was never +updated when Harness C was added, and Harness C contradicts it directly: the +same binary on the same input read 37,564.80 ms against Harness B's 24,358.86 ms +at roughly double the load, which is **54%**. A fourth reading taken for this +revision with the committed harness moved the 8,000-byte English figure by about +1.7x on load alone. So the agreement that makes this row's argument is the +exponent, the ratio against HF on identical output, and the direction — every +one of which is invariant across all four — and no absolute figure in this +document is quotable as a constant. ## Owed diff --git a/CMakeLists.txt b/CMakeLists.txt index aabdd3a54..c67dbc254 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -2399,6 +2399,34 @@ target_include_directories(vllm_music3_vocoder_conv_ab SYSTEM PRIVATE target_compile_features(vllm_music3_vocoder_conv_ab PRIVATE cxx_std_20) vllm_cpp_set_warnings(vllm_music3_vocoder_conv_ab) +# ── BIT-ROT GUARD: the BPE encode-cost harness (#1365) ─────────────────────── +# `tools/bench/bpe_encode_cost.cpp` is the artifact behind the timing figures in +# `.agents/specs/bpe-quadratic-merge.md`. It exists because three earlier +# harnesses for that row lived in prose only, so no reader could re-derive one +# figure and three rounds of certified constants each failed to reproduce. +# +# CI COMPILES IT AND RUNS NOTHING, and that is not an oversight. The row's own +# ruling is that a growth assertion over these figures is NOT viable on a shared +# runner, so there is nothing here for CI to decide; and one leg on the O(n^2) +# code costs tens of seconds of one core, which no test tier should spend. What +# CI does buy is that the file cannot rot behind a `Tokenizer::Encode` or +# `FromHfJson` signature change while still being the only artifact a reader can +# reproduce the row's numbers from — the #1246 failure, where the sole named +# harness for a measurement was compiled by nothing. +# +# An OBJECT library, like `vllm_music3_depth_stage_ab_*` above and unlike +# `vllm_music3_vocoder_conv_ab`: the recorded run recipe in the file's own header +# compiles the four tokenizer translation units directly with `g++` rather than +# linking `libvllm.a`, so a linked target here would guard nothing the compile +# does not and would only add an entry point nobody calls. +add_library(vllm_bpe_encode_cost OBJECT tools/bench/bpe_encode_cost.cpp) +target_include_directories(vllm_bpe_encode_cost PRIVATE + ${CMAKE_SOURCE_DIR}/include ${CMAKE_SOURCE_DIR}/src ${CMAKE_BINARY_DIR}/include) +target_include_directories(vllm_bpe_encode_cost SYSTEM PRIVATE + ${CMAKE_SOURCE_DIR}/third_party) +target_compile_features(vllm_bpe_encode_cost PRIVATE cxx_std_20) +vllm_cpp_set_warnings(vllm_bpe_encode_cost) + # ── Packaged shared/static libvllm (M3.5 Task 3, library-first packaging) ───── # The static `vllm` core lib above stays as-is (tests + examples link it and # reach internal C++ symbols). Here we ALSO package a SHARED libvllm.so that diff --git a/docs/USAGE.md b/docs/USAGE.md index 99289d35f..d38fcab7c 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -471,6 +471,29 @@ two edits back gives the GPT-4o row above, so the two share one scanner's character classes but stay separate patterns: they disagree on `don't` and on every digit run longer than one. +### Timing an encode on your own box + +`tools/bench/bpe_encode_cost.cpp` times `Tokenizer::Encode` on one synthetic +input, at the sizes you name, through a `tokenizer.json` you name. Use it when +you want to know what a prompt of some shape costs to tokenize here, or to +re-derive a figure somebody else recorded instead of trusting it. + +Nothing RUNS it: it is registered as no test and it is not a gate. Both halves +of that are deliberate — a growth ratio over these timings is not stable enough +to gate on a shared machine, and one leg on a long single-class input can cost +tens of seconds of one core. It IS compiled, as the never-linked OBJECT library +`vllm_bpe_encode_cost`, so it cannot rot behind a `Tokenizer::Encode` or +`FromHfJson` signature change while still being the artifact those figures are +reproducible from. Its header carries the exact `g++` and run lines; it builds +from the four tokenizer translation units directly and needs no `libvllm.a`. + +It prints one row per case and size, with the ids it produced and the +1/5/15-minute load average sampled around each row, under a banner saying the +output is a session reading and not a bound. Read it that way: on a 20-core box +the same input on the same binary has read 1.7x apart on load alone, while the +id counts came back identical. Quote a number from it only with its load beside +it, and take the minimum of several repetitions rather than one shot. + ### How much memory a Vulkan load needs On a unified-memory device (a DGX Spark) the Vulkan heap and system RAM are the diff --git a/scripts/check-agent-record.py b/scripts/check-agent-record.py index 4ffd2e018..aceee69de 100644 --- a/scripts/check-agent-record.py +++ b/scripts/check-agent-record.py @@ -620,12 +620,16 @@ # thing a token gate provably cannot see. It is also not a benchmark row: the encode # runs synchronously on the HTTP worker five lines before the only length check, so # `max_model_len` bounds none of it and `/tokenize` reaches it with no engine. -# MEASURED: 64 KB of ordinary English prose costs 25.3 s through the committed -# Mistral golden against HF `tokenizers` 0.22.2's 10.1 ms on the same file for -# byte-identical identifiers, and the fit over 1 KB to 64 KB has exponent 2.01. -# Those are contended minima on a 20-core box at load 4-90, not idle-host -# constants; the spec's `## Gates` owes the idle-host re-measure. The exponent, -# not the constant, is what makes this a row. +# MEASURED, and stated as the two SESSION-INVARIANT quantities only: over 1 KB to +# 64 KB of ordinary English prose the fit through the committed Mistral golden has +# exponent 2.01, and at 64 KB our cost is 2,507x HF `tokenizers` 0.22.2's on the +# same file for byte-identical identifiers. Both are ratios taken inside one +# session, so contention cancels. The ABSOLUTE milliseconds are deliberately not +# repeated here: they moved 54% between two runs of one binary on one input, so a +# constant copied into this comment would be a fourth place for a number nobody +# can reproduce to drift. They live in the spec's tables, each beside its own load +# average, and `## Gates` owes the idle-host re-measure. The exponent, not the +# constant, is what makes this a row. # `READY`, spec `specs/bpe-quadratic-merge.md`, issue #1365. # Bumped for a real new row, never to make a failing state transition pass. ENGINE_ROWS = 168 diff --git a/scripts/check-gate-commands.py b/scripts/check-gate-commands.py index 6ec12b9b3..0f13b7349 100755 --- a/scripts/check-gate-commands.py +++ b/scripts/check-gate-commands.py @@ -460,6 +460,16 @@ def audit() -> list[dict]: "SERVE-ASYNC-LLM", "SERVE-STREAM-USAGE", "TOOLS-STREAMING-PARSER", + # SPEC-BPE-QUADRATIC-MERGE (2026-08-19, issue #1365): GROWTH, re-pinned in + # the same change. The row was gated and command-free; its spec's `## Gates` + # now carries the `g++` build and the two run lines for + # `tools/bench/bpe_encode_cost.cpp`, the harness that replaces three + # prose-only ones, so the section names a command that can fail and the row + # joins this population. Named here so the growth is not silent: the harness + # itself is deliberately NOT a gate -- the row rules that a growth assertion + # over its figures is not viable on a shared runner -- and what became + # runnable is the RECIPE the section owes under AGENTS.md §Gates. + "SPEC-BPE-QUADRATIC-MERGE", }) diff --git a/tests/scripts/test_check_gate_commands.py b/tests/scripts/test_check_gate_commands.py index b8173b15d..4dfd59eb0 100644 --- a/tests/scripts/test_check_gate_commands.py +++ b/tests/scripts/test_check_gate_commands.py @@ -815,6 +815,27 @@ def test_hf_model_download_earns_its_runnable_baseline_entry(self): self.assertEqual(runnable - reduced, {"ENG-HF-MODEL-DOWNLOAD"}) self.assertEqual(runnable, set(gates.RUNNABLE_BASELINE)) + def test_bpe_quadratic_merge_earns_its_runnable_baseline_entry(self): + # SPEC-BPE-QUADRATIC-MERGE (#1365) was already gated and already carried + # a `## Gates` section; what changed is that the section now names the + # `g++` build and the run lines for `tools/bench/bpe_encode_cost.cpp`. + # So this is growth WITHIN the population, not arrival into it, and the + # baseline grows by one either way. + # + # Same shape and same reason as the two rows above: set equality holds + # for any membership and cannot say this row belongs, so this case says + # it by removing the entry and requiring the equality to go red. Without + # it, an entry added to quiet a red gate is indistinguishable from an + # entry added because a row gained a command. + verdicts = {r["id"]: r["verdict"] for r in gates.audit()} + self.assertEqual(verdicts.get("SPEC-BPE-QUADRATIC-MERGE"), "runnable") + reduced = set(gates.RUNNABLE_BASELINE) - {"SPEC-BPE-QUADRATIC-MERGE"} + self.assertNotEqual(reduced, set(gates.RUNNABLE_BASELINE)) + runnable = {r["id"] for r in gates.audit() if r["verdict"] == "runnable"} + self.assertNotEqual(runnable, reduced) + self.assertEqual(runnable - reduced, {"SPEC-BPE-QUADRATIC-MERGE"}) + self.assertEqual(runnable, set(gates.RUNNABLE_BASELINE)) + if __name__ == "__main__": unittest.main() diff --git a/tools/bench/bpe_encode_cost.cpp b/tools/bench/bpe_encode_cost.cpp new file mode 100644 index 000000000..705f88c0d --- /dev/null +++ b/tools/bench/bpe_encode_cost.cpp @@ -0,0 +1,243 @@ +// BPE encode-cost harness (#1365) — the artifact behind every timing figure in +// `.agents/specs/bpe-quadratic-merge.md`. +// +// WHY THIS FILE EXISTS. Three earlier harnesses for that spec lived in prose +// only. Nothing in the tree let a second reader re-derive one figure, and three +// successive rounds of certified constants each failed to reproduce for the +// next reviewer. AGENTS.md §Gates requires the exact build and run recipe to be +// recorded; a recipe nobody can execute is not one. So the harness is committed +// and the numbers are not. +// +// WHAT IT MEASURES. Wall time of `vllm::tok::Tokenizer::Encode` on one +// synthetic input, at the byte sizes given on the command line, through a +// `tokenizer.json` given on the command line. Encode is the whole path a +// request pays: added-token pre-pass, pretokenize, BPE merge, vocabulary +// lookup. It is deliberately NOT `BpeMerge` alone, because the request pays the +// whole thing and a component figure cannot be compared with HF's `encode`. +// +// WHAT ITS OUTPUT IS. A SESSION READING, never a bound. Every row carries the +// 1/5/15-minute load average sampled around it, because that is the term that +// moved these figures by 54% between two runs of one binary on one input. Do +// not copy a number out of this program into a spec, a matrix row, or a test +// bound without the load beside it, and do not treat any number it prints as +// reproducible on another box, another load, or another day. Take min-of-k and +// say what k was. +// +// DELIBERATELY NOT A REGISTERED TEST and deliberately NOT A GATE. A growth +// assertion over these figures was considered for this row and REJECTED — see +// `.agents/specs/bpe-quadratic-merge.md` §Tests to port item 3 — because the +// two halves of a ratio are independently preemptible on a shared runner and +// the red and green distributions overlap there. This program exists so that a +// human or an agent can re-derive a figure deliberately, on a host whose load +// they have looked at, not so that CI can decide anything from one. +// +// BUILD. The recipe is in a block comment because a `//` line may not end in a +// backslash under `-Werror=comment`. +/* + g++ -O2 -std=c++20 -I include -I src -isystem third_party \ + tools/bench/bpe_encode_cost.cpp \ + src/vllm/tokenizer/bpe.cpp \ + src/vllm/tokenizer/tokenizer.cpp \ + src/vllm/tokenizer/pretokenizer.cpp \ + src/vllm/tokenizer/unicode_data.cpp \ + src/vllm/model_executor/model_loader/gguf_reader.cpp \ + src/vllm/model_executor/model_loader/read_only_file_mapping.cpp \ + -o /tmp/bpe_encode_cost +*/ +// +// RUN. +/* + # the SentencePiece arm, whose pre_tokenizer declares "split": false, so the + # whole prompt is one word. English prose is the input because that is what + # is slow there. + /tmp/bpe_encode_cost tests/parity/goldens/tokenizer_mistral/tokenizer.json \ + --case english --repeats 5 --sizes 1000,8000 + + # the byte-level arm. A single-character run is one pretoken, which is the + # shape that reaches the same regime through the split regex. + /tmp/bpe_encode_cost tests/parity/goldens/tokenizer_qwen36/tokenizer.json \ + --case a --case space --case newline --repeats 5 --sizes 1000,8000 +*/ +// +// SIZES ARE NOT DEFAULTED UPWARD ON PURPOSE. On the O(n^2) code one 65,536-byte +// English leg costs tens of seconds of one core, so a size list is something +// the caller states, having decided to spend that. +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vllm/tokenizer/tokenizer.h" + +namespace { + +// The 1/5/15-minute load average, as one string, read fresh. Linux only; on a +// host without /proc/loadavg the harness says so rather than printing a figure +// with no load beside it, because an unlabelled figure is the defect this file +// exists to stop. +std::string LoadAverage() { + std::ifstream in("/proc/loadavg"); + if (!in) return "UNKNOWN"; + double one = 0, five = 0, fifteen = 0; + if (!(in >> one >> five >> fifteen)) return "UNKNOWN"; + char buf[64]; + std::snprintf(buf, sizeof(buf), "%.2f/%.2f/%.2f", one, five, fifteen); + return buf; +} + +// Repeats `unit` until the result is at least `bytes` long, then truncates on a +// UTF-8 boundary. Truncating mid-codepoint would feed the tokenizer invalid +// UTF-8 and measure the error path. +std::string Repeat(std::string_view unit, size_t bytes) { + std::string out; + out.reserve(bytes + unit.size()); + while (out.size() < bytes) out.append(unit); + size_t end = bytes; + while (end > 0 && (static_cast(out[end]) & 0xC0) == 0x80) --end; + out.resize(end); + return out; +} + +struct Case { + const char* name; + const char* unit; +}; + +// Each case is one input SHAPE. The first is ordinary English prose, which is +// what a user who pastes a document sends; the rest are single-class runs, +// which is what the split regex turns into one pretoken. +constexpr Case kCases[] = { + {"english", "The quick brown fox jumps over the lazy dog. "}, + {"a", "a"}, + {"space", " "}, + {"newline", "\n"}, + {"tilde", "~"}, + {"cjk", "\xe7\x9a\x84"}, +}; + +const Case* FindCase(std::string_view name) { + for (const Case& c : kCases) { + if (name == c.name) return &c; + } + return nullptr; +} + +std::vector ParseSizes(const char* spec) { + std::vector sizes; + const char* p = spec; + while (*p != '\0') { + char* end = nullptr; + const unsigned long long v = std::strtoull(p, &end, 10); + if (end == p || v == 0) { + std::fprintf(stderr, "bad --sizes near \"%s\"\n", p); + std::exit(2); + } + sizes.push_back(static_cast(v)); + p = end; + if (*p == ',') ++p; + } + return sizes; +} + +void Usage() { + std::fprintf(stderr, + "usage: bpe_encode_cost [--case NAME]... " + "[--sizes N,N,...] [--repeats K]\n" + "cases:"); + for (const Case& c : kCases) std::fprintf(stderr, " %s", c.name); + std::fprintf(stderr, "\n"); +} + +} // namespace + +int main(int argc, char** argv) { + if (argc < 2) { + Usage(); + return 2; + } + const std::string tokenizer_path = argv[1]; + std::vector cases; + std::vector sizes; + int repeats = 3; + for (int i = 2; i < argc; ++i) { + const std::string_view arg = argv[i]; + const bool has_value = i + 1 < argc; + if (arg == "--case" && has_value) { + const Case* c = FindCase(argv[++i]); + if (c == nullptr) { + Usage(); + return 2; + } + cases.push_back(c); + } else if (arg == "--sizes" && has_value) { + sizes = ParseSizes(argv[++i]); + } else if (arg == "--repeats" && has_value) { + repeats = std::atoi(argv[++i]); + if (repeats < 1) { + Usage(); + return 2; + } + } else { + Usage(); + return 2; + } + } + if (cases.empty()) cases.push_back(&kCases[0]); + if (sizes.empty()) sizes = {1000, 8000}; + + const std::string load_at_load = LoadAverage(); + const auto load_start = std::chrono::steady_clock::now(); + const vllm::tok::Tokenizer tokenizer = + vllm::tok::Tokenizer::FromHfJson(tokenizer_path); + const double load_ms = + std::chrono::duration( + std::chrono::steady_clock::now() - load_start) + .count(); + + // The banner is part of the output on purpose: a pasted table without it is + // a number with no provenance, which is exactly how the figures this file + // replaces got certified. + std::printf("# bpe_encode_cost -- SESSION READING, NOT A BOUND\n"); + std::printf("# tokenizer: %s\n", tokenizer_path.c_str()); + std::printf("# load at tokenizer load (1/5/15): %s, load took %.2f ms\n", + load_at_load.c_str(), load_ms); + std::printf("# repeats k=%d, reporting MIN over k; the min is the least\n", + repeats); + std::printf("# contended estimate available, not an idle-host figure\n"); + std::printf("case\tbytes\tids\tk\tmin_ms\tmax_ms\tload_before\tload_after\n"); + + for (const Case* c : cases) { + for (const size_t bytes : sizes) { + const std::string text = Repeat(c->unit, bytes); + const std::string before = LoadAverage(); + double min_ms = 0; + double max_ms = 0; + size_t ids = 0; + for (int r = 0; r < repeats; ++r) { + const auto t0 = std::chrono::steady_clock::now(); + const std::vector out = tokenizer.Encode(text); + const double ms = std::chrono::duration( + std::chrono::steady_clock::now() - t0) + .count(); + ids = out.size(); + if (r == 0 || ms < min_ms) min_ms = ms; + if (r == 0 || ms > max_ms) max_ms = ms; + } + const std::string after = LoadAverage(); + std::printf("%s\t%zu\t%zu\t%d\t%.3f\t%.3f\t%s\t%s\n", c->name, + text.size(), ids, repeats, min_ms, max_ms, before.c_str(), + after.c_str()); + std::fflush(stdout); + } + } + // The spread between min and max within one k is printed above for the same + // reason the load is: when it is large, the box moved under the measurement + // and nothing on that row is quotable. + return 0; +}