docs(e2e): spike scheduled server nightly

Inventory the pinned vLLM entrypoint and V1 e2e CI topology, define a fail-closed CPU-presubmit versus scheduled-GB10 split, and decompose the implementation into independently gateable leaves.

FOLLOWING_AGENTS_PROTOCOL

Assisted-by: Codex:GPT-5 [Codex]

Author: localai-org-maint-bot

.agents/coordination.md

@@ -777,6 +777,7 @@ record surfaces (`.agents/specs/dflash-spec-decode.md` §0/§6, roadmap ROAD-V1-
 checklist, `.agents/model-matrix.md` the DFlash spec row + checklist, ledger, state, this
 note, README, `docs/BENCHMARKS.md`). Did NOT touch any model/kernel/runtime source ⇒ SACRED
 gates byte-identical by construction (not re-run).
+| `CLAIM-SERVE-E2E-NIGHTLY-SPIKE-1` | `SERVE-E2E-NIGHTLY` (spike only) | Codex (GPT-5) | isolated worktree `/home/mudler/_git/vllm.cpp/.claude/worktrees/codex-serve-e2e-nightly-spike`, branch `codex/serve-e2e-nightly-spike`, base `upstream/main` `9721e848` | Owns exactly NEW `.agents/specs/server-e2e-nightly.md`; the `SERVE-E2E-NIGHTLY` row in `.agents/engine-matrix.md`; its snapshot cells in `.agents/roadmap_v1.md`, `README.md`, and `docs/BENCHMARKS.md`; this claim row; and one append-only `.agents/state.md` entry. **SPIKE/RECORDS ONLY:** inventories pinned-vLLM tests and CI, defines the CPU-CI vs scheduled-DGX split, hardware/data/dependency gates, and row-sized implementation leaves. Owns no source, test, workflow, model, benchmark, or GPU file; performs no GPU execution and makes no support/performance claim. | `SPIKE` | 2026-07-24 |
 
 **DFlash D0-redo + D1 note (2026-07-26, `CLAIM-DFLASH-D0D1`, DONE, NOT pushed — FULL
 SHA reported to caller).** UNBLOCKS + advances `SPEC-DFLASH` (rows `SPEC-DFLASH`;

.agents/engine-matrix.md

@@ -42,11 +42,11 @@ forensics: roadmap_v1.md and the parity ledger.
 | Sampling and generation | 15 | 0 | 2 | 0 | 0 | 7 | 0 | 1 | 5 |
 | Structured output and tools | 7 | 0 | 3 | 0 | 0 | 1 | 0 | 0 | 3 |
 | Speculative decoding | 10 | 0 | 1 | 0 | 0 | 3 | 0 | 3 | 2 |
-| Serving, API, CLI, library | 20 | 3 | 2 | 1 | 0 | 5 | 2 | 1 | 6 |
+| Serving, API, CLI, library | 20 | 3 | 2 | 2 | 0 | 5 | 2 | 1 | 5 |
 | LoRA and adapters | 2 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 1 |
 | Long context and attention | 10 | 0 | 0 | 0 | 1 | 5 | 1 | 0 | 3 |
 | Loading, tokenizer, config | 9 | 1 | 3 | 0 | 0 | 2 | 1 | 1 | 1 |
-| **Total** | **126** | **8** | **17** | **3** | **4** | **42** | **8** | **8** | **35** |
+| **Total** | **126** | **8** | **17** | **4** | **4** | **42** | **8** | **8** | **34** |
 
 ## Engine core and scheduling
 
@@ -186,7 +186,7 @@ claims it.
 | `SERVE-CPP-API` | Rich `LLM` and `AsyncLLM` C++ API | T1 | `vllm/entrypoints/llm.py:66,422`; `vllm/v1/engine/async_llm.py:70` | - | - | `planned: specs/cpp-api.md` | `INVENTORIED` | - |
 | `SERVE-CLI-BENCH` | Serve and latency/throughput/serve benchmark modes | T0 | `vllm/entrypoints/cli/serve.py:44`; `vllm/entrypoints/cli/benchmark/main.py:29` | separate binaries + explicit scheduler-capacity flags `examples/server/main.cpp:63,96,116,170`; `examples/bench/main.cpp:40,109`; `examples/bench/bench_core.h:96,468` | server help contract `examples/CMakeLists.txt:34`; benchmark `tests/examples/test_bench.cpp:15,48` | `planned: specs/cli-serve-bench.md` | `PARTIAL` | - |
 | `SERVE-GATE-ONLINE` | Same-corpus online correctness, TTFT/TPOT/ITL, throughput and peak-memory gate vs vLLM v0.25.0 | T0 | `vllm/benchmarks/serve.py:1,581-615`; [v0.25 audit](sync/2026-07-12-702f481.md); `tests/benchmarks/test_serve_cli.py:1` | Schema-v5 harness plus [trace controller](../include/vt/cuda/cuda_profiler_control.h#L13), [production component driver](../scripts/dgx-gdn-packed-component.sh), and fail-closed [component finalizer](../tools/bench/gdn_packed_component.py) | **BINDING `9ecd9d0`: 114/124** (async default ON; mem 4/4, c1 20/20, c2 20/20, c16 19/20, c4 & c32 18/20, c8 15/20; `benchmark_binding` refers here, superseding `3f256ab` 55/124 and `246a23c` 49/124, both retained immutable). Two-grid totality with `f0fb727` (111/124) is 115/124 effective parity vs vLLM 0.25.0 (27B). Async CLOSED the c16/c32 ITL tails (ours now BEATS vLLM: c16 p99 1.055, c32 p90 1.034/p99 1.078) and leaves a stable c8 `p99_itl` ~0.86 residual, ROOT-CAUSED (2026-07-18, `CLAIM-C8-P99-TAIL-1`, [spec](specs/c8-p99-itl-tail-2026-07-18.md)) as IRREDUCIBLE-AS-MIRRORED: our deterministic synchronous forward keeps co-admitted c8 requests in byte-identical lockstep where vLLM's async-future jitter de-phases them; the c16/c32 INVERSION proves this is the trailing edge of the per-step determinism that wins c16/c32 + throughput, not a capability gap (scheduler + async placeholder byte-identical, `tests/vllm/v1/test_scheduler_wave.cpp`, [tail spec](specs/tail-stall-analysis-2026-07-16.md)). Full grid + per-binding forensics: roadmap_v1.md + parity ledger; no packed speed credit | [online serving gate](specs/cuda-online-serving-gate.md); [merged GDN projections](specs/gdn-merged-input-projections.md); [packed decode](specs/gdn-packed-decode.md) | `ACTIVE` | CLAIM-SERVE-GATE-1 |
-| `SERVE-E2E-NIGHTLY` | Server conformance and real-model nightly suites for all release gates | T0 | `tests/entrypoints/openai/`; `tests/v1/e2e/`; `.buildkite/test-pipeline.yaml` | current unit/conformance tests only; no scheduled DGX suite | `tests/vllm/entrypoints/openai/test_conformance.cpp:1`; `tests/parity/test_qwen36_paged_engine.cpp:78`; `tests/parity/test_qwen27_paged_engine.cpp:110` | `planned: specs/server-e2e-nightly.md` | `INVENTORIED` | - |
+| `SERVE-E2E-NIGHTLY` | Server conformance and real-model nightly suites for all release gates | T0 | `tests/entrypoints/openai/`; `tests/v1/e2e/`; `.buildkite/test-pipeline.yaml` | current unit/conformance tests only; no scheduled DGX suite | `tests/vllm/entrypoints/openai/test_conformance.cpp:1`; `tests/parity/test_qwen36_paged_engine.cpp:78`; `tests/parity/test_qwen27_paged_engine.cpp:110` | `planned: specs/server-e2e-nightly.md` | `SPIKE` | `CLAIM-SERVE-E2E-NIGHTLY-SPIKE-1` |
 | `SERVE-CLI-CHAT` | Interactive chat and complete commands | T1 | `vllm/entrypoints/cli/main.py:18-34` has no direct chat/complete command at the pin; project extension | - | - | `planned: specs/cli-chat-complete.md` | `INVENTORIED` | - |
 | `SERVE-POOLING-ENDPOINTS` | Embeddings, pooling, score, rerank, classify HTTP surface (`/v1/embeddings`, `/pooling`, `/score`, `/rerank`, `/classify`). **SPIKED 2026-07-28 (`CLAIM-POOLING`):** the whole pooling task class is scoped in [pooling-task-class.md](specs/pooling-task-class.md) — endpoints depend on the pooling RUNNER (W3) which depends on the pooler OP (`ENG-POOLER-SEQ`, W1 landed) and a concrete pooling model. Endpoint protocol + handler port is the W4 brick | T2 | `vllm/entrypoints/pooling/embed/api_router.py:28`; `vllm/entrypoints/pooling/scoring/api_router.py:37,71`; `vllm/entrypoints/pooling/classify/api_router.py:26` | - | - | [pooling-task-class.md](specs/pooling-task-class.md) | `SPIKE` | `CLAIM-POOLING` |
 | `ENG-POOLER-SEQ` | The non-generative POOLER OP — turn hidden states into a pooled embedding/logit row instead of a sampled token. **W1 LANDED + CPU-GATED 2026-07-28 (`CLAIM-POOLING`, NOT pushed):** the sequence pooling methods `CLSPool`/`LastPool`/`MeanPool` (+ `GetSeqPoolingMethod` factory) over a packed `[num_tokens, hidden]` CPU buffer keyed by a minimal `PoolingCursor` (CLS/MEAN reject partial prefill, LAST allows it, MeanPool upcasts to float32) and the activation heads `PoolerIdentity`/`PoolerNormalize` (L2 `F.normalize`)/`PoolerMultiLabelClassify` (sigmoid)/`PoolerClassify` (sigmoid if `num_labels<2` else `softmax`). Unit-gated vs DOUBLE-PRECISION references, RED-first. RESIDUALS (named, spec §Work breakdown): the pooler HEADS composite + `DispatchPooler` task routing (W2), the pooling RUNNER + first concrete pooling model (W3), the endpoints (W4), tokwise `AllPool`/`StepPool` (W5) | T2 | `vllm/model_executor/layers/pooler/seqwise/methods.py:35-121`; `vllm/model_executor/layers/pooler/activations.py:106-158`; `vllm/v1/pool/metadata.py:13-71`; `vllm/config/pooler.py:16`; `tests/model_executor/layers/test_pooler_methods.py`, `tests/model_executor/layers/test_pooler_activations.py` | `include/vllm/model_executor/layers/pooler/methods.h` + `src/vllm/model_executor/layers/pooler/methods.cpp`; `include/vllm/model_executor/layers/pooler/activations.h` + `src/vllm/model_executor/layers/pooler/activations.cpp`; `include/vllm/model_executor/layers/pooler/pooling_metadata.h` | `tests/vllm/model_executor/layers/pooler/test_pooler.cpp` (CLS/LAST/MEAN + factory + Identity/Normalize/MultiLabelClassify/Classify, double-precision refs) | [pooling-task-class.md](specs/pooling-task-class.md) | `ACTIVE` | `CLAIM-POOLING` |

.agents/roadmap_v1.md

@@ -445,7 +445,7 @@ their area matrix.
 | A2 | GGUF real-file greedy parity on GPU (MVP loader gate) | ✅ **PASSED** — real APEX 35B GGUFs (Compact+Balanced, all supported k-quants), 28/28 assertions, 16/16 greedy token-exact vs same-file llama.cpp oracle, checkpoint-gated test+goldens merged (e2b93cf); remaining breadth: no 27B GGUF exists, NVFP4-type-40 dequant + i-quants deferred |
 | A3 | `test_ops_fused_chain` FMA-contraction fix | ✅ merged bf48edb (`-ffp-contract=off` host-wide) |
 | A4 | De-Python the build: vendor Triton AOT artifacts per-arch (`triton_aot_vendored/<arch>/` + MANIFEST; `VLLM_CPP_TRITON_REGEN` = maintainer-only Python) | ✅ **DONE** (54367cc..a432461; reproducibility hardening `09f1d23`) — `sm_121a` now has 48 generated C/H files + MANIFEST, including both bf16 `chunk_o` shapes; normal builds remain Python-free. Regen is explicit-target (`cuda:121:32`), line-info-disabled and byte-reproducible across source paths; the pure checker makes source/contract/artifact drift fatal and mutation-tests missing/extra/changed artifacts. A4 remains closed; fresh current-main CUDA/runtime/performance validation belongs to the two ACTIVE `CLAIM-PR3` kernel rows (evidence: porting-inventory §9). |
-| `SERVE-E2E-NIGHTLY` (formerly A5) | e2e suites per gates.md (server conformance nightly on dgx etc.) | ☐ next; leaf spike required |
+| `SERVE-E2E-NIGHTLY` (formerly A5) | e2e suites per gates.md (server conformance nightly on dgx etc.) | ◐ `SPIKE`; `CLAIM-SERVE-E2E-NIGHTLY-SPIKE-1` inventories the CPU-CI/scheduled-DGX split without running GPU work |
 | A6 | **User-facing surface closure** (audit `CLAIM-DOCS-COVERAGE`, 2026-07-24): make what SHIPPED reachable and truthfully documented. The audit found 2 class-D overstatements, 5 stale doc lines, 4 shipped-but-unreachable features and 153 undocumented env vars | 🚧 Tier 1 (falsehood repair) DONE in the audit commit. **Tier 2 DONE 2026-07-24 (`CLAIM-DOCS-T2-FIXES`)**, 3 of its 4 items: the D1 guard (a per-connector worker-half capability predicate; the disk connector is now REFUSED at engine construction on every device instead of silently shortcutting prefill for KV nothing writes — **its worker half remains unimplemented and is NOT claimed**), server `--tool-call-parser`/`--reasoning-parser` (40 tool names / 7 reasoning parsers now reachable over HTTP; defaults reproduce the old hardcode exactly), and server `--kv-transfer-config` mirroring vLLM's own CLI + JSON shape. Also landed from Tier 3/4: `docs/KV-OFFLOAD.md`, the README flag rows, the three missing Gemma model rows, and the `SERVE-C-ABI` 17→19 record repair. **Comprehensive refresh DONE 2026-07-25 (`CLAIM-DOCS-REFRESH`)**: **Tier 4** `docs/ENVIRONMENT.md` (26 user-facing env vars documented of 153 swept, kernel-internal tail on `scripts/env-doc-allowlist.txt`) + its CI checker `scripts/check-env-doc.py` (mutation-tested, fails on a new undocumented+unallowlisted production env var); the **Tier 3** README refresh to current git history (spec-decode `--speculative-config` now documented as shipped, the "not user-visible" class-B falsehood fixed, C ABI `VLLM_ABI_VERSION 6`, CLI spec-config flag); new usage guide `docs/SPECULATIVE-DECODING.md`. STILL OPEN: `vllm-cli` structured-output/sampling flags (Tier 2 item 4, code, not docs). Full table: [specs/docs-coverage-audit.md](specs/docs-coverage-audit.md) |
 
 ## B. Research tracks (complete)