refactor(serving): serve plain HTTP; TLS moves to tailscale serve - #374
Closed
paulclou wants to merge 23 commits into
Closed
refactor(serving): serve plain HTTP; TLS moves to tailscale serve#374paulclou wants to merge 23 commits into
paulclou wants to merge 23 commits into
Conversation
…ort tiers (#1) The pinned vLLM commit's vendored DeepSeek-V4 tokenizer collapses reasoning effort: `low` and `high` render the same (no) prefix and the model's true `max` tier is unreachable. This mod applies vLLM main's own fix -- copied verbatim -- to the effort path only, at container launch. - deepseek_v4_encoding.py: single REASONING_EFFORT_MAX constant -> the three-entry REASONING_EFFORT_PROMPTS dict + DEFAULT_REASONING_EFFORT, and the max-only emit condition -> the all-levels condition. Byte-identical to the model's own encoding_dsv4.py (low/high/max = 0/476/526-char prefixes). - deepseek_v4.py: one added elif so low/minimal/medium map to "low" instead of falling through to "high". No other line changes; the thinking on/off logic is left exactly as pinned. Anchor-matched single-occurrence replacement, aborts if an anchor is not found, ast/compile-validated, idempotent, atomic write. Additive: no tracked file changes. Activate per launch with --apply-mod mods/fix-dsv4-effort-levels. Verified live: high prompt tokens 5 -> 84 after apply; low/high/max reasoning depth separates behaviorally. Claude-Session: https://claude.ai/code/session_014fob2bXG5ve2F1ifgbD6Dx Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…etting) (#2) TLS is a single centralized setting rather than a per-recipe option: run-recipe.py injects --ssl-certfile/--ssl-keyfile into every model's `vllm serve` command (like ensure_ray_backend does for the Ray backend), so the whole fleet is served over HTTPS from one place instead of editing each recipe. Cert/key default to /root/.cache/vllm/certs/{cert,key}.pem inside the container (~/.cache/vllm is mounted by default); override with SSL_CERTFILE / SSL_KEYFILE, or set either to empty to serve plain HTTP. Beyond the usual reasons for TLS, this avoids a network-sandbox truncation class: sandboxes that filter egress through an HTTP proxy (e.g. fence) cap plain-HTTP requests with a fixed http.Client timeout that also interrupts reading the streamed body, silently cutting long-streaming completions (high reasoning-effort turns that generate past the cap). HTTPS takes the sandbox's raw CONNECT tunnel, which is uncapped. - ensure_ssl(): idempotent, no-op when the path is empty or not a serve command - README: "Serving over HTTPS (TLS)" section Claude-Session: https://claude.ai/code/session_01Krp8GpaQJARWGHeWitJRKD Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Redundant: the current prebuilt b12x image ships vLLM main's three-tier reasoning_effort fix natively (deepseek_v4_encoding.py REASONING_EFFORT_PROMPTS + deepseek_v4.py low/minimal/medium -> low). The mod also no longer applies (anchor refactored away) and aborted launch. Effort tiers work natively upstream.
* feat(recipes): add Qwen3.8-27B-NVFP4 MTP and DSpark recipes Dense hybrid Qwen3.8-27B in NVFP4, targeting 262144-token context on one or two DGX Sparks. Two speculative configurations, matching the two regimes it was measured in: the in-checkpoint MTP head for concurrent serving, and the separate 5-layer DSpark drafter for single-stream and agent-loop work. Both pin --no-enable-flashinfer-autotune, which is required on multi-node hosts without GPUDirect; autotune deadlocks engine startup there with no error (vllm-project/vllm#52291). * docs(recipes): cite the source gist and record its deviations Both recipes derive from a published measurement writeup. Name it, and state where they knowingly depart from its launch block so the differences are not mistaken for transcription errors: TP=2 and the inverted flashinfer-autotune flag both come from the gist's own two-Spark appendix, and the DSpark recipe's 0.80 memory utilization is the value its FP8 appendix measured that drafter at rather than the 0.82 its launch block implies.
…easurement (#6) * feat(recipes): add Qwen3.8-27B-NVFP4 1M YaRN variant Static YaRN factor 4.0 to 1,048,576 tokens, kept as its own recipe because the scaling applies to every request regardless of length and costs short-context quality. Measured on 2x DGX Spark: KV pool 4,989,721 tokens (4.76x concurrency at full length), engine init 143 s, and a 323,078-token prompt completed in 278.7 s, confirming YaRN works past the native 262,144 window. Prefill is the limit at ~1,159-1,393 tok/s, so a cold 1M prompt takes 15-18 minutes. * chore(recipes): follow the upstream autotune default and admit video Drop --no-enable-flashinfer-autotune from the Qwen3.8 recipes: the vllm#52291 deadlock cannot occur on the image this cluster runs (no set_autotune_process_group; flashinfer_autotune() takes the per-rank path at world_size > 1), autotune completed twice at ~11 s, and decode was unchanged. Five of six upstream multi-node recipes leave it at the default. Raise the video limit to 1: measured cost in KV pool is 3,043,521 vs 3,041,754 tokens, i.e. nothing, at the checkpoint's shipped video_preprocessor_config. * docs(recipes): record how reasoning effort actually resolves The served checkpoint ships Qwen's chat template plus Unsloth's superset patches, so effort needs no flag: vLLM forwards the request's reasoning_effort into chat_template_kwargs, and the template accepts xhigh (default), medium and low, plus high via Unsloth's alias. vLLM's API also accepts minimal and max, which the template rejects with a 400. Also note that the sampling inherited from generation_config.json is the thinking-mode preset, so clients disabling thinking must send the card's instruct preset themselves, and point at the 1M variant for longer contexts. * feat(recipes): serve the 1M variant as qwen3.8-27b-1m The two native-context recipes share one served name because a client cannot tell them apart and should not have to - MTP and DSpark take byte-identical requests. The 1M variant is different: it quadruples the context window, which a client must be configured for, and pi cannot discover that on its own because it does not enumerate /v1/models. Only one service binds port 8000, so the recipe that is not running 404s on its name, which is a better failure than a mid-conversation 400 about context length.
* test(tools): add a serving benchmark harness The Qwen3.8 tuning so far rested on a shell script that timed one prompt at one concurrency level. That measures a single point of a wide distribution: on the same server, single-stream decode ranges from 26.0 tok/s on prose to 31.9 tok/s on repetitive output, because speculative acceptance is strongly content-dependent. This harness measures four prompt families across three suites -- single stream decode, concurrency scaling, and prefill rate from TTFT -- and reads speculative acceptance from /metrics deltas instead of inferring it. It also reports per-draft-position acceptance, which is what actually decides num_speculative_tokens: position i's rate is the probability the i-th drafted token survives, and depth stops paying once that rate no longer covers the drafter pass producing it. On prose the rate decays 0.77 / 0.64 / 0.46, while on repetitive output it stays flat at 0.94 -- so the optimal k is not one number for all traffic. Token counts come from the server's own usage accounting, and concurrency slots use distinct prompts so prefix caching cannot inflate the result. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(tools): stop the prefill suite measuring a warm prefix cache The prefill prompts were built from range(target // 2) over a shared filler, which made each size a strict prefix of the next. The larger sizes therefore scored partly against blocks the smaller sizes had already populated -- the server reported a 19.3% prefix cache hit rate on what was supposed to be a cold prefill, and the 224k point came out at 1386 tok/s. With a per-size salt the same measurement gives 845 tok/s at 421k tokens, so the contaminated curve was overstating long-context prefill by a wide margin and understating how steeply the rate decays with length. Also stop the streaming reader hanging after the server is done: it now finishes on the terminal chunk rather than waiting for [DONE] on a kept-alive socket, sends Connection: close, and takes a bounded timeout. A 421k-token prefill previously returned 200 OK server-side while the client sat on the socket for 15 minutes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(tools): add a quality regression probe for config changes Throughput tuning on these recipes changes attention backends, KV headroom and RoPE scaling, and those changes differ in whether they are supposed to be numerically neutral. A backend swap and extra KV headroom should leave greedy output identical; static YaRN should not, since it rescales RoPE for every request including short ones. Without a way to tell those apart, a throughput win and a silent quality regression look the same. This records the exact completion and a digest per prompt for a fixed greedy set, so two configs can be diffed directly, and grades the probes that have checkable answers so a divergence can be called better or worse rather than merely different. Baseline on the 1M YaRN config in non-thinking mode: 11/11 graded. The set is a tripwire rather than a benchmark -- it will not rank models, only tell you whether a config change moved the output. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(tools): package the outstanding tuning experiments as one runnable pass Five questions remain open on the Qwen3.8 recipes -- validating the shipped Triton numbers with the current harness, one confirming point for the speculation-depth cost model, Triton vs FlashInfer on the 1M recipe, max_num_seqs, and the KV headroom vLLM offers in its boot log. Each needs a different serve config, so each costs a full engine restart and the cluster to itself. This drives the whole matrix unattended: generate the variant recipe, boot, wait for /v1/models, run bench-serving.py and quality-probe.py, stop, repeat. Every variant is measured for output quality as well as throughput, so a speed-up and a silent regression cannot be confused. Three things it is deliberately careful about. It refuses to start if a server is already answering, rather than benchmarking someone else's config. It aborts a variant whose transform changed nothing, since running the unmodified recipe under the variant's name would quietly produce a wrong answer. And it records the backend and cudagraph lines the engine actually logged, because vLLM downgrades cudagraph_mode silently when the backend cannot support it -- which is the whole reason this work exists. Variant transforms are shell functions rather than sed strings: the first version passed a sed program containing \n through echo, which interpreted it and produced a silently truncated command. Not yet run end to end -- cluster access was down when it was written, though every tool it calls has been run individually against this cluster and all four recipe transforms are verified to produce the intended edit and dry-run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(tools): measure decode at context depth Prefill rate says how long you wait to start; it says nothing about what a session feels like once it is deep into a large context. That is the number deciding whether a large context window is usable or merely available. The depth suite prefills to a given context and times only the generation that follows, subtracting measured TTFT so prefill is excluded rather than amortised in. It records acceptance per depth, which is what makes the result legible when speculation behaviour changes with context length. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(tools): keep campaign depth targets inside each recipe's max_model_len The depth suite ran a fixed 175000 target for every variant. The filler runs ~1.64 prompt tokens per unit of depth, so that lands near 287k tokens -- the point on the 1M recipes, where crossing the native 262,144 window is exactly what is being measured, but past max_model_len on the 262,144-token ones, where the request is rejected and the variant wastes a full boot for nothing. Depth targets are now per recipe family, and 1m-nospec is ordered ahead of 1m-triton since the speculation cliff is the more valuable open question. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(tools): fail a variant when the engine dies instead of waiting it out The boot loop only ever polled for success, so a launch that crashed in its first two minutes was indistinguishable from a slow one and cost the whole BOOT_TIMEOUT. That happened on the first real run: a worker hit a transient httpx.ConnectTimeout reaching huggingface_hub during init, died at 14:36, and the script kept printing dots until its 25-minute ceiling. It now also watches the serve log for fatal signatures -- WorkerProc failure, engine-core init failure, OOM, any traceback -- and for the launcher exiting without a server, and reports the matched exception rather than the last 20 lines, which by then are shutdown noise. Silence is not success: a poll loop that greps only for the happy path stays quiet through exactly the failures worth catching early. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(tools): add DSpark k=7 and k=3 campaign variants The cost model that made k=3 win for the MTP head should not transfer to DSpark, and the reason is structural rather than empirical: DSpark drafts a block of num_speculative_tokens tokens in one parallel pass (vllm/v1/worker/gpu/spec_decode/dspark/speculator.py) rather than running k sequential drafter passes, and the checkpoint is trained at block_size 7. A 3-wide pass should cost nearly what a 7-wide one does on a bandwidth-bound device while capping the reward at three accepted tokens. Both k values are included because the existing 34.5 tok/s figure for DSpark k=7 came from the older single-prompt script and is not comparable to anything this harness produces. Measuring the pair together keeps the comparison honest. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
FlashInfer declares AttentionCGSupport.UNIFORM_SINGLE_TOKEN_DECODE, so with speculative decoding every multi-token verify step fell out of full CUDA graphs and vLLM downgraded cudagraph_mode to PIECEWISE. ~2.7-2.8 accepted tokens per step were returning only ~1.35x over no speculation. TRITON_ATTN declares AttentionCGSupport.ALWAYS. Setting it on both the model and the drafter takes single-stream decode from 26.8-27.7 to 39.4-39.6 tok/s on the MTP recipe and from 22.6-26.8 to 34.5 tok/s on DSpark, measured TP=2 over 256 greedy tokens on the two-Spark cluster. Both flags are required: with only --attention-backend the drafter re-selects FlashInfer and the warning persists. The 1M recipe is deliberately left on FlashInfer pending a prefill measurement; long-context prefill is its binding cost, not decode. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ion cliff (#10) * docs(recipes): record why FlashInfer cannot be fixed, and correct 1M prefill Two comment corrections, no behaviour change. The Triton comment recorded the symptom but not the cause, which invites someone to try flipping a FlashInfer flag. The reason is architectural: get_cudagraph_support() returns UNIFORM_BATCH only when can_use_trtllm_attention() holds, and supports_trtllm_attention() ends in is_device_capability_family(100). GB10 is sm_121, family 120, so it is always False on this hardware. The 1M recipe's prefill figures were measured with prompts that were nested prefixes of one another, so they scored against a warm prefix cache -- the server reported a 19.3% hit rate on a supposedly cold prefill. Re-measured with per-size salted prompts the rate is 2,156 / 1,623 / 845 tok/s at 25k / 101k / 421k tokens, which puts a cold 1M prefill near 30 minutes rather than the 15-18 previously claimed. Also records that max_num_seqs 4, not memory, is what caps aggregate throughput on the 1M recipe: 22.6 / 37.1 / 82.6 / 81.7 tok/s at concurrency 1 / 2 / 4 / 8. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(recipes): record the speculation cliff past the native window Decode on the 1M recipe barely cares about context depth and then falls off a cliff sitting exactly on the native 262,144 boundary: 23.0 tok/s at 254,652 tokens against 8.5 at 287,519. It is not bandwidth, which predicts about 1.5x against the observed 3x. Past that point the MTP head accepts zero drafted tokens (per-position 0.000/0.000/0.000 against 0.93/0.68/0.51 at 254k), so every step pays for three sequential drafter passes and keeps none of them. YaRN carries the base model past its trained positions; it does not carry the draft head. That is pointed for this recipe, whose whole purpose is contexts beyond 262,144 -- exactly where its speculation is pure overhead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(recipes): record why DSpark keeps k=7, and where it loses to MTP Measured both k values on this recipe with TRITON_ATTN, same harness, so the numbers are comparable to the MTP recipe for the first time. The prediction going in was that k=3 would regress, because DSpark drafts a block of k tokens in one parallel pass rather than k sequential ones, so a narrower block should cost nearly the same while capping the reward. That was wrong for ordinary traffic: k=3 comes out marginally ahead on prose, chat and code, and 21% ahead at concurrency 8. The reason is visible in the per-position rates -- at k=7 on prose they run 0.49 / 0.27 / 0.11 / 0.05 / 0.02 / 0.00 / 0.00, so positions 3-6 contribute about 0.06 accepted tokens between them. There was almost nothing there to cap. k=7 is kept regardless, because it is the only configuration in which this recipe beats the MTP one at all: 96.6 against 51.1 tok/s on highly predictable output, where every draft position accepts at ~0.95. Dropping to k=3 trades that for a few percent on prompts the MTP recipe already serves better. Also records the comparison this recipe had never been given honestly. Against qwen3.8-27b-nvfp4.yaml it loses on prose (25.7 vs 35.2), chat (25.7 vs 35.2), code (34.9 vs 38.5), concurrency-8 aggregate (128 vs 202) and KV pool (3.02M vs 4.72M). The separate drafter is weaker on real text -- acceptance length 1.93 against 2.66 on prose. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(recipes): stop claiming a cause for the acceptance cliff The header explained zero acceptance past 262,144 by saying YaRN reaches the base model but not the draft head. That is wrong. For method=mtp vLLM sets draft_model_config = target_model_config, so the head *is* the target checkpoint and shares a single rope config; the boot log shows one rope_parameters block. The base model is not extrapolating there either -- with factor 4.0, position 287k maps to ~72k in the original basis. What is measured stands: 0 accepted out of 246 drafted at 287,519 tokens, against acceptance 3.12 at 254,652, with the boundary on 262,144 exactly. The head still runs and emits tokens; the target rejects all of them. Exactly zero rather than merely low suggests something categorical -- invalid position indexing, a clamped rotary cache -- rather than quality decay, but that is flagged as a hypothesis rather than asserted. A recipe comment that states a confident wrong cause is worse than one that records the effect and says the cause is unknown, because the next person debugging this will believe it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* perf(recipes): retune the 1M recipe against measured behaviour Three changes, each measured on the two-Spark cluster, all in the same direction: the recipe was tuned for a worst case that its traffic never hits. Drop speculative decoding entirely. Past the native 262,144-token window the MTP head accepts nothing at all -- acceptance 1.00, per-position 0.000/0.000/0.000, against 3.12 at 254,652 tokens. YaRN rescales positions for the base model but the draft head does not survive it, so every step past 262k paid for three drafter passes and kept none. Decode at 287,519 tokens: 8.5 tok/s with speculation, 14.4 without. Below the boundary speculation is still worth +28% to +47%, so this is deliberate: the recipe exists to serve contexts past the native window, and short-context work belongs on the base recipe, which does it at 35-39 tok/s against this recipe's 25. max_num_seqs 4 -> 8. The cap assumed every request is a full 1M tokens; real traffic is nowhere near that, and it was costing most of the machine. Measured aggregate at concurrency 8: 81.7 -> 141.6 tok/s, KV pool unchanged. gpu_memory_utilization 0.82 -> 0.88, which vLLM's own boot log had been offering. 5,453,845 KV tokens against 4,997,535, no OOM, decode unchanged. Quality probe 11/11 on every variant measured. The header's old rationale for max_num_seqs 4 and for keeping the MTP head is rewritten rather than deleted, since both were reasonable priors that measurement contradicted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(recipes): back the 1M backend choice with a same-recipe A/B The header justified keeping FlashInfer with slopes taken from two different recipes. Re-measured on this recipe alone, FlashInfer against TRITON_ATTN: decode 25.6 vs 35.9 at 1.5k, 22.8 vs 12.2 at 213k, 8.5 vs 3.6 at 287k, and prefill 1,623 vs 1,054 tok/s at 101k. Step time grows ~0.83 ms per 1k tokens of context on Triton against ~0.11 on FlashInfer, crossing near 42k. Also records that the A/B ran with speculation enabled, so it compares the backends rather than the configuration this recipe now ships. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(recipes): drop the disproven YaRN explanation from the nospec rationale Same correction as the layer below: the draft head shares the target's config under method=mtp, so YaRN reaching one and not the other cannot be the cause. The decision to drop speculation rests on the measurement (8.5 vs 14.4 tok/s at 287,519 tokens), which is unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test(tools): cover qwen3.8 image input with vision probes quality-probe.py exercised only the text path, so a config change could regress image input without the tripwire noticing. Add three probes against an inline 128x128 PNG - glyph, colour, and both-plus-instruction-following - so a partial failure is legible, and --no-vision to skip them when the served model is text-only. The fixture is inline base64 to keep this a single stdlib-only file with no binary to lose, and small because preprocessor_config.json floors image cost at a 65536-px^2 shortest_edge: 96px, 128px and 160px sources all bill 103 prompt tokens. Verified against vllm@qwen3.8-27b-nvfp4 at TP=2: 14/14 graded, vision-1..3 passing. * docs(recipes): record qwen3.8 image input as verified The recipe has passed --limit-mm-per-prompt since it was added, and nothing in the repo said whether that flag did anything. It does: the checkpoint is Qwen3_5ForConditionalGeneration with a 27-layer vision tower the NVFP4 quantization leaves in bf16. Record what the probe measured on 2026-08-21 - PNG and WebP score identically, image cost is floored at a 65536-px^2 shortest_edge - plus the two client-side failures that do not name their own cause: the endpoint is HTTPS with a bearer key (plain HTTP answers "Empty reply from server"), and thinking is on by default, so a small max_tokens returns content: null with finish_reason "length". The -1m and -dspark headers already defer to this file, so this is documented once rather than three times.
* test(recipes): pin a 1M ds4f variant and add window/peak instrumentation The base ds4f recipe leaves max_model_len at "auto", which vLLM resolves with _auto_fit_max_model_len(). Six boots produced six different windows and never 1M: 995328, 987136, 940288, 926720, 921856, 833792. The advertised context was whatever memory happened to be free at boot, swinging ~20% across restarts, with only one INFO line in the log as evidence. deepseek-v4-flash-0731-1m.yaml pins max_model_len to 1048576 (the model's real ceiling: YaRN factor 16 over original_max_position_embeddings 65536) so a shortfall becomes a boot failure rather than a shorter window, and buys the headroom with gpu_memory_utilization 0.85 -> 0.90 and max_num_seqs 8 -> 4. The header records the fallback ladder and why there is no KV-dtype lever. tools/assert-window.sh checks both halves of the claim: no auto-fit reduction in the boot log, and /v1/models advertising exactly the expected length. bench-serving.py gains a "peak" prompt family carrying the exact prompt the third-party DGX Spark writeups use for their headline tok/s, so our peak is comparable to a published number instead of merely similar. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(tools): add ds4f campaign variants and a systemd guard Five DeepSeek-V4-Flash variants: eugr's recipe untouched (which serves a different window on every boot), the pinned 1M variant, k=3, no speculation, and one that mimics the third-party config -- attention backend on AUTO with the V2 model runner off, which is what their older build forces -- so the cost of that stack choice is measured rather than assumed. The guard is the expensive lesson from today. Stopping the containers is not stopping the cluster: the vllm@ template carries Restart=on-failure with RestartSec=30, so `launch-cluster.sh stop` against a live unit reports success and systemd relaunches the old recipe half a minute later. The next launch then either measures the wrong model or dies on "Address already in use". The campaign now refuses to start while any vllm@ unit is active and prints the stop commands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(recipes): record the measured 1M ds4f behaviour and close the NVFP4 question Two boots both advertised the full 1,048,576 window (KV pool 1,779,018 then 1,977,584 tokens), so the pin holds and the memory variance that used to move the context length now lands in spare headroom. The headline is that decode does not decay with depth: 48.4 tok/s at 928,582 tokens with acceptance still above 3, i.e. the DSpark drafter keeps earning its passes 14x past the native 65,536-token YaRN window. That is the opposite of the Qwen3.8 MTP cliff, so the "drop speculation for long context" conclusion must not be copied across model families. What 1M actually costs is TTFT - 16.3 minutes to first token. Records one unexplained result rather than hiding it: a 449,406-token prompt returns completion_tokens=1 with finish_reason "stop" and empty text, reproducible across two filler salts at temp 0, while 224,706 and 928,582 answer normally. The server logs 200 OK, so a client sees a silent empty answer. Also records that --kv-cache-dtype nvfp4_ds_mla dies in ~2s at VllmConfig.validate_nvfp4_kv_cache_with_mla, which rejects the whole nvfp4 family for every MLA model - so the 432 B/token layout is unreachable here and fp8_ds_mla's 584 B is the floor. Third-party recipes advertising "1M NVFP4 KV" delete that validator and pin the page back to fp8's envelope. Noted so nobody spends another boot rediscovering it, together with the open upstream hang report vllm#40969 that this recipe's cudagraph mode is exposed to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(recipes): keep official's max_num_seqs and record the util ladder Measurement, not preference: util 0.90 with official's max_num_seqs 8 boots with the full 1,048,576 window and 19.22 GiB of KV (1.72x), so halving max_num_seqs was never needed. The deviation from the official DGX Spark profile is now two values -- gpu_memory_utilization 0.85 -> 0.90, and the window pinned to the 1,048,576 official intends implicitly by omitting the flag. Why 0.85 cannot work here, with the arithmetic: 0.85 x 121.69 GiB = 103.4 GiB budget, minus 81.34 GiB weights, minus ~11.5 GiB activations/non-torch/graphs/ scratch, leaves ~10.6 GiB against the 11.04 GiB a full-length request needs. Two boots measured 10.33 and 10.61. And why 0.95 cannot: vLLM gates on free memory at startup and sees ~111 GiB while the loader stages weights, against 115.61 wanted. Dropping the page cache does not lift that gate -- with mods/drop-caches running on both nodes and Cached at 1.4 GiB it still read 111.04 -- because the missing memory is the load itself, not stale cache. The ceiling is ~0.912, so 0.90 sits near the top of the band with ~1.5 GiB slack. Corrects two stale claims in the previous header. The "empty completion at 449k" was not a depth limit: /v1/completions gives this thinking model no chat template, so its first token is a near-tie between EOS, '</think>' and continuation, and greedy sometimes picks EOS for zero tokens with 200 OK. The same prompt at ~1.04M answered correctly, and forcing past that token retrieved the planted needle at both depths. Deep context is usable, not merely allocatable -- which the previous header failed to say because every depth-suite prompt told the model to ignore its context. Benchmark on the chat endpoint. Also records that max_model_len: auto is a documented vLLM tuning option rather than eugr's invention, so the base recipe's varying window is a deliberate trade: never advertise what you cannot serve. This recipe takes the other one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(recipes): give the 1M ds4f variant its own served-model-name Both ds4f recipes bind port 8000 and the base one serves deepseek-ai/DeepSeek-V4-Flash-0731, so without a distinct id a client cannot tell a 524288-token window from a 1,048,576-token one -- and the difference is not cosmetic, since a cold full prefill here is 16.3 minutes. Same convention as qwen3.8-27b-1m, whose entry documents the same reasoning: 1M gets its own name because it is a batch capability you choose deliberately. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * perf(recipes): drop VLLM_MOE_SKIP_PADDING to match the official recipe Measured A/B on this recipe rather than inherited from the base one. Both sides verified by reading /proc/<serving-pid>/environ and resolving vllm.envs, since the recipe exports env inside the container's launch script where docker exec printenv shows nothing either way: off resolved False, unset resolved True. off (base recipe's "0") 11/11 graded 57.11 tok/s mean over 15 runs unset (library default) 11/11 graded 58.11 tok/s The 1.8% sits inside the per-family spread, which moved in opposite directions on the same change - chat 42-62 -> 39-50 while repetitive 54-69 -> 71-78 - so it is not a throughput result. The two digest differences were 'Yes' vs 'yes' and '53,59,61' vs '53, 59, 61' on equally graded probes, i.e. formatting, not the semantic drift a misrouted-expert bug would produce. So the flag buys nothing measurable here and the default is what the official DGX Spark profile ships. eugr's reason for setting it on the base recipe is undocumented and may still hold there; this changes only this recipe. The header now lists every remaining deviation with its reason. The one that cannot be closed is gpu_memory_utilization: official's 0.85 leaves 10.33-10.61 GiB of KV against the 11.04 GiB a 1,048,576-token request needs, so the official command cannot boot at its own implied window on 2x DGX Spark. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(tools): stop one failed probe from discarding a whole quality run
quality-probe.py raised out of main() on the first HTTP error, so the JSON was
never written and --compare had nothing to read even though every other probe
had answered. Hit it tonight on DeepSeek-V4-Flash: the model has no vision tower
and its recipe passes no --limit-mm-per-prompt, so the two vision probes 400 and
took three complete 12-probe passes with them. The scores printed to stdout and
looked fine, which is what made it silent -- the run was gone, not the result.
--no-vision exists for exactly this case and should be used for a text-only
served model, but a usage error should not cost a run either. Each probe now
records its error and continues, and compare() reports a probe that carries an
error instead of indexing past a missing digest.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs(recipes): record the ds4f tuning sweep, including what not to retry
Four levers measured against a matched same-session baseline at --repeats 10
(50 runs per config), with t quoted because the noisy families carry sd ~13
tok/s and |t| < 2 is unresolved rather than a result. None was adopted; the
recipe is unchanged. The point of the block is that each of these is a config
someone would otherwise spend a night rediscovering.
num_speculative_tokens 7 (the card's value): rejected, prose -22%, and the
mechanism measured rather than argued - draft positions 5 and 6 accept at
0.012/0.005 on prose against 0.81/0.65 on code.
--async-scheduling: no effect, t=0.09, and exactly 0.0% on the one
low-variance family.
--enable-chunked-prefill: decode -3.4% (unresolved) for +2.6 GiB of KV, and
vllm#40969 did not reproduce across 150+ requests.
max_num_batched_tokens 16384: will not boot - KV need 11.04 -> 17.85 GiB while
availability falls to 12.09.
max_num_batched_tokens 4096: the largest lever found, pool 1.62x -> 2.76x, at
-7.7% decode and chat -23.1% (t=-2.56).
Both tradeoffs buy KV headroom this recipe does not need at a 1M window, so they
are documented as levers for a different goal rather than applied.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(tools): let bench-serving target one prompt family and repeat prefill
Two additions aimed at cost, not features. Decode spread is acceptance-driven,
so prose and chat carry sd ~13 tok/s (~30% of mean) while peak carries ~2:
resolving a 3% effect needs n~60 on a noisy family and n~20 on a tight one.
--families points the run at the tight family, which answers the same question
about 8x cheaper than brute-forcing all five to n=60 (~25 min versus ~3 h).
--prefill-repeats gives the prefill suite the repeats the decode suite already
had; every prefill figure so far has been single-shot, which is thin for the
short sizes where the measured chunked-prefill gain was largest. Each repeat
gets its own salt, or repeat two onward would read a warm prefix cache and score
a cache hit as a fast prefill.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* perf(recipes): enable chunked prefill on the 1M ds4f recipe
The five-family sweep said decode -3.4% (t=-0.74) and I recorded it as a
tradeoff not worth taking. That was the noise floor talking. Re-measured on the
peak family alone at n=20 -- sd ~3.9 against prose/chat's ~13, so SE 1.23 --
decode is +0.54 tok/s (+0.8%, t=+0.44). No cost, and the sign flipped.
Prefill with 3 repeats per size (it had never had repeats):
8k 1761 +- 1023 -> 1942 +- 749 unresolved, sd is 58% of the mean
32k 1898 +- 375 -> 2328 +- 138 +23%, spreads barely overlap
100k 2137 +- 101 -> 2171 +- 97 flat
32k is agent-context range, so that is the gain worth having. Concurrency at a
full window 1.63x -> 1.73x. Quality 11/11 with one benign digest divergence.
Two first-pass numbers did NOT survive repeats and are corrected in the header:
"+83% prefill at 6.9k" and "+2.6 GiB KV" were both single-sample artifacts. The
lesson is in the tooling now -- prefill takes --prefill-repeats, and --families
makes the tight-variance instrument cheap enough to use by default.
Left in view: the 8k point is unresolved, and vllm#40969 is an open hang report
against this exact flag, not reproduced here across 150+ requests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* revert(recipes): drop --enable-chunked-prefill, it was already the default
3749007 enabled a flag that was already on. SchedulerConfig declares
enable_chunked_prefill = True in this build, and every boot logs
enable_chunked_prefill=True with or without the flag -- including the two boots
I labelled "baseline" and "chunked" and then compared. That A/B was on-vs-on.
So the numbers I adopted it for are not effects: +23% prefill at 32k, +2.6 GiB
KV, +0.8% decode. They are variance between identical configs, and as such they
are the most useful measurement in this whole sweep, because they size the noise
floor of the box: KV available 16.9-19.6 GiB, prefill at 32k 1898-2328 tok/s,
peak-family decode differing 0.8% at n=20 each. Anything smaller than that is
unmeasurable here regardless of what flag is set.
--async-scheduling is redundant for the same reason: async_scheduling defaults to
None, which resolves to True unless something is incompatible, and dspark is
explicitly compatible. Both "tests" were measuring nothing, and both produced
plausible numbers -- one of which flipped sign between runs of the same config.
The serving config is now identical to what is on main. The header records both
redundancies so the next person checks a default before benchmarking it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs(recipes): mark the async-scheduling claim as inferred, not observed
The chunked-prefill redundancy was read off the engine config dump
(enable_chunked_prefill=True with or without the flag). The async one was not:
that dump does not print async_scheduling, so its resolved value was never
observed. The source chain is strong - default None, five disable branches that
each log a warning none of which fired, dspark explicitly compatible, mp
executor supporting it - but it is inference, and the previous commit stated both
with equal confidence. Records how to settle it: boot --no-async-scheduling and
diff.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
PREBUILT_RUNNER_IMAGE and PREBUILT_B12X_RUNNER_IMAGE become default-preserving env overrides, so a specific nightly can be deployed through the normal pull-tag-copy flow while :latest is unverified. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(recipes): adopt fleet scheduling for the ds4f 1M recipe * docs(recipes): move the ds4f 1M measurement ledger to docs/tuning * docs(dev-guide): record the recipe-header and tuning-ledger convention * docs(tuning): record the fleet-scheduling post-change measurement * feat(recipes): adopt max_num_batched_tokens 4096 for multi-orchestrator fleets
…rn, local-repo builds Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT
… baseline (#26) Delete the fork's three qwen3.8-27b recipes (MTP, DSpark, 1M YaRN) in favor of tuning from upstream's qwen3.8-27b-nvfp4-dflash2.yaml, per the fork-minimal policy. Their measured findings move to docs/tuning/qwen3.8-27b.md along with the campaign plan for tuning the upstream defaults; the retired recipes remain recoverable at 9794369. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Cluster grew from 2 to 4 Sparks over a CRS812-DDQ switch. Adopt the standing TP=4 plan from the tuning ledger: batched tokens back to the official 8192, utilization back to the official 0.85 (pending first-boot verification), max_num_seqs 12 -> 16 for fleet headroom. TP=2 history and adoption rationale recorded in docs/tuning/deepseek-v4-flash-0731-1m.md. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
… backports (#24) * feat(recipes): add GLM-5.3-Flash NVFP4 for the 4x cluster (ported from kingjones30, MIT) * feat(recipes): add GLM-5.3-Flash MM+Ray variant replicating MiaAI; document DFlash2 status * feat(recipes): GLM EXL3+DFlash2 recipe * docs: in-file provenance for vendored docker/glm53 files chat_template.jinja, glm53-flash_SM121.py, and the reference Dockerfile are byte-identical to MiaAI-Lab/GLM-5.3-Flash-NVFP4-Dual-DGX-Spark files/ @aed98a13ca75 (MIT) — verified by diff against the source repo. Cite that in each file so attribution travels with the file, matching Dockerfile.glm53-mm and the mods README. The jinja comment is {#- -#} trimmed; rendered output confirmed byte-identical on chat, multimodal, and tool-call payloads. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(recipes): set VLLM_MLA_NOPE_PAD_ROPE=1 for glm-5.3-flash-nvfp4 The fix-glm53-nope-rope-pad mod is env-gated; without the env var the NoPE pad stays dormant and TP4 boot fails in memory profiling with 'pe_dim must be 64 for fp8_ds_mla' (concat_and_cache_mla). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * fix(recipes): disable FlashInfer autotune for glm-5.3-flash-nvfp4 at TP4 At TP=4 the autotune warmup wedges indefinitely in a sparse-MLA op (py-spy: identical stack across samples, all ranks 96% GPU, zero log/cache writes for 95+ min). Autotuner averages tactic timings across ranks via collectives; TP2 deployments never hit this. Heuristic fallback is the supported path and MoE uses marlin anyway. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * feat(recipes): switch glm-5.3-flash-nvfp4 to tonyd2wild's TP4-validated image Stock day-0 image + runtime mods wedges at TP=4: uninitialized indexer top-k memory (torch.empty) sends the sparse-MLA gather through garbage indices - all ranks spin at 96% GPU in the first dummy run. tonyd2wild's sm121-v11 stack fixes this (patch_v7: full(-1) init + bounds guard) plus 8 more day-0 bugs, validated at TP4 on 4x GB10. Drop the runtime mod, its env gate, and the autotune workaround - all superseded by the patched base. DFlash2 overlay is inert under MTP; ND-licensed drafter not loaded. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * fix(recipes): plain fp8 KV for glm-5.3-flash-nvfp4 on tonyd2wild image His image fixes NoPE for the plain-fp8 MLA path (his launcher: --kv-cache-dtype fp8); it has no fp8_ds_mla pad, so fp8_ds_mla dies with 'pe_dim must be 64'. fp8_ds_mla belongs to the stock-image+mod stack. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * feat(recipes): port tonyd2wild's TP4 launcher verbatim for glm-5.3-flash-nvfp4 Replace the kingjones30-derived flag set with tonyd2wild's validated TP4 config (launch-glm53-tp4-24g.sh): DFlash2 k=7, fp8_e4m3 KV + 24 GiB/rank kv-cache-memory pin, 1M context, max_num_seqs 6, block 2304, batched-tokens 8192, enforce-eager, thinking off. Only eugr-required deviations: node wiring left to launch-cluster, drafter via HF cache id, one-shot drop-caches instead of his continuous flusher, no docker memory pin. Mixing his image with the old flag set died at boot (fp8_ds_mla has no NoPE pad in his image). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * fix(recipes): serve glm-5.3-flash-exl3 by local snapshot path Glm5NextProcessor.from_pretrained open()s processor_config.json relative to the model argument, so the HF id form crashes at APIServer init even though the checkpoint ships the file. Resolve the mounted HF-cache snapshot dir at launch and serve that path; served-model-name keeps the API name stable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * fix(recipes): cap glm-5.3-flash-exl3 max_model_len at 32K for GB10 persistent_topk (GLM sparse indexer) cannot launch on GB10: the FilteredTopK fallback needs 128KB smem/block, GB10 has 101KB. At 900K max_model_len boot profiling engages the sparse path and EngineCore dies (total_ctas=77 > 48). Same wall crashed NVFP4 serving at >32K prompts (ctas=90). 32K stays under the sparse-activation threshold. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * fix(recipes): language-model-only + drop-caches for glm-5.3-flash-exl3 Boot survived to API-server MM warmup, then the host OOM-killer took the serve process (exit 137): the multimodal front-end costs ~15.7 GiB on the API node, unaffordable on 121 GiB UMA at 0.87 util. Serve text-only (also matches the NVFP4 recipe for benchmark parity), drop the now-moot mm-limit/skip-mm-profiling flags, and add the drop-caches mod for UMA page-cache hygiene at launch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * feat(mods): route sm120 off persistent_topk to fix >32K context on GB10 persistent_topk cannot launch on GB10 (48-CTA persistent budget vs 77-90 required; FilteredTopK fallback needs 128KB smem vs 101KB available), so any request past sparse-indexer activation (~32K tokens) killed the engine on both the NVFP4 and EXL3 stacks. The cooperative_topk selector already excludes capability family 120; persistent_topk's selector forgot to - add the same exclusion in both indexer variants so sm120 takes the generic top_k_per_row_decode kernel in the existing else branch. Fixes #27. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * feat(recipes): raise glm-5.3-flash-exl3 to 128K context via sm120 topk mod With mods/fix-glm53-topk-sm120 the sparse indexer routes to the generic topk kernel on GB10, unblocking >32K requests. 131072 fits the ~346K token KV pool at TP4; Mia's nominal 900000 cannot (pool must hold one max-length sequence). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * feat(recipes): apply sm120 topk mod to glm-5.3-flash-nvfp4 Unlocks >32K requests on GB10 (validated on the EXL3 stack: RULER perfect at 64K). The 1M window and 3.9M-token KV pool become genuinely usable. Refs #27. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * docs: measured TP4 results, GB10 32K topk ceiling + fix, bug ledger Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * docs: EXL3 RULER 120K results Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * feat(recipes): pin 24 GiB/rank KV for glm-5.3-flash-exl3 Align KV allocation policy with the NVFP4 recipe (tonyd2wild's explicit kv-cache-memory pin) for side-by-side capacity comparability. Mia's unpinned 0.87-util approach leaves KV as a profiling residue - ~8 GiB/rank at TP4 because vLLM conservatively accounts the EXL3 stack's workspace/graph peak. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * docs: NVFP4 131K perfect RULER, VT artifact cleared, quant verdict Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * Revert "feat(recipes): pin 24 GiB/rank KV for glm-5.3-flash-exl3" This reverts commit 60e1b22. * docs: KV-pool gap is policy+runtime footprint, not page cache (flusher-verified) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * feat(mods): vendor Mia's 2026-08-30 overlay backports for both recipes Two fixes from MiaAI-Lab's updated EXL3 kit, applied at launch to both stacks (same day-0 vLLM base 487ecf187, anchors verified in both images): - kpool tail slot-map clamp: the one-block circular scratch cache is indexed past its row for pos >= block_size; long generations (~2K tokens) write through garbage block ids and silently corrupt other layers' indexer state. Mechanism credited to vcruz305. - XGrammar termination backports (vLLM #52805/#53046): stop feeding a terminated matcher in multi-token speculative batches. Benchmarks published earlier are unaffected (max 256-token generations); real serving with long outputs needs the clamp. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * docs: measured unpinned NVFP4 KV pool (7.19M tokens, 1.85x pin) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * docs: unpinned KV A/B verdict - pin is load-bearing (131K retrieval 9/16 unpinned vs 25/25 pinned) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * feat(recipes): glm-5.3-flash-serve - evidence-based production config Every axis deliberate, decision log in the header: pinned KV (A/B proved load-bearing), 12 seats, 262K honest window, prefix caching on, thinking on with deepseek_r1 parser per model-card guidance (needs reasoning smoke test), explicit probabilistic spec sampling, eager kept pending a graphs A/B. Benchmark-reference recipes unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * fix(recipes): serve NVFP4 by pinned snapshot path Upstream README-only pushes move refs/main to sparse revisions whenever any online tool (lm-eval, benchy tokenizer loads) touches the repo, and boots then crash on missing processor_config.json. Pin the serve target to the validated 357b45cc snapshot, same pattern as the exl3 recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * feat(recipes): glm-5.3-flash-base - official vLLM recipe imported for GB10 Line-for-line port of the recipes.vllm.ai GLM-5.3-Flash multi-node TP profile: kv fp8, MTP-5, glm47/glm45 parsers, everything else at vLLM defaults. Only GB10-mandatory deviations: arm64 (tony) image, marlin MoE, topk-sm120 mod, snapshot-path pin, launcher-owned node wiring. Checkpoint kept LibertAIDAI per decision. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * fix(recipes): base recipe needs --trust-remote-code for LibertAIDAI checkpoint Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * fix(recipes): glm-5.3-flash-serve KV to auto per Paul - no pin Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * fix(recipes): base recipe strictly minimal - drop HF_HUB_OFFLINE Not in the official recipe; the snapshot-path pin already covers the refs-churn problem it targeted. Remaining deviations: container/arm64, marlin, topk mod, snapshot path, trust-remote-code - all mandatory. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * chore(recipes): delete superseded serve draft - base recipe is the ground-up config Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * feat(recipes): glm-5.3-flash-serve = official vLLM recipe replicated Contract: replicate recipes.vllm.ai GLM-5.3-Flash exactly, except image (aarch64), checkpoint (LibertAIDAI), and TP x nodes (4x1 Sparks). Two boot/correctness-mandatory additions only: trust-remote-code and marlin MoE. topk mod rides the launcher. Supersedes glm-5.3-flash-base.yaml - one canonical ground-up recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * feat(recipes): serve max_num_seqs 16 per Paul Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * feat(recipes): serve max_num_batched_tokens 8192 per Paul Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * fix(recipes): serve needs HF_HUB_OFFLINE=1 (Glm5NextProcessor path resolution) Approved by Paul (option B). tonyd2wild's launcher sets the same. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * fix(recipes): serve gpu_memory_utilization 0.85 - GB10 UMA cannot boot vLLM's 0.92 default Approved by Paul. Engine ValueError: 111.95 GiB desired vs 110.19 free. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * fix(recipes,docs): unpin NVFP4 KV everywhere - A/B proved no benefit Protocol-matched retest: unpinned scores 1.00/0.96/1.00 @131k (n=25), identical to pinned, with 1.85x pool. Earlier degradation claim was a generation-budget measurement artifact; docs verdict corrected. Per Paul. Sole remaining deviation from tony's launcher in the bench recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * feat(recipes): serve thinking on by default + served-model-name alias Per Paul: enable_thinking true via default-chat-template-kwargs (server default, client-overridable) and glm-5.3-flash-nvfp4 as the API alias. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * fix(recipes): serve reasoning parser deepseek_r1 + clear_thinking glm45 on vLLM silently discards reasoning (probed: 310 think tokens generated, reasoning_content empty, no tag leakage). deepseek_r1 per LibertAI card. clear_thinking=true replaces the no-op enable_thinking kwarg. Approved by Paul; doc-priority policy recorded. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * feat(recipes): serve DFlash2 k=7 spec decode per Paul 52.5 vs ~38 tok/s measured over MTP-5. License-flagged: drafter is CC BY-NC-ND; MTP config is the commercial fallback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * feat(recipes): serve explicit max-model-len 1048576 per Paul Matches checkpoint max_position_embeddings; identical to the implicit resolution (no behavior change), stated for boot-failure-on-shortfall per the ds4f-1m policy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * feat(recipes): serve block-size 2304 per Paul Aligned to sparse-indexer 128-token tiles (18x128); both validated GB10 deployments use it. Graphs stay on (default) per Paul. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * feat(recipes): serve adds mia-backports mod per Paul kpool long-generation corruption clamp + XGrammar termination fixes - directly relevant to agent serving (long outputs, structured tool calls). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * serve recipe: slim header per AGENTS.md, soften glm45 note, state prefix caching - Move contract/tuning/thinking background from the recipe header to docs/GLM53_FLASH.md 'The serve recipe' section (headers stay minimal). - glm45 parser comment softened: our early probe read the wrong response field; glm45 is not disproven, deepseek_r1 kept per LibertAI card. - --enable-prefix-caching stated explicitly (same as vLLM default; makes the assumption visible). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * docs: serve-config formal campaign results + ruler_vt harness-artifact diagnosis Decode 65.2 tok/s (+24% vs bench ref, graphs on), GSM8K 90/88, RULER retrieval perfect 8K-131K. ruler_vt depth dip (0.88@64K, 0.736@131K) diagnosed with logged samples as max_gen_toks truncation: think tokens count against the budget and the reasoning parser strips them from content; same prompts score 1.00 at budget 1024. New eval rule: budget >= 1024 for RULER against reasoning-parser endpoints. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * docs: propagate session learnings to ds4f ledger and runbook - ds4f ledger: max_gen_toks-counts-think-tokens trap (found on the GLM serve config; applies to any reasoning-parser endpoint, this recipe included) - budget >= 1024 for evals, check --log_samples before believing a low generative score. - runbook: progress-verification guidance for long operations (server token counters over GPU%, known false-stall phases). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * docs: MM inference validated on the serve config Two smoke probes against the live endpoint (tony image, default MM front-end): shapes/colors/rendered-text image and a bar chart, both described exactly in 1.5-2.8 s. No extra flags needed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * restructure: single launchable GLM recipe; reference ports to docs/ - recipes/glm-5.3-flash-serve.yaml -> recipes/glm-5.3-flash-nvfp4.yaml (the production config becomes the sole GLM recipe in recipes/) - tony-flags bench, EXL3, and MM ports -> docs/reference-recipes/ with a README stating each one's validation status (MM: never booted here) - docs/GLM53_FLASH.md: paths updated; stale 'EXL3 primary' and 'choose MM variant for images' claims corrected (superseded by TP=4 measurements and native-MM validation) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * cleanup: drop orphaned mod and never-built MM build context - mods/fix-glm53-nope-rope-pad: referenced by no recipe (served the retired kingjones fp8_ds_mla port; Mia's EXL3 overlay carries its own NoPE handling) - Dockerfile.glm53-mm + docker/glm53/: build context for the MM variant that was never built or booted here; vendored verbatim from MiaAI-Lab @aed98a13ca75 (MIT) and recoverable from there or history - MM reference yaml: build_args commented, header marks it reference-only and points at the upstream build context - docs updated; tests/test_recipes.sh 65 pass (1 pre-existing minimax failure also present on main, untouched by this branch) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * docs: record chat-template identity chain (zai == LibertAI == Mia's baked copy) Verified byte-identical 2026-08-30; Mia's --chat-template flag is redundancy, not customization, and the check is a tokenizer-side counter-datapoint on the parked LibertAI corruption claim. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * docs(mods): provenance README for mia-backports (bit-exact verified, upstream commits pinned) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * docs: reasoning-effort matrix measured - default==max, high fails hard problems 12-cell probe (low/high/max/unset x 3 difficulty tiers, greedy): effort scales with difficulty (max thinks 9-17x high on hard problems and is the only level that solves them); server default stays max; temp-0 not bit-deterministic on this stack (~5-50% token variance). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT * docs: card-derived findings - glm45 confirmed broken on stock vLLM, corruption risk downgraded LibertAI card mandates deepseek_r1 on vLLM (glm45 silently discards the reply against this template's prompt-side <think>; tony's build likely patched); tony's enable_thinking:false is a no-op (template has no such kwarg); checkpoint generation_config = zai's eval sampling (1.0/0.95); the card's input_scale/fused-MoE bug likely explains tony's corruption report - marlin sidesteps it, downgrading the parked checkpoint risk. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yxmAjkbgUojWBZmwpq1wT --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(recipes): add glm-5.3-flash-uncensored-nvfp4 Copy of glm-5.3-flash-nvfp4 with a single deviation: the checkpoint (orcarouter/GLM-5.3-Flash-Uncensored-NVFP4, served as glm-5.3-flash-uncensored-nvfp4). Image, mods, and tuning identical to the measured base recipe. * docs(glm53): validate uncensored recipe + extract shared benchmarking guide Full measurement campaign for glm-5.3-flash-uncensored-nvfp4 on 4x Spark TP=4, protocol-matched to the base NVFP4 recipe. Quality is at or above the stock checkpoint and the abliteration is confirmed effective: - GSM8K 200q 5-shot: 91.0/91.0 (base 89.0/87.5) - RULER niah_single_2/niah_multikey_1/ruler_vt: 1.0/1.0/1.0 at 8K, 64K, and 131K (topk mod validated >32K on this checkpoint) - Refusal probe: 0/8 benign over-refusal, 0/8 sensitive-tier refusal - MM vision smoke: shapes/text image and bar chart described exactly - Speed at parity; DFlash2 acceptance 0.751/3.24 tok-step (lossless via rejection sampling). k stays at 7: DFlash2 drafts in one parallel pass, so k is not worth sweeping down. Adds tools/refusal-probe.py (abliteration tripwire) and a new docs/BENCHMARKING.md holding the model-agnostic harness mechanics (TLS pfSense CA, llama-benchy/lm-eval invocations, lm-eval landmines, spec-decode acceptance) that GLM53_FLASH.md now references instead of repeating. Records the deliberate model-card env-var deviations (VLLM_SSM_CONV_STATE_LAYOUT / VLLM_KV_CACHE_LAYOUT) and why they are no-ops on this build. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Remove the fleet-wide --ssl-certfile/--ssl-keyfile injection added in PR #2. The certificate it depended on came from the retired pfSense CA for the dead .leviathan suffix, and vLLM only reads cert files at startup, so it could never renew. TLS now terminates in tailscale serve on the head node, which holds a publicly trusted certificate for the node's tailnet name and renews it itself. Docs and examples move to the https://<node>.<tailnet>.ts.net:8000 URL and drop the pfSense trust-bundle instructions. BREAKING CHANGE: the launcher no longer serves HTTPS by default. Clients that used https://<host>:8000 directly must switch to the tailnet URL, or pass --ssl-certfile/--ssl-keyfile after -- on the command line. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Author
|
Opened against the wrong repository by mistake; this belongs on the paulclou fork. Closing, apologies for the noise. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Problem. PR #2 made the launcher inject
--ssl-certfile/--ssl-keyfileinto everyvllm servecommand. The certificate it relied on came from the retired pfSense CA, was issued for the dead.leviathansuffix, and vLLM only reads cert files at startup, so nothing could renew it. Every client saw a warning or needed a custom trust bundle.Solution. Remove the injection and the
SSL_CERTFILE/SSL_KEYFILEsettings. TLS terminates intailscale serveon the head node, which holds a publicly trusted certificate for<node>.<tailnet>.ts.netand renews it itself. Docs, the benchmarking guide, the ds4f recipe header, and the refusal-probe usage example move to the tailnet URL and drop the pfSense bundle steps.Scope. Launcher code and docs only. No recipe command changes. Operational rollout (systemd drop-in with
--host 127.0.0.1,tailscale serveon the head node, deleting the old cert files) happens outside this PR.BREAKING CHANGE: the launcher no longer serves HTTPS by default. Clients that hit
https://<host>:8000directly must switch to the tailnet URL, or pass the SSL flags after--.Verification
tests/test_recipes.sh: 65 pass, 1 fail (README match: minimax-m2-awq), identical to origin/main.tests/test_launch_cluster_image_sync.sh,tests/test_launch_cluster_vllm_pr.sh: pass.glm-5.3-flash-uncensored-nvfp4anddeepseek-v4-flash-0731-1m(4 dummy nodes,--config /dev/null) andqwen3.5-35b-a3b-fp8 --solorender no--ssl-*flags.Commit is unsigned (GPG agent was locked); can be re-signed while the PR is draft.
🤖 Generated with Claude Code