qwen38: CUDA VRAM tier for Qwen3.8-Flash-Next -- hot experts, and the dense trunk as int8 residents (+43 % on one 8 GB card, +60 % on two)

[kreuzzelg]

Follows up on #1242 ("no Qwen3.8 GPU backend yet") and jazir555's ask on #1250; JustVugg
said in #1242 it is open and, in #1045, that this is item C with B's placement logic inside it.
Nine commits on current dev (#1344, #1360, #1361 and #1374 are in); rebased 10.09.

What it does

Qwen3.8-Flash-Next gets a GPU path through the same qwen36_tier.c as Qwen3.6, in two parts.

1. Expert tier, fp8 streaming mode (qt_init_fp8). Experts keep streaming from disk into
the per-layer RAM LRU (cap); the tier holds copies of the hot ones in VRAM as a third stage
above it and computes those on the GPU with #817's grouped_*_f8w kernels (e4m3 bytes + the
checkpoint's 128x128 block scales, no conversion). Staging copies slab and scales at qt_note
time -- the engine's RAM slot may be recycled by the next token -- and promotes by heat
(room, or a budget-neutral swap against the coldest resident). q38_moe_decode: qt_issue
after routing, the CPU computes only the non-resident experts in route order, qt_take
before the shared expert. No heat-file warmstart, on purpose (measured: heat does not carry
over between prompts on this model; inside a run the locality is strong).

2. The dense trunk in VRAM (stage 1). The trunk is the largest fixed cost of a decode
token here and it is bandwidth-bound (8 GiB BF16 read per token). The engine offers every
dense matmul matrix of at least 1 MiB to the tier's placer by name and layer (DeltaNet
projections, attention, hyper-connection mixers, shared expert, router, lm_head -- 553
matrices); the placer takes them before the experts (read every token), the engine
quantizes what was accepted to int8 per row at start (2 s, 3.96 GiB on one card) and answers
decode GEMVs from VRAM with one round trip per matmul. Prefill, the DeltaNet recurrence, QSA,
PLE and the CPU fallback stay as they are. The tier side is generic: qt_trunk_offer(name, layer, bytes) / qt_place_of / qt_dense_init / qt_dense_matmul -- the qwen36
lmhead/dnproj calls are thin wrappers now. coli plan --gpu prices the trunk
(4.0 GB int8 trunk + 2.1 GB hot tier).

3. A backend fix the trunk uncovered (b86e55e): coli_cuda_matmul staged its input and
output in the same x/y device buffers as the expert group, which runs asynchronously on
its own stream between qt_issue and qt_take. qwen38 computes the shared expert in that
window, so with the trunk in VRAM every dense GEMV overwrote the in-flight group's input and
its output landed where qt_take reads -- no CUDA error, only wrong tokens, worse the more
experts were resident. The dense matvec has its own buffers now. qwen36 never called the
dense path inside that window (dnproj in the DeltaNet block, lm_head after the last layer),
so its outputs were unaffected; the tier tests and the R6 calibration are not touched.

Also: a decode timer bank apart from prefill (COLI_TIMERS=1; the Speed line divides by
the whole generation time, prompt included, and read like a decode figure), the fixture in
the release's two-shard layout (in one file the scale sidecars interleave and the engine
silently takes the per-matrix fallback), and the diagnostics I needed:
Q38_TRUNK_CPU_INT8=1 (the same int8 rows on the CPU: what the quantization alone does,
no GPU needed), Q38_TRUNK_SELFTEST=1 (each placed matrix once, GPU against CPU int8),
Q38_TRUNK_MIN_KB, Q38_TRUNK_SKIP, Q38_TRUNK_GPU=0.

Measured (Threadripper PRO 3945WX 12C, 94 GB, RTX 3070 8 GB + Quadro RTX 4000 8 GB)

Checkpoint: Qwen/Qwen3.8-Flash-Next-FP8 as
published, read directly (no conversion, no container of our own; the int8 trunk is built at
start in 2 s). The engine test's fixture is synthetic, make qwen38-tiny-fp8-generate, no download.

Prompt of 315 tokens plus 100 generated, cap 224 (the RAM LRU serves 84 % of expert reads),
OMP_PLACES=cores, 12 threads, checkpoint in the page cache, decode bank, five runs back to
back with nothing else on the machine. Greedy text identical to the BF16 CPU run in every
configuration (100 tokens).

	CPU only	experts only, one card	trunk, one card, no experts	trunk + experts, one card	trunk + experts, two cards
VRAM: trunk / experts	--	0 / 6.1 GB	4.0 / 0.1 GB	4.0 / 2.1 GB	1.8 + 2.2 / 4.3 + 4.3 GB
VRAM share of routed experts	--	43 %	2 %	22 %	52 %
dense trunk (resident-mm), ms/token	284	285	52	63	89
routed-expert GEMV on the CPU	179	118	182	151	93
expert reads (page cache)	168	164	162	160	145
decode, ms/token	744	754	557	523	466
decode tok/s	1.34	1.33	1.80	1.91	2.14

The expert tier on its own does not move the wall time (it removes GEMV time and spends
about as much on its per-layer round trips and on staging -- the GEMVs were a quarter of the
token); the trunk on its own is +34 %; together +43 % on one card and +60 % on two, where the
placer spreads the trunk by free room and the experts on both. TTFT (183 s, prefill on the
CPU) and peak RSS (60 GB at cap 224) do not change.

Numerics: GPU int8 against the same int8 rows on the CPU, relative error ~1e-7 on all 553
matrices; int8 against BF16, 1 - 3 % per matrix, greedy text identical over 100 tokens;
perplexity on wikitext-2 (8 x 512, 2048 scored tokens): 1.880 with trunk + tier on the GPU against
1.845 BF16 on the CPU (+1.9 %). Probes: the fp8 expert tier is neutral, lm_head contributes nothing,
routers kept in BF16 (Q38_TRUNK_SKIP=router) recover only 0.3 points (1.875) -- the per-row int8
error is spread over the trunk. Group-scaled int8 for the trunk is the follow-up.

Side finding for the docs of both engines: OMP_PROC_BIND=close without OMP_PLACES packs
libgomp's threads onto SMT siblings; OMP_PLACES=cores gave +6 % on qwen38 and +30 % on
qwen36 (11.0 -> 14.4 tok/s, one card, cold).

Tests

test_qwen36_tier_fp8.c, test_qwen36_tier_dense.c (placement by name, handles, fallback,
shutdown), test_qwen38_tier_engine.c (qwen38.c through its own main() on the FP8 fixture
against the fake backend; pass 1: experts, cap 1 so the RAM slot is recycled after every
qt_note; pass 2: every dense matrix of the fixture on the fake tier, which now computes
fmt 1 from the uploaded bytes, oracle within limits, every offer placed, every handle
released), CI steps qwen38-tiny-fp8-check and qwen38-tier-engine-check; the planner
test asserts the trunk bytes and the plan line. ASan/UBSan clean on the engine test; all
tier tests and the Python suites pass. test_cli_output pinned qwen38 as CPU-only (--gpu
refused up front); with supports_accelerator it asserts the accelerated engines' contract
instead (--gpu needs the CUDA build on a CPU-only binary). Re-run after the rebase on
6f9117f: greedy text identical over 100 tokens, 1.87 tok/s, same residency. While measuring
I hit an unrelated crash in dev's warmstart (#1422, fix in #1423); it does not touch this diff.

Not in this PR

Activations on the GPU per layer (stage 2, would also take the experts' round trips away),
prefill on the GPU, an expert arena per device (cudaMalloc granularity costs 22 % of the
fp8 experts' VRAM), a trunk split by layer across two unequal cards (the placer spreads by
free room today), Vulkan/Metal.

Thanks on Claude Code

[JustVugg]

Reviewed and merging. The backend fix (separate staging buffers for the resident dense matvec, so it no longer shares x/y with the in-flight expert group) is a correctness change worth having on its own; the placer generalisation keeps qwen36's lmhead/dnproj behaviour; everything else is opt-in under COLI_CUDA=1 with the BF16 CPU path as the reference and fallback, priced by the planner and pinned by the fake-tier engine test. One note for the docs later: the int8 trunk is a quantization of the dense path, so the greedy-identical result on 100 tokens is the measurement, not a guarantee, which is why Q38_TRUNK_CPU_INT8=1 and Q38_TRUNK_GPU=0 matter. Thank you for the two-card numbers.