Skip to content

docs: Qwen3.8-2.4T-A95B UD-Q1_0 has no reproduction recipe and no checkpoint pin in USAGE.md #1194

Description

@localai-bot

Qwen3.8-2.4T-A95B UD-Q1_0 serves on one GB10 today: the borrowed-mmap tier makes it
fit (ENG-EXPERT-STREAM, #912), --offload-config's vllm_cpp key configures it
(#1119), and --device cuda is refused at load with the footprint arithmetic instead of
dying mid-stream (#1123, #1132). None of that is reachable from the docs as a recipe.

The checkpoint is not pinned anywhere a user reads. AGENTS.md requires every ported
model to document the checkpoints it was built and gated against in docs/USAGE.md
file name, size, exact repo and revision, sha256 for a quantized artifact. Revision
567d3e6ac26c5474b18311e619c04350fb9a5556 appears only inside .agents/issue-index.md
row #1123 and .agents/specs/expert-streaming.md. docs/USAGE.md names the checkpoint
three times and pins it zero times, so nobody outside the agent record can reproduce a
single number this project publishes about it.

Nor is there an end-to-end recipe. docs/USAGE.md's "Streaming routed experts from
disk (capacity mode)" is a good reference for the mechanism — schema, precedence, where
the config form does not reach, the statistics line, the CUDA refusal — and it is only
that. It does not say which file to point --model at, what the load costs, what decode
costs, or where the ceiling is. The four sibling per-model recipes in the same file
(Muse Glimmer 30B, Nemotron-3.5-Lightning-30B, MiniMax-H3, MiniMax-Music3) all do.

Two defects in that section, found while reading it.

  1. Both example invocations name -00001-of-00008.gguf. The checkpoint has ten
    shards: UD-Q1_0/Qwen3.8-2.4T-A95B-UD-Q1_0-00001-of-00010.gguf through
    -00010-of-00010.gguf, 397,256,393,248 bytes total, confirmed against the HF API at
    the pinned revision and consistent with the 1702-record census in Qwen3.8-2.4T on --device cuda: the model loads, then the first forward dies on cudaMalloc: out of memory #1123. Copying the
    published line gets a file-not-found.
  2. Line 3848 writes the server as ./build/vllm-server. examples/ is added with
    add_subdirectory and nothing overrides RUNTIME_OUTPUT_DIRECTORY, so the binary is
    at build/examples/vllm-server — which is what the same file writes at lines 1401 and
    2634, and what .agents/specs/expert-streaming.md records as the working invocation.
    Line 2354 carries the same defect for Muse Glimmer.

Not a gap in the mechanism reference. The fix is a recipe section beside its four
siblings that links to the streaming section rather than restating it, so each fact keeps
one home.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions