Integrazione PR upstream: CUDA Qwen3.8 (#1424), QLoRA (#626), API logprobs/echo (#1353) - #1
Merged
Merged
Conversation
Upstream dev rejected any non-null `seed` with HTTP 400. Per-request seeding has no wire path to any engine's RNG, so accept and discard it instead of refusing the request, and document the no-op honestly: glm and inkling seed their own process-global RNG from `SEED` at launch, never per request, and no other engine reads `SEED` or a per-request seed at all. Proven with a test asserting the seed value never reaches the SUBMIT wire frame. Adjacent consequence: renamed the golden-fixture capture case "err_seed" to "seed_accepted" in c/tests/golden_fixture_capture.py (it now captures a 200, not a 400), and updated that file's docstring to match. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…the prompt On /v1/messages with stream: true, the HTTP 200 and SSE preamble were committed before the engine validated the prompt, so a pre-accept refusal (e.g. context-length exceeded) surfaced as a committed 200 followed by a truncated stream. Defer the commit into a start_stream() closure fired on the engine's on_accept, matching the OpenAI streaming path: a pre-accept refusal is now a clean HTTP 400 in the Anthropic error envelope, and a healthy engine's stream is byte-for-byte the same as before. This adds a new invariant to the Engine.generate() contract: any implementation that never calls on_accept now loses the entire Anthropic SSE body, since nothing downstream fires until that callback does. Production cannot hit it -- on_accept is always called no later than the first decoded token -- but an engine binary predating the ACCEPT frame (JustVugg#597) only calls it on the first DATA/DONE, so a cold, multi-minute prefill against such a binary now sends zero bytes over the wire during that wait, where the previous behavior sent message_start plus a keepalive ping every 10 seconds. A client with a short idle timeout could newly time out in that specific pairing. Also guard the commit itself against a client that vanishes at the exact moment the engine accepts: a write failure there used to unwind out of the generation loop with nothing sent, leaving the request stuck in the engine's pending map with no CANCEL ever going out. And give the streaming test double in test_anthropic_messages.py's FakeEngine an on_accept call, matching the one test_openai_server.py's FakeEngine already has, since the deferred commit needs it to exercise the stream at all. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Adds logprobs_options(), a pure helper that normalizes the completions integer-logprobs/echo shape and the chat boolean logprobs/top_logprobs shape into one (engine_k, echo, display_k) tuple, enforces the LOGPROBS_TOP_K_CAP top-k range, and refuses an unsupported engine with a named 400 instead of a silent no-op. A non-boolean `echo` is a named 400 on both endpoints regardless of whether `logprobs` was requested, and chat's `top_logprobs` is type- and range-checked even when `logprobs` is false or absent, so a malformed value is never silently let through by a sibling field that made the check seem moot. Adds six pure response-assembly helpers that turn the engine's numeric logprobs records into the two OpenAI response shapes: the legacy completions object (tokens/token_logprobs/top_logprobs/text_offset) and the chat logprobs.content array. They place echoed prompt positions by the wire's own position field rather than arrival order, reconstruct text offsets correctly across multibyte characters split between tokens, serialize non-finite logprobs as JSON null, and label the chosen token by value rather than by top-k table rank -- deterministically, via a documented first-exact-match tie-break, when the engine's 6-decimal printed value collides between two distinct candidates (the wire record carries no chosen-token id to disambiguate by identity). Wires it all into the request path: the SUBMIT writer opts in with logprobs=k only when requested and supported, generate() parses the engine's numeric tail on DATA/ECHO frames and collects the resulting per-token records into prompt/generated lists (bounded by COLI_LOGPROBS_ACCEPT_TIMEOUT so a pre-JustVugg#597 engine that silently rejects the extended SUBMIT header cannot wedge the caller forever), and the completions and chat handlers assemble the OpenAI-shaped logprobs objects from them. Streaming together with logprobs is refused with a named 400, as is any logprobs request against an engine that does not implement the numeric channel. Completions echo now reconstructs the prompt into `text` itself, not only into the logprobs object; a filtered stop token's own record is dropped so the logprobs arrays never describe a token the client did not receive; prompt-echo records are retained only when the request actually asked to see them -- the engine sends every ECHO frame regardless, so an opted-in request that never asked to see the echo table would otherwise hold a full prompt-length record for nothing. Adds the corresponding docs section and the disclosed golden-fixture- capture rename for the case whose status code changed ("err_logprobs" to "logprobs_served"). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
/v1/completions' `prompt` now also accepts a JSON array: either a flat
list of integer token ids (a pre-tokenized prompt) or a list of
strings/token-id-lists (a batch, dispatched as independent generations
sharing one HTTP response). Each array is validated structurally
regardless of size: it must not be empty, and its element types must
be homogeneous -- all strings, all token ids, or all token-id arrays,
never mixed -- each a named 400 on `prompt`. A single-member array
produces exactly what the same prompt would have produced as a
single-prompt request, including a nested batch-of-one token-id array,
which unwraps to the identical flat behavior. A flat token-id array is
glm-only; every other engine refuses it with `unsupported_parameter`
rather than silently reinterpreting the ids as text.
A real batch (more than one member) dispatches one choice per member,
`choices[i].index == i`, in the same order the array was given, over
one HTTP response (`stream: true` is refused outright, `param:
"stream"`; a batch's `n` other than 1 is refused the same way the flat
path refuses it, `param: "n"`). Validation runs in one fixed order,
matching the flat path for `cache_slot` and the array's own shape and
homogeneity checks, with one exception: `generation_options`
(`max_tokens`, `n`, `temperature`, `top_p`, `stop`) is validated once
per member, after the array shape check but before any member is
submitted to the engine -- so a malformed member fails the whole batch
before any other member's generation has started, and its `param` is
rewritten to `prompt[i]` (except `stream`/`cache_slot`, which are
request-level). A batch's assembled response is held in memory in full
until its single write, so its generated-token total (every member's
effective `max_tokens`, after the server's own clamp to its configured
`--max-tokens`/`--ngen`, summed across members) must not exceed
`PROMPT_BATCH_COMPLETION_BUDGET` (65,536), checked after per-member
validation so a malformed member is still reported by its own defect
first. With `echo` and `logprobs` the same hold additionally retains
every member's echoed prompt positions, bounded instead by
`PROMPT_BATCH_TOKEN_BUDGET` (65,536) times the per-position top-k
table (`LOGPROBS_TOP_K_CAP`, 32) -- roughly double the generated-side
figure in the worst case. Both budgets, and the batch member-count cap
(`PROMPT_BATCH_CAP`), apply to real batches only; a length-1 array is
exempt and keeps the flat single-prompt path's own oversize handling.
Each member is submitted through the engine sequentially, each with
its own isolated decoder state, so one member's malformed UTF-8 tail
or tool-call parsing never contaminates another's. A client-fault
error from any one member (context-length exceeded, an unsupported
token id, and so on) is reported with `param` rewritten to `prompt[i]`
and keeps the engine's own `code`; a server-fault error keeps its own
status/code/param and only gains the member index in its message. A
client that disconnects mid-batch is checked before each member's
submit and stops the loop there instead of running the remaining
members for a dead socket. The scheduler admission is held only for
the submit-and-collect phase and released before the response is
serialized and written, not across it.
Adds the corresponding docs/api.md section and the disclosed golden-
fixture-capture rename for the case whose status code changed on glm
("err_array_prompt" to "array_prompt_token_ids"; every other engine
keeps its 400, now with the named `unsupported_parameter` code).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
group_score is not wired up in this build, and its eventual contract changes the response shape (continuation-only logprob arrays), so silently ignoring the opt-in would hand a client a differently shaped answer than it asked for. The completions-only guard now refuses a truthy `group_score` with a named 400 before either the flat or the array/batch request shape is intaken. Truthiness is checked by identity against `None`/`False`, not equality -- `0 in (None, False)` is true in Python, so an equality check would let `group_score: 0` or `0.0` slip past the guard unrefused. Chat and Anthropic requests never look at the field at all; the guard is completions-only by design. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Teach the stdout dispatcher to read past four group-scoring frame kinds (GRPP, GRPG, GRPS, GRPE) a future engine may interleave with ordinary request frames on the same pipe, so this server stays correct if ever paired with such an engine. No engine in this tree emits them yet; they serve a group-scoring wire channel proposed separately. Payload-carrying kinds (GRPP, GRPG) are drained with the same size bound and terminator check as DATA, naming their own kind on a truncated payload; header-only kinds (GRPS, GRPE) are consumed with no further effect. None ever reach a request's event queue. The ECHO frame arm gets the same kind-naming treatment: its payload read now reports "truncated engine ECHO payload" on a short read, matching the size and terminator checks ECHO's own arm already uses, instead of falling back to the shared "DATA" wording. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A write onto a dead engine's stdin raised BrokenPipeError, a
ConnectionError subclass that do_POST's client-hangup handler silently
swallows -- the client saw a closed connection instead of a 500.
SUBMIT's IMAGE-plus-header write now wraps its OSError into a named
RuntimeError, and the CANCEL/STOP writes now share one checked
_write_frame helper with the same wrap; both surface as HTTP 500
engine_error, as long as the response has not already been committed
(a failed write on an already-committed stream ends the stream instead
of producing a 500). Wire bytes for every frame kind are unchanged.
The engine's stdin is a raw, unbuffered pipe, so a single write() call
can take fewer bytes than it was given; SUBMIT and _write_frame now
loop, re-offering the remainder until every byte is consumed, and a
write that makes no progress (`None` or 0 returned) fails closed as
the same named engine-write error instead of spinning forever. A
two-thread test pins that _write_frame's lock keeps concurrent frames
from interleaving on the wire.
Documents the resulting protocol contract in docs/api.md: server-to-
engine writes are checked and engine-to-server frames are strictly
validated (a malformed frame fails every in-flight request with a 500
and stops the dispatcher rather than desynchronizing the stream, and
that list now includes the four group-scoring frame kinds this program
also added), and the engine child still runs under the default SIGPIPE
disposition. Also corrects the SIGPIPE-disposition precondition test:
it captured `kwargs.get("restore_signals", True)` and asserted it
truthy, which cannot distinguish "no kwarg passed" from an explicit
`restore_signals=True` -- production passes no such kwarg at all.
Assert the kwarg is genuinely absent instead, and pin Python's own
default (`True`) separately via `inspect.signature` so a future stdlib
change to that default would still be caught here.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
self.pending[request_id] was only cleared on three paths: a SUBMIT write failure, the logprobs/token-id accept-timeout, and the dispatcher's own DONE/ERROR frame handling. Every other way generate() can exit -- most concretely, a broken engine stdin surfacing through the CANCEL/STOP write a disconnected or stopped client triggers -- left the request's queue behind in self.pending indefinitely. The leak is bounded (a later dispatcher failure clears the whole map via _fail_pending), but nothing before that point ever reclaims the slot. Wrap the request's lifetime, from admission through every raise or return, in a try/finally that pops self.pending[request_id] unconditionally. The SUBMIT-write failure branch and the accept- deadline branch each had their own explicit pop for this same reason; both are now redundant and are folded into the single finally. This gap predates this branch: the same pending map, the same generate() shape, and the same unwrapped stdin.write() calls for CANCEL/STOP are already present on dev, so the fix applies there too, not only to the logprobs/echo path added here. Added a regression test that forces the leak's most direct trigger -- a disconnected client's CANCEL write failing against a broken engine pipe -- and asserts self.pending is empty afterward; it fails against the old code and passes with the finally in place. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Two of docs/api.md's array-prompt contract sentences had no falsifying test: a token-id batch's aggregate prompt-token budget counts actual tokens (only string batches, counted in UTF-8 bytes, were ever exercised), and a batch that omits cache_slot is admitted once, with every member sharing the single slot the scheduler picked. Both tests were verified to fail against a reverted version of the corresponding server behavior before landing. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ed Anthropic stream A failed CANCEL/STOP write reaching an already-committed stream was only pinned at the Engine.generate() level; nothing proved what a client actually sees on the wire once that failure reaches do_POST's committed branch. Drive a real Engine + fake subprocess whose STOP write fails through a live streaming request and read the raw socket: the earlier SSE bytes survive, no second status line or error body is spliced in, the stream just ends, and the failure is logged. The Anthropic /v1/messages streaming path defers its HTTP 200 until the engine accepts, but nothing exercised the window itself: what a client sees while the engine is still "prefilling". Hold generate() open with BlockingEngine and read the socket mid-wait to confirm zero bytes arrive, then release it and confirm the deferred 200 and full SSE sequence still land once the engine finally accepts. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The bound-blob check hashed the fixture's working-tree bytes. On a checkout that converts line endings (git's autocrlf on the Windows CI runner) the same committed blob arrives with CRLF and the digest no longer matches, which is what the Windows UCRT64 job reported. Normalise CRLF to LF before hashing and measuring; the pinned digest and length are those of the committed LF bytes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… in MoE" JustVugg#1464: an RTX 3080 under WSL2 failed every prompt with request failed: hybrid batched block failed in MoE and a day of flag-flipping could not narrow it, because that message is the generic tail of the layer block. moe_token_pipeline and v4_moe_batch_union can fail in some thirty places -- an expert that would not read, an upload that would not land, the GPU expert group refusing, a routing table missing, a plain malloc -- and every one of them returned a bare -1. The block could not tell an out-of-VRAM card from a bad read over /mnt/c, so neither could the reporter. Every failure site now records a reason (layer, expert where there is one, and the step) through moe_fail(), thread-local and cleared on entry so a stale one cannot outlive the call that set it, and both block tails append it: hybrid batched block failed in MoE: layer 12: the GPU expert group of 6 experts refused (a CUDA allocation or launch failed; check nvidia-smi for free VRAM) block computation failed in MoE: layer 3: reading expert 137 from the expert store failed No behaviour changes on any success path; the only difference is the text in the error buffer. Everything lives in the COLI_V4_UNIT_BLOCK_HYBRID unit, which needed <stdarg.h> of its own since set_error's unit is a different object. tests/test_v4_moe_reason_source.py pins the shape: the only `result = -1` left in those two functions are the "not finished yet" job markers, both functions clear the reason on entry, and both block tails read it. It fails on the tree before this commit. The tiny V4 oracle stays token-exact (target long, CLI, serve, prefix reuse).
fix(deepseek-v4): the MoE step says why it failed, instead of "failed in MoE"
The invite we publish is dead again. The Discord API on the code in the tree:
GET /api/v10/invites/FkyrEeJR -> {"message": "Invite is expired.", "code": 50270}
It appears in ten places: the header badge and the community line of all
four READMEs, and twice in site/index.html. Anyone clicking Discord from the
repo or the site lands on an error page.
Replaced with an invite created with no expiry, verified live before this
commit:
GET /api/v10/invites/RXV83nSZdk -> guild "Colibri - inference AI",
expires_at: null, ~130 members
The previous replacement (JustVugg#1363) died because Discord's default invite
lifetime is seven days; the first candidate for this one had
expires_at 2026-09-20 and was rejected for the same reason.
docs: replace the expired Discord invite (all 4 READMEs + site)
A header the MoE engines share instead of a set of GEMVs each: planar unsigned int4 with one f32 scale per 64-block, f32 activations by default (the SIMD paths reproduce the scalar reference bit for bit), an int8/VNNI activation mode kept opt-in, gate+up in one pass, and xf_moe_run() that runs a whole layer as (expert, row-chunk) items in two OpenMP regions. qwen36 is the first user. Its slots now keep the container's int4 packed (repacked to planar once at load) instead of unpacking every expert to int8, and the MoE goes through the runner. QWEN_EXPERT_KERNEL=0 restores the historical path; the CUDA expert tier keeps its own. Measured on the real gs64 container at cap 256 on an 8-core AVX-512 box, 1024-token greedy decode, text byte-identical to the old path: 12.8 -> 15.7 tok/s, MoE 34 -> 20 ms/token (11 kernel, 6 residual misses fetched one at a time), peak RSS 29 -> 17 GB. The DRAM floor for the int4 bytes is 9 ms and the kernel alone reaches 10. tests/test_expert_ffn pins the numerics (SIMD == scalar, repack round-trip, layer == per-token loop). CI gains an A/B on an int4 gs64 tiny fixture (make_qwen36_tiny.py --inter 64): old path and kernel must emit the same ids at caps 1, 2, 8, 16. The existing int8 gate is untouched, that container never enters the kernel.
expert_ffn.h: one routed-expert kernel for the MoE engines, qwen36 first
… expert layout The shipped fixture stores the routed experts in BF16, so nothing that reads the native FP8 path -- q38_native_fp8_expert_tensors, the block-scale bank, the 4.7 MiB slab per expert -- runs against it, and a GPU expert tier for this engine would have no fixture to be exact against without the 185 GB checkpoint. With --fp8-experts every routed expert matrix is quantized to e4m3 with one scale per 128x128 block (weight_block_size [128, 128], scales as BF16 weight_scale_inv, one block per matrix at fixture size), and the reference is generated from the same dequantized values, so ref.json is the arithmetic of the bytes on disk. In memory the experts are the fused gate_up_proj [E, 2I, H] / down_proj [E, H, I] parameters; save_pretrained splits them into the per-expert tensors the release ships, and the rewrite addresses those names. Shared expert, router and everything dense stay BF16 like the release. Verified: the engine loads it as FP8=native and matches 8/8 at cap 1 and 4; Q38_NATIVE_FP8=0 (expanded-f32 path) matches 8/8 as well. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a
…fixture shards Wires the qwen36 tier into the Qwen3.8 engine. After routing, q38_moe_decode asks the tier which of the token's experts are resident (qt_issue); the CPU loads and computes only the rest, in route order, through the same q38_expert_get_batch / q38_expert_get path with the reduced list, reports each computed native-FP8 slot with qt_note (the tier copies during the call; the slot is recycled by the next token), and qt_take adds the GPU outputs before the shared expert. q38_tier_start runs after model_init: COLI_CUDA=1, native FP8 experts, every layer's block-scale bank resident, then qt_init_fp8 with the e4m3 table; qt_stats prints next to the RAM cache hit rate. The RAM LRU (cap/layer) is unchanged; VRAM is a third stage above it. No heat-file warmstart: measured on the real checkpoint, the hottest set of one prompt covers a different prompt's routes at chance level (4.7 % for one 8 GB card, 9.3 % for two; rank correlation -0.2 per layer), while locality inside a run is strong (decode: LRU 32/layer 55 %, 128/layer 90 %). The tier therefore promotes at qt_note time only. Makefile: qwen38 links qwen36_tier.c and the backend object with CUDA=1, like qwen36; without CUDA the header's inline stubs keep it toolkit-free. Fixture: make_qwen38_tiny.py --fp8-experts now writes two shards plus the index, gate/up and down_proj apart, as the release does. That is what makes a layer's gate/up weight_scale_inv sidecars one compact range and the down sidecars another -- the invariant behind q38_prepare_expert_scale_bank. One file interleaves down/gate/up per expert and sent the engine down the per-matrix fallback, so the tier never came up on the fixture. Tests: tests/test_qwen38_tier_engine.c runs qwen38.c through its own main() on the FP8 fixture against the fake CUDA backend (cap 1, so the RAM slot is recycled right after every qt_note): tier up in fmt 8, LUT published, uploads in gate/up/down triples of one byte per element, hits counted, budget and byte accounting exact, no pointer retained, oracle tokens and logits unchanged. Excluded from TEST_BINS (needs the generated fixture) and run from the new qwen38-tier-engine-check target; qwen38-tiny-fp8-check runs the native and expanded paths against the oracle. Both added to the qwen38-tiny-check CI job. ASan/UBSan clean. Hardware check on the fixture (RTX 3070, COLI_CUDA=1 COLI_GPUS=1): tokens identical to the CPU build at cap 1 and 2, 25 VRAM hits at cap 2. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a
… the docs Planner. qwen38's descriptor sets supports_accelerator=True; `coli plan --gpu` prices a hot tier for it and exports COLI_CUDA/COLI_GPU/CUDA_EXPERT_GB next to COLI_PLAN_CAP. test_resource_plan's qwen38 case asserts the planned device, budget and environment instead of the former CPU-only refusal. Docs. docs/qwen38.md gets a GPU section: what the tier does for this engine, how to build and run it, that heat does not carry over between prompts on this model (held-out 4.7 % / 9.3 % against 5.7 % / 11.5 % chance) while in-run locality is strong (LRU 32/64/128 per layer: 55/79/90 % of decode routes), the VRAM cost per expert at cudaMalloc granularity (the accounting itself is in the tier PR this branch sits on), and where a decode token's time goes. docs/qwen36-cuda-tier.md says the tier serves two engines. The engine test's exp_bytes expectation follows the tier's dev_alloc_footprint like the tier tests do. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a
COLI_TIMERS=1 printed one "total" bank divided by the number of decode forwards. With a 315-token prompt that made the routed-expert line read 930 ms per forward, of which the decode share is about 250 ms (measured per expert: 0.5 ms for the three FP8 GEMVs) -- the rest was the prompt's batched expert work. generate() now snapshots the bank when the prompt's forward is done, and the report prints "prefill" and "decode" banks after "total". No change to the counters themselves. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a
The engine's Speed line divides the generated tokens by the whole generation time, prompt included; the earlier draft of this section had built its budget on that and on the single timer bank. Decode-only figures now: about 0.8 s per token, of which 192 ms routed-expert GEMVs on the CPU; the tier takes 45 % (one card) to 59 % (two cards) of those onto the GPU with identical tokens, and the wall time does not move, because the per-layer round trips and the staging of promoted experts cost what the GEMVs saved. Says so. Also recommends OMP_PLACES=cores (libgomp packs SMT siblings otherwise; +6 % here, +30 % on qwen36). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a
…generic tier dense API Tier. The placer's decision now lives on the offer: qt_place_of(name, layer) answers for any component an engine offered, not only lmhead/dnproj, and auto_place walks every offer (lmhead first, then offer order, which is layer order). New qt_dense_init/qt_dense_matmul/qt_dense_count: an int8 per-row matrix resident on one device, addressed by a handle the engine keeps in its weight -- the lmhead/dnproj mechanism without the tier learning names. Freed at shutdown. tests/test_qwen36_tier_dense.c pins offers by arbitrary name, per-offer placement, budget deduction, fmt-1 upload, handles, refusals. Engine. Every trunk matrix of at least 1 MiB (DeltaNet qkv/z/out, attention q/k/v/o and the QSA indexer, both hyper-connection mixers, shared expert, router, lm_head) is offered before qt_init; what the placer accepts is quantized to int8 per row at start (max|w|/127, OpenMP over rows) and uploaded once. q38_weight_matmul answers S == 1 from the handle; prefill and any failure take the BF16 copy, which stays as the reference. Q38_TRUNK_GPU=0 keeps the trunk on the CPU; the engine test sets it because the fake backend counts matmuls but computes nothing. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a
coli_cuda_matmul staged its input and output in the same x/y device buffers the expert group uses (coli_cuda_expert_group_issue), and the group runs on its own non-blocking stream while the engine thread keeps computing between qt_issue and qt_take. qwen38 calls the dense path in that window -- the shared expert's three GEMVs -- so with the trunk resident in VRAM every dense GEMV overwrote the in-flight group's input, and its output landed in the buffer qt_take reads back: no CUDA error, only wrong tokens, worse the more experts were resident (2.6 % VRAM hits: a divergence after a dozen tokens; 19 %: degenerate text from the second word). The dense matvec now has dx/dy of its own; the group keeps x/y. qwen36 never called the dense path inside that window (dnproj sits in the DeltaNet block, lm_head after the last layer), so its outputs were unaffected. Verified on the real checkpoint: with the trunk on the 3070 and the expert tier at 23 % VRAM hits (345 LFRU swaps), the 30-token greedy output on the 315-token prompt is identical to the CPU int8 reference and to the BF16 run. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YLtHawRZDYQNsBmANKXGgt
… tier, planner prices the trunk Stage 1 of the dense trunk in VRAM, measured on the real checkpoint after the backend fix (Prompt B 315 + 100 tokens, cap 224, decode ms/token, greedy text identical to the BF16 CPU run in every configuration): CPU BF16 744 = 1.34 tok/s experts only, one card 754 = 1.33 trunk on one card, no experts 557 = 1.80 trunk + experts, one card 523 = 1.91 (+43 %) trunk + experts, two cards 466 = 2.14 (+60 %) Engine: Q38_TRUNK_CPU_INT8=1 keeps the same int8 rows on the CPU and answers decode GEMVs from them -- the quantization's effect on its own, no GPU needed (perplexity, token parity); Q38_TRUNK_MIN_KB and Q38_TRUNK_SKIP choose what is offered; Q38_TRUNK_SELFTEST=1 checks every placed matrix once against the CPU int8 rows (all 553 within 1e-7). The quantizer is one function for both paths; the int8 rows go with the weight. Tests: the fake backend can compute fmt 1 from the uploaded bytes (fake_dense_compute), so the engine test now runs a second pass with every dense matrix of the fixture on the fake tier and demands the oracle within its limits (the cosine is exactly the CPU int8 reference's), every offer placed, every handle released at shutdown. Planner: a trunk_inventory hook on the family descriptor (qwen38: the offered matmul matrices of at least 1 MiB, int8 bytes) feeds analysis["trunk_int8_bytes"]; build_plan takes the trunk out of the first device's VRAM before the experts, reports tiers.vram.trunk_bytes and a "dense trunk as int8 residents" decision, and the plan line reads "4.0 GB int8 trunk + 2.1 GB hot tier". Analysis cache version bumped. Docs: docs/qwen38.md "The dense trunk in VRAM (stage 1)" (mechanism, variables, numerics, the measured table, the staging-buffer finding), docs/qwen36-cuda-tier.md (the generic dense API, the backend fix). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YLtHawRZDYQNsBmANKXGgt
…json Checkpoints distributed with an overlay shard (community abliterations, for one) keep the superseded tensors physically in the base shards and let the index say which copy is authoritative. The duplicate-name check refused such a directory outright (JustVugg#1479), which made it unloadable without rewriting the shards. Now, when two indexed shards carry the same name, the index decides: the copy it maps the name to is read (replacing the earlier entry in place or skipping the later one, whichever order the scan met them in), the other is ignored, and one summary line says how many names were resolved that way. The index is parsed lazily, only on the first duplicate, so a clean container never pays for it. No index, or an index that does not settle the name, keeps the refusal, now with a line saying why the index could not help. tests/test_st_overlay covers both scan orders, the index preferring the base copy, and the two refusals (fork, POSIX only). Closes JustVugg#1479
The engine read its end-of-generation ids from generation_config.json alone. A converted container without that file (the converter's --indir path copied no metadata at all; the GLM-5.3-Flash config.json carries the ids only under text_config) never stopped and ran to --ngen (JustVugg#1478). stop_ids.h is the shared reader: generation_config.json, then config.json at the top level, then config.json's text_config, and the caller learns which one answered. glm53 logs the source once, or a warning when no file declares any id. The converter now copies the metadata files that sit next to --indir shards, and when generation_config.json is still missing but config.json declares the ids it writes the minimal file, so every reader agrees. tests/test_stop_ids covers the three sources, the int and list forms, the cap, a generation_config.json without ids, and a missing directory; tests/test_convert_glm53_meta covers the converter helpers. Closes JustVugg#1478
…hput The attached-chat footer was a hard-coded "~N tok · Ss" with N estimated from the text. Measurement and rendering are now apart: gen_stats() takes the server's exact completion_tokens when the streamed usage block carries them (the request asks for it with stream_options.include_usage) and falls back to the chars/4 estimate, marking which; render_gen_stats() lays it out for --stats full (tokens, seconds, tok/s), compact (tokens, tok/s) or off, with COLI_CHAT_STATS as the default. Exact counts drop the tilde. The private-engine chat keeps its own exact footer and honours off. Closes JustVugg#1475
st.h: resolve duplicate tensor names through model.safetensors.index.json (JustVugg#1479)
glm53: stop ids from config.json too, and the converter keeps them (JustVugg#1478)
coli chat: configurable generation statistics with exact token throughput (JustVugg#1475)
coli picks the engine from config.json. Without it `coli info` still printed the GLM engine's status line, as if that were the engine to run, and the error the other commands raised said only "cannot read config.json". A user who had copied the shards of Qwen3.8 without the small files read both, ran the GLM engine by hand, and reported the engine's "missing model.embed_tokens.weight" as a bug. Now `coli info` says the engine is unknown until config.json is present, counts the shards found, and names the files to copy from the model repo; the FamilyConfigError every other command raises carries the same guidance.
st_die_missing counted shard filenames, so a directory with every shard present and a tensor that never existed in that checkpoint (a Qwen3.8 copied without config.json, run by the GLM default, which looks for model.embed_tokens.weight) got "every shard is present, please report it". model.safetensors.index.json says more than a file count: whether the name exists in this checkpoint at all, which shard should hold it, and whether that shard is here. The three answers are three messages: wrong engine for this directory (with the coli info pointer and the files to copy), shard missing (with the one-file download line), shard present but not the file the index describes. Without an index the historical wording stands. tests/test_st_missing runs each case in a forked child and reads its stderr (POSIX only).
--model names the directory the converted weights are written to; a user who pointed it at a downloaded Qwen3.8 read it as "the model to convert" and watched coli start fetching the default repo, GLM-5.2, into that directory. Now a --model directory holding shards or a config.json stops the command before anything runs, with the answer the user most likely wanted: a family that runs its official checkpoint needs no conversion (coli chat --model <dir>); a family with a converter gets the --indir line for shards already on disk; anything else gets the fresh directory form. --repo's default is applied inside the command so the help can say what --model is not.
coli: a directory without config.json names no engine
st.h: a missing core tensor is diagnosed through the checkpoint's index
…eckpoint-dir coli convert: refuse an output directory that already holds a checkpoint
Ported from the qlora-train branch (pre-v1.0 base): safetensors adapter format with base-fingerprint gating (LORA_UNSAFE=1 to override), atomic writer, CPU residual application via ADAPTER=<dir>. Without an adapter, inference output is bit-identical (tiny oracle TF 32/32 on this base). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The full training path from the qlora-train branch, ported to colibri.c: manual backward through the GLM block with activation checkpointing (routing replay; recompute bitwise-equal to retention), expert-grouped streaming with row-batched fwd/bwd, budget manager (shed expert cache before the OS swaps), masked-CE SFT dataset reader, atomic checkpoints with bitwise resume, coli_train CLI. Gates: test_train_linear (PyTorch f64 parity), test_train_tiny (trajectory vs torch, rel 2.4e-6), test_train_resume (bitwise) — all wired into make test. Real-model probe on this base reproduces the M7 baseline exactly (loss 3.5764, 239s/step, identical expert I/O). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Merged around the new r_top8_par work; metal-test green for both suites. Dispatch is opt-in (COLI_METAL=1, COLI_TRAIN_METAL_MIN rows) and off by default: no wall-clock win yet (experts are fmt=4 grouped on CPU) and the f32 GEMM forward numerics differ from the CPU idot path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
m7_smoke.sh (build -> doctor -> 1+10 steps -> adapter-in-inference) and m8_overfit.sh (overfit + on/off eval), adapted to the colibri binary name; persona + dolly tokenized datasets; the M7 runbook and the implementation brief the milestones were built against. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Tiny-oracle end-to-end repro for reviewers: base greedy matches the PyTorch reference 20/20 -> coli_train overfits a rank-16 adapter onto a chosen alternative continuation (~90s CPU) -> ADAPTER set reproduces the trained continuation 20/20 -> ADAPTER unset matches the reference 20/20 again. Needs torch once (tiny model generation); training and inference are pure C. tools/make_tiny_sft.py builds the coli-sft-v1 set from the oracle's own prompt ids, no tokenizer involved. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One forward per question: force-closed think block, 'Answer:' suffix, 2 greedy tokens (DRAFT=0), first A-D letter wins. Same protocol both sides so the gap is the adapter's. Deterministic seeded subset spread across subjects; question pool fetched once from the HF datasets server and cached locally (nothing committed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The engine build broke on the Windows CI runners (lora.h:183) because mkdir(path, 0755) is POSIX; MSVC/mingw expose _mkdir(path) only. Also split on backslash so nested Windows-style paths create correctly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…missed Same UCRT64 failure class as lora.h: mkdir(dir,0755) is POSIX-only. The test already includes lora.h via colibri.c, so reuse LORA_MKDIR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- c/data/m8_tokenized is no longer committed: m7_smoke/m8_overfit generate it from data/m8_persona.jsonl via tools/prepare_sft.py on first use (tokenizer from the snapshot). Verified byte-identical to the removed fixtures when regenerated from the zai-org/GLM-5.2 tokenizer (seed 0, all six files). - lora.h added to the colibri rule prerequisites — caught by the new test_makefile_deps meta-test on current dev. - test_train_tiny stays in TEST_BINS (unchanged; noted per review). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts: # .gitignore
# Conflicts: # c/tests/test_openai_server.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Porta nel fork tre PR ancora aperte su JustVugg/colibri, unite su
main(f028d26).coli_train).gitignore, tenute entrambe le partic/tests/test_openai_server.py: import uniti (incluso_image_bytes_from_url) e classi di test di entrambe le partiEscluse: JustVugg#825 DirectStorage (24 blocchi di conflitto nel motore C, base vecchia di >1100 commit), JustVugg#1338 Vulkan e JustVugg#1323 Metal (non per NVIDIA), JustVugg#1313 e JustVugg#1397 (problemi aperti).
Verifiche
python -m unittest tests.test_openai_server: 348 test, OK (3 saltati)tests.test_anthropic_messages: OKNota JustVugg#1424: il trunk passa da BF16 a int8 (perplexity +1,9% nel test dell'autore).
🤖 Generated with Claude Code