[Bug]: eos_token_id dropped from converted config.json - non-terminating generation

[montana-rs]

Commit or version

v1.10.2 (also verified on current main). 12a5c46

Environment

Fedora 44 (Arch-based EndeavourOS host also reproduces the conversion-side issue), x86_64, Intel i7-13700HX, 31 GiB RAM, 6 GiB VRAM (RTX A1000 Laptop, engine ran CPU-only), 1 TB gen4 NVMe, gcc 15 (system gcc).

Reproduction steps

1. Convert a GLM-5.3-Flash checkpoint (HF zai-org/GLM-5.3-Flash) with coli convert.
2. Diff config.json source vs converted output: eos_token_id present in source, absent in converted output.
3. Run coli chat -- generation continues past the model's natural EOS until --ngen is exhausted.
   Workaround (verified): manually copy eos_token_id into the converted config.json, after which generation terminates correctly.

Expected behavior

Converted config.json carries eos_token_id (or the engine reads generation_config.json's EOS list, as the source checkpoint provides both), so generation stops at the model's natural EOS.

Actual behavior and logs

Converted config lacks EOS ids entirely. Generation continues past natural EOS: the model produces its thinking block and then repeats/degrades until the token limit. After the manual workaround, generation terminates correctly.

[JustVugg]

Confirmed, and the cause is a little different from the title: the converter does not rewrite config.json (it copies it as is), but the engine read its stop ids from generation_config.json only. Two ways to end up without them: coli convert --indir copied no metadata files at all, and the GLM-5.3-Flash config.json declares eos_token_id only under text_config, so a hand-copied config.json did not help either. That is why your manual copy into the top level of config.json fixed it once the engine looked there.

Fix in #1481, on both sides:

• the engine reads generation_config.json, then config.json at the top level, then config.json text_config, and logs where the ids came from ([glm53] 3 stop id(s) from config.json (text_config)), or warns when nothing declares any;
• the converter copies the metadata files that sit next to --indir shards, and when generation_config.json is still missing but config.json declares the ids it writes the minimal file.

So a container converted before this fix also stops correctly with the new engine, with no manual edit.

[montana-rs]

Confirmed working on our converted GLM-5.3-Flash container with the new engine: generation now terminates correctly, no manual config edit. And thank you for the mechanism correction - the converter-copies-as-is detail plus the text_config nesting explains why the initial "converter dropped it" read was wrong and why the manual top-level copy only worked by luck. The lookup-order logging is a nice touch; knowing where the stop ids came from would have saved debugging.

[JustVugg]

Merged into dev (#1481, merge commit 035699f). Please try it on your converted container as it is, without the manual eos edit: the engine should print [glm53] 3 stop id(s) from config.json (text_config) (or from generation_config.json) at load and stop at the model's natural EOS. A fresh coli convert also writes generation_config.json into the container now.

[montana-rs]

Both dev-branch checks pass on my setup:

• Converted GLM-5.3-Flash container loaded as-is, no manual edit: [glm53] 3 stop id(s) from generation_config.json at load, exactly as expected since the container has the generation_config.json from the source checkpoint.

[rennf93]

Same signature on CPU-only + bcache, independent reproduction:

• Hardware: UGREEN NAS, Intel N150 (4C, no GPU), 2x NVMe bcache (writethrough) backing the model store, 128GB RAM. colibri v1.10.2, GLM-5.3-Flash-colibri-int4-g64.
• Behavior: first chat completion starts the engine generating and it never terminates - glm53 pins 500-566% CPU indefinitely (343+ CPU-minutes observed on one instance, 80+ on another). Every subsequent request queues to the 300s queue-timeout ("Timed out waiting for the inference engine", code queue_timeout) or surfaces as client-side connection errors.
• Workaround: DRAFT=0 (also what coli tune recommends on this box). With MTP disabled the engine idles ~30% between requests and every generation completes normally.
• Consistent with eos_token_id being dropped in conversion: with no EOS the decode loop never stops, and the MTP draft loop amplifies it into a permanent spin.

Happy to test a fixed conversion when it lands.

[JustVugg]

@rennf93 thanks for the report, but I do not think yours is the same defect, and it is worth separating the two before you spend time on it:

1. The container you name, Justvugg/GLM-5.3-Flash-colibri-int4-g64, ships generation_config.json with eos_token_id: [154820, 154827, 154829] (I checked the repo just now), and v1.10.2 reads exactly that file. So on your box the engine did have its stop ids. The bug fixed here is for containers that came out of a conversion without that file.
2. glm53 has no speculative draft loop: DRAFT is a knob of the GLM-5.2 engine (colibri) and glm53 never reads it. So DRAFT=0 cannot have changed what the engine did; what changed between the two attempts was most likely the page cache. With 128 GB of RAM the second run finds the 12 GB of dense weights and the hot experts already in memory, the first run read them cold through bcache.
3. The numbers fit a cold first request rather than an infinite one: the measurement in docs/glm53-flash.md is about 44 s per token cold on 6 cores with an ordinary disk, and a token touches 4.8 GB of experts. On an N150 (4 cores) with a few hundred tokens of thinking, the first completion is hours, and every request behind it hits the 300 s queue timeout, which is the second half of what you saw.

Two things would settle it, on a dev checkout (the fix is merged there):

• the engine's stderr for the first request: it now prints [glm53] 3 stop id(s) from generation_config.json at load, and with COLI_TIMERS=1 a per-token line; if tokens are being produced slowly, it is speed; if the answer runs to --ngen and stops there, it is the stop ids after all;
• the same prompt twice in one server session, without touching DRAFT: if the second is fast, it was the cold cache.

If the first shows tokens flowing at seconds each, the right issue is a performance one for that hardware (4 cores, bcache) and I would rather track it on its own than under this fix.

[JustVugg]

One more note for anyone landing here from the Hub: Justvugg/GLM-5.3-Flash-colibri-int4-g64 (the container in the report above) is complete, 62 of 62 shards, 194.7 GB. Justvugg/GLM-5.3-colibri-int4-g64 (the 744B GLM-5.3) is still being uploaded, one shard at a time: 87 shards are there as of tonight and the rest are on their way. Until the last shard lands, an engine pointed at a partial download of that repo stops on a missing tensor at load (and, with #1487, says which shard is missing). That is the upload, not this issue; there will be a note in the repo README when it is complete.

[rennf93]

Tyvm, @JustVugg !

Data as requested, on v1.11.0 (upgraded, COLI_TIMERS=1 set in the serve env):

• Load: engine loads, API healthy; the stop-id stderr line does not surface through the container log, noting in case it should.
• Same prompt twice in one server session (count 1-10, max_tokens 64, thinking off):
  • Cold (right after container recreate): 75.7s wall for 20 completion tokens (~3.8 s/token)
  • Warm (immediately after): 61s for 20 tokens (~3.0 s/token)
• So: warm is ~20-25% faster, but the fundamental rate on this box is seconds-per-token - which matches your read: on a 4-core N150 the bottleneck is compute (plus cold expert reads through bcache), not an infinite loop. The "343 CPU-minutes" observation was legitimate slow generation of a longer agentic response, made worse by my own client timeouts and restarts resetting the cache.

Agreed it is a performance topic for this hardware class rather than this bug. Closing the loop from our side - thanks for the detailed analysis, it corrected our diagnosis.

(DRAFT=0 is indeed a no-op on glm53 - left it out of the compose config going forward.)

[JustVugg]

Fixed by #1481, merged into dev (035699f): the engine reads eos ids from generation_config.json, then config.json (top level, then text_config), and the converter carries the metadata from --indir and writes generation_config.json when only config.json has the ids. Ships in the next release. Closing the original report; @rennf93 your symptom (cold first request on a 4-core box) is not this bug, as discussed above: open a separate performance issue with the dev timers if it persists.