[olmoe] Engine dies on prefill > ~1024 tokens with default cache cap — reload storm saturates the eviction loop

[assistantia-info]

[olmoe] Engine dies on prefill > ~1024 tokens with default cache cap — reload storm saturates the eviction loop

Version: v1.10.1 · Engine: c/olmoe.c · Model: OLMoE-1B-7B int8 (5 shards, 7.41 GB container) · Gateway: coli serve (OpenAI-compatible API)

Summary

With the default implicit cache cap (implicit_cap=8 slots/layer for olmoe, family_registry → openai_server.py:cap_for_arch), any chat completion whose prompt exceeds ~1024 tokens (~4 000+ characters) kills the engine process mid-prefill. The gateway survives and returns HTTP 500 (or the client connection breaks). With --cap 64 (all 64 experts/layer resident), the exact same request completes normally — which both isolates the defect to the cache/eviction path and provides a workaround.

Reproduction (minimal)

# 1. Start the gateway with defaults (implicit cap 8 for olmoe)
coli serve --model <model-dir>            # port 8000

# 2. Send a ~5 000-char (~1 250 token) user prompt
curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"olmoe-colibri",
       "messages":[{"role":"user","content":"<~5000 chars of any text> … Answer in one sentence: what is the text about?"}],
       "max_tokens":60, "stream":true}'

# Expected: engine process dies mid-prefill; gateway returns 500 / connection failure.
# 3. Same request with --cap 64: completes (TTFT ~133 s on an i5-7300HQ, then 4.5 tok/s decode).

Evidence — bisect on prompt length (same model, same machine)

Prompt	Outcome
~4 000 chars / ~1 000 tokens	completes (TTFT 172-183 s — the reload storm is already visible)
6 000 chars	fetch failed (engine dead, connection reset)
6 789 chars	HTTP 500 (engine dead, gateway reports)

Threshold is stable at ~1024 prefill tokens; repeated runs alternate between the two client-visible failure modes.

Root cause hypothesis (documented upstream, not new)

chat_olmoe.sh — the engine's own reference script — already warns:

« This engine doesn't dedupe repeated expert picks within a single prefill batch,
so a too-small cache causes constant reload storms; caching every expert
sidesteps that entirely. »

With cap 8, a long prefill cycles far more distinct experts than 8 slots/layer can hold. The eviction loop's last-resort branch (« every slot in-flight ») then saturates and the engine dies. Our measurements corroborate the storm mechanism directly:

• cap 8, short prompt: decode ~2.0 tok/s, warm TTFT ~9 s — the cache is too small for the working set even before the crash zone.
• cap 64, same session: decode 3.9-4.5 tok/s, warm TTFT 1.9-2.5 s, and the crash zone prompt completes (1 385-token prefill, TTFT ~133-153 s across two sessions).
• Boundary, from the full matrix: cap 8 dies on the long leg in 4/4 cells; cap 32 survives it in 4/4 cells but at ~1.9 tok/s decode and ~237 s prefill — the defect sits between cap 8 and cap 32, and the useful performance sits at cap 64.

(Method note: the serve-mode turn breakdown reports expert_disk_s = 0.0 uniformly — including on the slow cap-32 cells — so that field is not usable as reload-storm evidence in serve mode; the crash boundary and the performance cliff above carry the evidence.)

Full 12-point matrix (cap 8/32/64 × PILOT 0/1 × CTX 2048/4096) attached in the measurement report (#1441) — the crash reproduces on every cap-8 cell (4/4) and on no other cell; cap 32 survives the same prompt but at half the decode rate (~1.9 tok/s vs 3.9 at cap 64) and double the prefill time (~237 s vs ~153 s), which places the defect boundary between cap 8 and cap 32 and the performance cliff between 32 and 64 (the all-experts-resident threshold).

Environment

• CPU i5-7300HQ (4c/4t, AVX2), 16 GB RAM, CPU-only, Windows x64.
• colibri v1.10.1, OLMoE-1B-7B int8, dense 953 MB + 64 experts/layer × 16 layers (6.46 GB experts, 6.31 MB/expert), KV fp32.
• Freshness at filing time: upstream current release is v1.10.2 — c/olmoe.c is byte-identical between the v1.10.1 and v1.10.2 tags (direct diff) — the defect path is unchanged in the current release.

Suggested directions (happy to prototype any of them)

1. Dedupe expert picks within a single prefill batch (the storm's fuel) — the fix chat_olmoe.sh implies.
2. Bound the in-flight set / add backpressure instead of the last-resort « every slot in-flight » eviction, so a saturated cache degrades latency rather than killing the process.
3. Document the interaction: long prefills × small caps = crash zone, with --cap 64 (or a cap sized to experts/layer) as the workaround.

(3) can ship today; (1) is the real fix; (2) is the defensive layer.

---

Conçu dans Électra et piloté par Super Z GLM 5.3 — mesure protocole C6, journal reproductible.

[JustVugg]

I tried to reproduce this before answering, on the same model, and I could not
make the engine die. Here is exactly what I ran, because the difference between
your result and mine is the interesting part.

OLMoE-1B-7B int8, the same 7.4 GB container shape, driven through the engine's
serve protocol directly (no gateway in the way), cap 8, a 6048-character prompt
that tokenises to 1204 tokens, well past your ~1024 threshold:

DONE 1 STAT 16 0.045 22.8 2.87 1204 1

It completes. Sixteen tokens, 22.8% expert hit rate, 2.87 GB RSS, and about six
minutes of wall clock. Slow exactly the way you describe, but alive. Linux,
12 threads. The same prompt at cap 64: 99.4% hit rate, 0.215 tok/s, 7.74 GB.

So the reload storm is real and I measured your ratio, but the death is not
reproducing on my platform. That matters for the fix, because the two hypotheses
lead to different code: the while (lru < 0) branch you point at waits for
an in-flight slot to publish and rescans, so if it saturated the engine would
hang, not exit. Something else is killing your process.

What is going in anyway

#1445 removes the configuration this needs. The default cap for olmoe stops
being a constant 8 and is derived from the RAM budget once the dense weights are
resident: on a 16 GB box the whole expert set is 6.5 GB, so the cache sizes
itself to hold all 64 and you land on the configuration you already measured as
working and twice as fast. --ram also reaches the engine now (#1443).

That is not the same as fixing the crash, and I am not going to claim it is.

What would let me fix the real one

If it still dies after #1445 with an explicit --cap 8, three things would
close it:

1. Does it exit or does it hang? An exit code, or Windows Event Viewer's
   entry for the process. 0xC0000005 says memory, 0xC0000409 says stack,
   a clean non-zero exit says the engine chose to stop and printed something.
2. Its stderr. The gateway captures it; a crash inside the engine usually
   prints one line before going. That line, or its absence, splits the space in
   half.
3. Whether PILOT=0 changes it. Your matrix says the crash is in all four
   cap-8 cells including PILOT 0, which already rules the prefetch worker out as
   the sole cause, but confirming it after the cap change is cheap.

Leaving this open, not closed by #1445: the configuration goes away, the defect
does not, and I would rather have it recorded than tidied.

Thank you for the bisect on prompt length and for the note that
expert_disk_s = 0.0 is not usable as evidence in serve mode. That saved me
from chasing a zero.

[assistantia-info]

Both, actually — and your hang-vs-exit distinction reframed our data, so we re-ran the death prompt today with instrumentation built for exactly your three questions (client timeout raised from 360 s to 900 s, the gateway's engine-death diagnosis captured from its console, health/profile sampled after each leg, explicit PILOT=0). Same machine, same model, same container, cap 8, through the gateway:

• 6.7 k-char system prompt (2 291 prompt tokens after tokenization) → HTTP 200 in 787 s. The engine was alive the whole way: 13 forwards, 12 completion tokens out, gateway health 200 afterwards, wrapper process alive, RAM available dropping 7.3 → 4.0 GB during the leg, and the gateway's engine-death diagnosis never triggered — because the engine never exited. That is the hang mode your while (lru < 0) analysis predicts: it waits, and on our 4-thread box it waited through a 2 291-token prefill at ~2.9 tok/s and answered.
• A deeper leg (2 478 tokens) was still actively prefilling at 764 s when we stopped it (gateway health 200, scheduler showing the request active, zero engine-termination lines in the console, and the six most recent Windows Application event-log entries all Level=Information — no crash, no WER). The effective prefill rate had collapsed below 1.3 tok/s: the storm deepens with prompt length, which matches your 22.8 % vs 99.4 % hit-rate measurements through a different machine.

Now your three questions, precisely:

1. Exit or hang? Today, twice: hang. The engine did not exit under instrumentation. The exit mode is real but intermittent on our box — in the original bisect (same prompt file, same machine, default cap 8), 1 500 c completed in 131 s, 3 000 c in 238 s, then 6 000 c broke the client connection and 6 789 c returned HTTP 500 with the gateway's own body "The colibri engine failed to process the request." The very same 6.7 k-char prompt survived today. We did not capture an exit code or an Event Viewer entry back then because those runs predate the instrumentation; the harness now captures all of it, so the next exit-mode recurrence will produce the exit code, the stderr tail, and the event-log entry.
2. Its stderr. Same honest answer: nothing to show, because the engine never exited while instrumented — the gateway's engine_diag (which waits on the child, maps negative return codes to signals, and drains the last lines of the engine's stderr) never fired. The one hard datum from the bisect deaths is that the gateway, not the client, declared the engine failed (the 500 body above), and in the 6 000 c case the client connection broke outright — which would fit the gateway itself dying or hard-resetting, though we cannot distinguish that today.
3. PILOT=0. Confirmed twice over: today's instrumented legs ran with explicit PILOT=0 and stayed alive, and the 12-cell matrix's four cap-8 cells include both PILOT=0 cells. The prefetch worker is excluded as the sole cause.

A correction we owe you on the matrix: the report's "crash reproduces on every cap-8 cell (4/4)" overstated it. Those four cells were client timeouts at 360 s with the engine still running — the gateway's health endpoint showed the long request still active (admitted 3, completed 2), and no engine-termination line appears anywhere in the 12-cell journal. With the timeout raised to 900 s, the same class of leg completes in 787 s. The matrix's honest content is the performance cliff and the cap-8/cap-32 boundary; the word "crash" in our issue should be read as "does not complete within 360 s", and the actual engine-failure reports were only ever the two bisect legs.

The deltas between your run and ours (you said the difference is the interesting part, and we agree):

• Gateway in the path: ours goes through the OpenAI-compatible Python layer, which spawns the engine and pipes its stderr; you drove the serve protocol directly.
• CPU: i5-7300HQ, 4c/4t, Windows 10 x64, CPU-only, 16 GB with ~7.3 GB free at leg start.
• Tokenizer density: our French probe at 6 668 chars tokenizes to 2 291 tokens (~2.9 c/tok); your 6 048-char prompt at 1 204 tokens (~5 c/tok). Your "well past the ~1024 threshold" leg was roughly half the prefill work of ours — and ours still survived today, which is why we now believe the exit mode needs more than prompt length to trigger.

Next step on our side: when the #1445 derived default lands in a release, we will re-run the explicit --cap 8 leg with the same instrumented harness and report the artifacts here — and if the exit mode recurs before that, the exit code and stderr will finally be on record. The repro is scripted and journaled, so it is one command.

Thank you for #1445 and #1446 — merged with CI assertions before we even came back to this thread. The RAM_GB report was the easiest one to write because the defect shape was already named in #855; the credit for that precedent is the whitelist comment sitting right above the three-family list.

[assistantia-info]

The re-run you asked for is done — same instrumented harness (client timeout 900 s, gateway console captured, health/profile sampled after each leg, explicit PILOT=0, Event Viewer drained), same machine, same model, same 6 668-char probe (2 291 prompt tokens), plus one addition: a leg at the cap #1445's formula derives for this box.

One caveat first, in the interest of an honest test: #1445 lives on dev (merge 23cd9f5) and no release carries it yet (latest is v1.10.2), and this Windows box has no C toolchain to build from source. But the change adds exactly one branch — if (cap <= 0) — and leaves the explicit-cap path untouched (git diff v1.10.1..v1.10.2 -- c/olmoe.c is empty; the derivation lives strictly inside the new branch). So "v1.10.1 binaries + explicit --cap 8" exercises the same code path your question is about.

Explicit --cap 8, PILOT=0, post-#1445 (by code identity):

• 6 668-char probe (2 291 tokens) → HTTP 200 in 675 s. Engine alive the whole way: 13 forwards, 12 completion tokens, gateway health 200 afterwards, wrapper process alive, RAM available 7.6 → 4.5 GB during the leg, engine_diag never fired. The same hang mode as our previous instrumented run (787 s) — your while (lru < 0) prediction, again.
• 9 915-char escalation (2 478 tokens) → still prefilling at the 900 s client cutoff. Gateway health 200, wrapper alive, zero engine-termination lines in the console, Event Viewer 6/6 Information. No exit, so no exit code; no stderr to show — question 2 stays honestly empty, the engine never exited under instrumentation.

The derived default, emulated. With no --ram, your formula computes budget = resident + 0.88 × OS-available at model-init time. Measured at the decision point on this box (fresh engine, dense resident): 1.8 GB resident, 5.7 GB available → budget 6.9 GB, projected KV 1.07 GB, 6.3 MB per expert slot → cap 34. That lands in the surviving band of our matrix (cap 32 completes the long leg, cap 8 does not), so on an idle 16 GB box the new default should put this machine on a working configuration — your "without anyone having to know the lever" outcome, one notch below all-resident because the box runs other services.

An operational caveat your formula may want to know, measured by accident: our derived-cap leg ran while a 10 GB game had started on the box (97 % memory load, 0.4 GB available). Under that pressure the same formula derives a cap near 1 — the reload-storm configuration — because mem_available_gb() samples the OS at decision time. On a shared machine the derived default is sensitive to ambient load in a way a constant 8 never was; a floor, or the launcher passing a --ram hint, would protect it. We could not complete a clean derived-cap leg today for that reason (the game holds the RAM; tasklist shows no orphaned engine — the stops do kill the engine tree, verified after we suspected one).

Artifacts: full journal (both legs + the derivation arithmetic), a second journal for the aborted clean-leg attempt, console logs — same footprints as before, any of them shareable raw.

[assistantia-info]

The clean derived-cap leg is done. The game that broke the previous window is closed; the box was idle except for a browser (9.8 GB available at start, 38 % memory load, no game process in the top). Same instrumented harness, same machine, same model: explicit PILOT=0, ctx 4096, client timeout 900 s, health/profile sampled after each leg, gateway console captured.

Derived cap 34, idle box, PILOT=0:

• 6 668-char probe (2 291 prompt tokens) → HTTP 200 in 353 s. The same probe took 675 s at explicit --cap 8 on the same box earlier today: essentially twice as fast, and the engine was alive the whole way again — 13 forwards, 12 completion tokens, gateway health 200 afterwards, wrapper alive, RAM available 9.8 → 4.2 GB across the leg.
• 9 915-char escalation (2 750 prompt tokens) → HTTP 200 in 462 s, natural stop (finish_reason: "stop", 7 tokens out, profile turn recorded: wall 462.1 s, 8 forwards). This is the leg that never completes at cap 8 — under cap 8 it was still prefilling at the 900 s client cutoff, with the box collapsing to 0.3 GB available and the prefill rate degrading below 1.3 tok/s. At the derived cap it finishes with 4.0 GB still available. Same prompt class, same gateway, same instrumentation; the only difference is the cache cap.
• Full leg 13.9 min wall including both stops; canonical stop killed the engine tree cleanly, port dead, no orphaned engine afterwards.

What this closes. The operational caveat from our previous comment: we could not complete a clean derived-cap leg earlier today because the game held the RAM. It is now completed, and it lands where the matrix predicted — cap 34 is inside the surviving band (cap 32 completes the long leg, cap 8 does not), so the default your formula derives for this box in its steady operating state (1.84 GB resident, ~5.7 GB available at the decision point, budget 6.9 GB) puts the machine on a working configuration, confirmed by a clean run rather than by extrapolation. And at the clean leg's own decision point the arithmetic is even more favourable: 1.83 GB resident + 0.88 × 7.95 GB available → budget 8.8 GB, i.e. the derivation lands deeper in the surviving band the quieter the box is — your "without anyone having to know the lever" outcome, on a box that also runs other services.

Two footnotes, both honest about their limits:

• The profile breakdown reads expert_disk_s: 0.0 on every leg of this run too — same serve-mode telemetry gap as filed in [olmoe] serve-mode PROF hardcodes the five phase timings — expert_disk_s reads 0.0 while the experts are the workload #1449, so the timings above come from the client clock and the gateway's own console, not from the phase counters.
• During the earlier loaded window today, the ~10 GB game coexisted with a cap-34 engine for the duration of a leg and the game stayed fluid — un-instrumented, anecdotal, but consistent: on this 4c/4t box the contended resource is RAM, not CPU. That reinforces rather than contradicts the ambient-load caveat: the failure mode of the derived default is memory pressure sampled at decision time (cap collapsing toward the reload-storm band), not runtime CPU contention.

Artifacts: clean-leg journal (decision point, both legs, post-mortems, stop sequence), same footprints as before — shareable raw.

[JustVugg]

Thank you for the instrumented re-runs, and for the honesty about the game holding memory in the earlier window. Between your three legs the picture is now clear enough to act on, and it is different from the title.

The engine does not die. Your own numbers say so: 2 291 prompt tokens, HTTP 200 in 787 s at cap 8, thirteen forwards, gateway health 200 afterwards, wrapper alive. What killed the first attempt was the client timeout, not the process. That matters because "dies" and "is twenty times slower than it should be" want different fixes, and you found the second one.

The fix you measured is already on dev. Your derived-cap leg is the important datapoint: the same 6 668-character probe took 353 s at the derived cap 34 against 675 s at explicit cap 8 on the same idle box. That is #1445, which replaced the constant eight slots per layer with a cap derived from the RAM budget. You were right that it is one branch (if (cap <= 0)) and that the explicit-cap path is untouched.

It is on dev and no release carries it yet, which is exactly the gap you named. A release is being cut from dev now, so the next one carries it along with #1443's sentinel work. On your box that should turn the default path into the 353 s case rather than the 675 s one, with no flags.

What I am doing about the rest. The telemetry complaint you and #1449 both raised is half fixed in #1460: olmoe now measures expert_disk_s instead of publishing a literal zero, so the next matrix you run will show where the time actually goes rather than a field that reads zero while the disk is the workload. The other four phase fields stay zero because they are genuinely unmeasured, and a guessed number would be worse.

Retitling this in my head as "default cap makes long prefill pathologically slow", which #1445 addresses. I would like to keep it open until you can confirm on a release build rather than close it on my own reasoning. When the release lands, please re-run the 6 668-character probe with no flags and post the wall time.