moe-cache: adaptive cpu-overlap calibration to replace fixed byte budget - #327
Merged
TheTom merged 1 commit intoAug 31, 2026
Merged
Conversation
The automatic --moe-cache overlap policy picked how many GPU-resident expert rows to route to the CPU using a hardcoded 8 MiB/token constant tuned on a 4x RTX 3090 rig. On a single RTX 5090 with limited CPU threads that assumption is wrong in the opposite direction: any CPU overlap adds latency instead of hiding it, measured +7% decode throughput at overlap=0 vs the old automatic default, and -4% at max overlap. Replace the fixed constant with a live calibration that runs at session start: a long unmeasured global warm-up (JIT, page cache, governor settling all swamp the real signal if skipped), then a round-robin sweep over candidate row counts -- including the original formula as one candidate -- averaged across multiple rounds so residual drift doesn't bias whichever candidate runs first. Locks onto the best-measured candidate for the rest of the session. Only active when overlap_cpu_rows is left automatic; explicit GGML_CUDA_MOE_CACHE_OVERLAP_CPU_ROWS still takes priority untouched. Validated end-to-end against a real 118B-A8B MoE model: converges to rows=0 on this hardware, matching the manually-discovered optimum (37.1-37.7 tok/s post-convergence vs 34.4 tok/s for the old fixed formula). Adds a dedicated unit test driving full convergence; all existing moe-cache tests still pass unchanged. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
This is definitely increasing token generation speeds for me, running DSv4 Flash UD-Q3_K_XL on an RTX 3090, RTX 3060, 128GB DDR5 and a Ryzen 7600. These results are taken from the same Hermes session, build switched between turns. Around 100k context. So the prompt isn't completely identical before and after but it's very close. Before: After: Appreciate these aren't precise like for like tests, but it's enough to show this PR is clearly faster in real world use. Happy to do more specific benchmarks if you need. |
Owner
|
Thanks, @giveen. I reviewed the calibration state machine and policy switch. The warm-up plus rotated round-robin design avoids the first-pass bias you found, explicit overrides still bypass calibration, short or batched sessions safely retain the old formula, and the chosen row count locks after one sweep. We also now have independent real-world confirmation from @stiggy2k16 on a 3090 + 3060 setup at roughly 100K context, which is useful coverage beyond the 5090. The red server check is the unrelated slot-restore crash fixed in #324. Merging this. |
TheTom
merged commit Aug 31, 2026
daa4a32
into
TheTom:feature/turboquant-kv-cache
10 of 28 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR: Adaptive cpu-overlap calibration for
--moe-cache autoReplaces a hardcoded byte-budget constant in the MoE expert cache's automatic
CPU/GPU overlap policy with a live calibration that measures real decode
throughput on the actual machine and picks the best setting itself, instead
of assuming a value tuned on different hardware.
Commit in this PR:
moe-cache: adaptive cpu-overlap calibration to replace fixed byte budgetBackground: what this actually fixes
When
--moe-cache autodecides how much GPU-resident expert work to also runon the CPU concurrently (the "overlap" policy — see
docs/backend/MOE-CACHE.md),it used a single compile-time constant:
moe_cache_overlap_bytes_per_token = 8 MiB(moe-cache.cu:56), tuned once via a 4x RTX 3090 sweep and shipped as-isfor every machine. There was no mechanism to measure whether that assumption
held on different hardware.
It doesn't hold on a single RTX 5090 with a modest CPU thread budget: the GPU
finishes its share fast enough that any CPU overlap work becomes the critical
path instead of hidden, extra throughput. The fixed formula was actively
hurting decode speed on this box, not just leaving performance on the table.
Test environment
RTX 5090 (32 GB VRAM, single GPU), Laguna-S-2.1 118B-A8B MoE model, Q5_K_M
(~91 GB, routed experts spill to CPU host memory), 8 CPU threads pinned
(
-t 8 -Cr 0-7 -tb 8 -Crb 0-7),--no-mmap,--fit on -fa on.1. Motivating measurement — manual sweep, forcing the existing
GGML_CUDA_MOE_CACHE_OVERLAP_CPU_ROWSoverrideBefore writing any new code, confirmed the fixed formula was actually
mistuned on this hardware by manually overriding the row count via the
already-existing (undocumented-in-CLI) env var, 3 completion requests per
setting, first request per server discarded as warmup:
autooverlap=0(forced)auto(old fixed 8 MiB/token formula)overlap=8(forced, max)Clean, monotonic — less CPU overlap is strictly better on this hardware. This
justified building an automatic calibration rather than just retuning the
constant, since the "right" answer here (zero) isn't expressible by the old
formula's design (it always hands over at least 1 row).
2. A real methodology bug caught and fixed before shipping
First calibration implementation tried each candidate sequentially in one
block (4 warmup scopes, 8 measured scopes, next candidate), no global
warmup. Result was wrong and worth recording as a cautionary log:
That's a monotonic warm-up decay curve (CUDA kernel JIT, page cache fill,
thread/frequency governor settling), not a signal from the row count —
whichever candidate runs first in a sequential sweep is unfairly penalized.
Fixed by adding a 64-scope unmeasured global warm-up before any candidate
timing starts, plus a round-robin sweep (each candidate gets several short
trials across rounds, rotating which candidate goes first each round) instead
of one long block per candidate. Re-run with the fix:
No more monotonic drift — values bounce around per candidate as expected from
real noise — and it converges to
rows=0, matching the manual sweep above.3. End-to-end validation against the real model
autooverlap=0The calibration log confirms it:
cpu-overlap calibration complete: rows=0 (27.211 ms/decode avg)— reached automatically, no manual tuning. Post-convergence throughput (37.1–37.7 tok/s) matches, and slightly beats, the
manually-discovered optimum, and beats the old automatic default by ~9%. The
one-time calibration cost (~154 decode steps, a few seconds) is paid once per
server process, not per request.
4. Unit tests
All 35 existing
test-moe-cachecases still pass unchanged (2 skipped forsingle-GPU/static-backend reasons, unrelated to this change). Added a
dedicated test,
cache-cpu-overlap-calibration, that drives 170 single-tokendecode scopes through a real session and asserts: the global warm-up
completes, exactly 18 round×candidate trial lines get logged (3 rounds × 6
candidates), and calibration reaches "complete" exactly once and stays locked
in afterward — so a regression to the warm-up length, the round-robin loop,
or a runaway re-triggering bug fails this test independent of real hardware
timing.
Known limitations
candidate set
{formula, 0, 1, 2, 4, 8}) are reasonable defaults for thisworkload, not swept against alternatives.
(where the config is shared across devices) is exercised by the existing
unit tests but not validated on real multi-GPU hardware.
workload that restarts very frequently with very short sessions, that cost
may not fully amortize (though it's a few seconds against typical model
load times of 15-30+ seconds for large MoE models).
Test commands
Unit tests:
cmake --build build --target test-moe-cache -j"$(nproc)" build/bin/test-moe-cacheManual override sweep (reproduces section 1):
Real calibration run (no override — reproduces sections 2/3):
AI usage: yes