Eval bug: llama-server OOM Over Time

[shizhx]

Name and Version

version: 10568 (f974005)
built with GNU 14.2.0 for Linux x86_64

Operating systems

Linux

GGML backends

CUDA

Hardware

RTX 2080 Ti 22GB

Models

https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

chat-template:
https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates

Problem description & steps to reproduce

OOM during a long context task

preset.ini

[*]
host = 0.0.0.0
port = 3000
threads = 8
parallel = 1

[qwen3.8-27b]
model = /home/user/Models/Qwen3.8-27B/Qwen3.8-27B-UD-Q4_K_M.gguf
load-mode = dio
n-gpu-layers = 999
cache-type-k = q8_0
cache-type-v = turbo4
flash-attn = true
kv-unified = true
batch-size = 2048
ubatch-size = 256
reasoning=true
reasoning-budget=65536
reasoning-format = deepseek
reasoning-preserve = true
temp = 1.0
top-p = 0.95
top-k = 20
min-p = 0.0
presence-penalty = 0.0
repeat-penalty = 1.0
ctx-size = 163840
spec-type = draft-mtp
spec-draft-n-max = 2
spec-draft-type-k=q4_0
spec-draft-type-v=q4_0
jinja = true
chat-template-file = /home/user/Models/chat_template.jinja
mmproj = /home/user/Models/Qwen3.8-27B/mmproj-F16.gguf
no-mmproj-offload = true
mtmd-batch-max-tokens = 512
image-min-tokens = 1024

initial VRAM usage:

First Bad Commit

No response

Relevant log output

Logs

[48663] /home/user/Source/llama-cpp-turboquant/ggml/src/ggml-cuda/ggml-cuda.cu:111: CUDA error
[48663] 7.01.164.113 E CUDA error: out of memory
[48663] 7.01.164.116 E   current device: 0, in function alloc at /home/user/Source/llama-cpp-turboquant/ggml/src/ggml-cuda/ggml-cuda.cu:621
[48663] 7.01.164.116 E   create_result
[48663] /home/user/Source/llama-cpp-turboquant/build/bin/libggml-base.so.0(+0x199a5) [0x7f2fd89349a5]
[48663] /home/user/Source/llama-cpp-turboquant/build/bin/libggml-base.so.0(ggml_print_backtrace+0x1df) [0x7f2fd8934d7f]
[48663] /home/user/Source/llama-cpp-turboquant/build/bin/libggml-base.so.0(ggml_abort+0x11e) [0x7f2fd8934f0e]
[48663] /home/user/Source/llama-cpp-turboquant/build/bin/libggml-cuda.so.0(+0x267553) [0x7f2fcda67553]
[48663] /home/user/Source/llama-cpp-turboquant/build/bin/libggml-cuda.so.0(_ZN18ggml_cuda_pool_vmm5allocEmPm+0x380) [0x7f2fcda80dd0]
[48663] /home/user/Source/llama-cpp-turboquant/build/bin/libggml-cuda.so.0(_Z12launch_fattnILi256ELi4ELi8EEvR25ggml_backend_cuda_contextP11ggml_tensorPFvPKcS5_S5_S5_S5_PKiPfP6float2ffffjfi5uint3iiiiiiiiiiiliiliiiiilEimibbbi+0xb05) [0x7f2fcdccc145]
[48663] /home/user/Source/llama-cpp-turboquant/build/bin/libggml-cuda.so.0(_Z37ggml_cuda_flash_attn_ext_mma_f16_caseILi256ELi256ELi4ELi8EEvR25ggml_backend_cuda_contextP11ggml_tensor+0x1d0) [0x7f2fcdea3fc0]
[48663] /home/user/Source/llama-cpp-turboquant/build/bin/libggml-cuda.so.0(+0x27d017) [0x7f2fcda7d017]
[48663] /home/user/Source/llama-cpp-turboquant/build/bin/libggml-base.so.0(+0x38eb7) [0x7f2fd8953eb7]
[48663] /home/user/Source/llama-cpp-turboquant/build/bin/libllama.so.0(_ZN13llama_context13graph_computeEP11ggml_cgraphb+0xa1) [0x7f2fd80e3611]
[48663] /home/user/Source/llama-cpp-turboquant/build/bin/libllama.so.0(_ZN13llama_context14process_ubatchERK12llama_ubatch14llm_graph_typeP22llama_memory_context_iR11ggml_status+0xea) [0x7f2fd80e734a]
[48663] /home/user/Source/llama-cpp-turboquant/build/bin/libllama.so.0(_ZN13llama_context6decodeERK11llama_batch+0x388) [0x7f2fd80ee1b8]
[48663] /home/user/Source/llama-cpp-turboquant/build/bin/libllama.so.0(llama_decode+0xb) [0x7f2fd80efd6b]
[48663] /home/user/Source/llama-cpp-turboquant/build/bin/libllama-common.so.0(_ZN33common_speculative_impl_draft_mtp7processERK11llama_batch+0x9cf) [0x7f2fd86dc26f]
[48663] /home/user/Source/llama-cpp-turboquant/build/bin/libllama-common.so.0(_Z26common_speculative_processP18common_speculativeRK11llama_batch+0x30) [0x7f2fd86cb400]
[48663] /home/user/Source/llama-cpp-turboquant/build/bin/libllama-server-impl.so(_ZN19server_context_impl6decodeERiiR11llama_batch+0x229) [0x7f2fd91a4669]
[48663] /home/user/Source/llama-cpp-turboquant/build/bin/libllama-server-impl.so(_ZN19server_context_impl12update_slotsEv+0x512) [0x7f2fd91aa8f2]
[48663] /home/user/Source/llama-cpp-turboquant/build/bin/libllama-server-impl.so(_ZN12server_queue10start_loopEl+0x1f1) [0x7f2fd9148951]
[48663] /home/user/Source/llama-cpp-turboquant/build/bin/libllama-server-impl.so(_Z12llama_serverR13common_paramsiPPc+0x420b) [0x7f2fd90ecabb]
[48663] /home/user/Source/llama-cpp-turboquant/build/bin/libllama-server-impl.so(_Z12llama_serveriPPc+0x1060) [0x7f2fd90ee8c0]
[48663] /lib/x86_64-linux-gnu/libc.so.6(+0x29ca8) [0x7f2fd8a35ca8]
[48663] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7f2fd8a35d65]
[48663] /home/user/Source/llama-cpp-turboquant/build/bin/llama-server(+0x11b1) [0x557b88ab21b1]