The pinned oracle cannot complete engine start-up for the NemotronH NVFP4 checkpoint on dgx:gpu0: ~76 GB of host RAM disappears at the first forward, and a configuration that passed on 2026-08-18 no longer reproduces

[localai-bot]

The pinned vLLM oracle (0.1.dev1+g555967922, the FlashInfer-only wheel) loads
the 20.1 GiB NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 checkpoint on
dgx:gpu0 and is then killed by a host-memory watchdog before it generates a
single token, on every configuration tried. Five runs, 2026-08-19/20, all
inside an rc lease, box idle at start (117 GB MemAvailable, no compute apps).

The five runs

run	configuration	min MemAvailable
20260820T002359Z	reference + max_logprobs=64	12597 MB
20260820T024243Z	+ enforce_eager=True, max_model_len=256, max_num_seqs=1	19433 MB
20260820T025540Z	+ num_gpu_blocks_override=8	19797 MB
20260820T030818Z	exactly oracle_only.sh attempt a	13941 MB
20260820T031644Z	+ kv_cache_memory_bytes=4 GiB	14846 MB

Logs under /workspace/a2q1-neartie/.

What the runs EXCLUDE

The record has carried this as "the step after compilation — profiling forward
or graph capture". These runs separate those:

• Not torch.compile. Run 5 hit the AOT cache: torch.compile took 0.30 s in total, Directly load AOT compilation from path .... Collapsed anyway.
• Not CUDA graph capture. Run 2 ran enforce_eager=True, so there is no
  capture in the process at all. Collapsed anyway.
• Not KV cache sizing. Run 3 overrode the block count to 8; run 5 set an
  absolute kv_cache_memory_bytes=4 GiB, which by its own log message
  "skipped memory profiling. This does not respect the gpu_memory_utilization
  config". Both collapsed, and at the same magnitude.
• Not another tenant. free -m at start: used 5045, available 117456;
  nvidia-smi --query-compute-apps empty.

What the shape says

Every run has the same signature. Host memory sits flat through the whole weight
load, then falls ~76 GB in ~60 s at a steady ~1.2 GB/s, beginning right after

Model loading took 17.85 GiB memory and 389.68 seconds
Setting attention block size to 4176 tokens to ensure that attention page size is >= mamba page size
Padding mamba page size by 0.19% ...

A steady linear climb of that size, invariant under every KV knob and present
with compile and capture both removed, is consistent with the NVFP4 Marlin
repack at the first forward — which the record already names as the path behind
every collapse on this box, and which a bf16 model does not have:

Your GPU does not have native support for FP4 computation but FP4 quantization
is being used. Weight-only FP4 compression will be used leveraging the Marlin kernel.
Using 'MARLIN' NvFp4 MoE backend out of potential backends: [...]

The profile forward runs regardless — gpu_worker.py:465-468 calls
profile_run() even when kv_cache_memory_bytes is set — so no engine knob
tried here avoids that first forward. At this pin there is also no environment
knob to force a different NvFp4 MoE backend: the selection walks
AVAILABLE_BACKENDS and returns the first supported
(fused_moe/oracle/nvfp4.py:275-292), and MARLIN is what a device without
native FP4 supports.

Why this is a REGRESSION and not a known limit

Run 4 is byte-for-byte the configuration of /workspace/nhspeed/oracle_only.sh
attempt a, which on 2026-08-18 bottomed out at minMemAvailable_MB=51528
and generated all three prompts (ORACLE TOKEN MATCH: 180/192). Same box,
same staged checkpoint, same wheel, same knobs. Today it reaches 13941 MB.

So a configuration this repository has a passing measurement for no longer
reproduces, and the ~37 GB difference is unexplained. That matters beyond this
row: the 2026-08-18 result is the evidence behind "the oracle DOES run a model
in a lease", and it is not currently reproducible.

Consequence

Any row needing an oracle-side number on this checkpoint is blocked on GB10.
Concretely it blocks the top-2 margin measurement
#1388 needs to decide whether
the A2-Q1 device mamba arm's one moved token is a near-tie or a defect, which is
what holds #1289.

Note the watchdog is doing its job and the box survived all five runs; the
process-group kill and its postcondition (leftover python under this job: 0,
memory returned to 96.8 GB, no compute apps) are verified rather than assumed.
Do not simply lower the floor — this box reboots rather than OOM-kills.

Next traceable step

Establish what changed on the host since 2026-08-18, before tuning anything
else: the 51528 MB run is the control and it is the thing that stopped working.