Misc. bug: Q4_0 repacking results in double RAM usage

[bartowski1182]

Name and Version

b4792

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-cli

Command line

./llama-cli -m microsoft_Phi-4-mini-instruct-Q4_0.gguf

Problem description & steps to reproduce

When loading the model, it uses 4.3GB of RAM

When using Q4_K_S (similar size) it only uses 2.7GB of RAM

First Bad Commit

No response

Relevant log output

[bartowski1182]

Best guess is there's a missing "free" after we've repacked the weights, and the original weights are kept in memory accidentally

[slaren]

Some parts of the model file will remain mapped. The same thing happens when partially offloading a model, in practice it is not likely to cause issues because the OS can reclaim that memory if necessary. Disabling mmap with --no-mmap should avoid this.

[bartowski1182]

Ah okay yes I see the RAM usage drop when using that option.. no performance concerns I assume when using it? Thanks for the speedy response!

[slaren]

It may affect model loading time, but it will not affect inference performance.