-
Notifications
You must be signed in to change notification settings - Fork 53
Open
Description
When trying to initialize the Gemma 3 4B model (gemma-3-4b-it-Q8_0.gguf) with the mmproj file (e.g., mmproj-BF16.gguf) on a MacBook Pro 16” M1 Max (32GB RAM), the initialization times out with:
TimeoutException (TimeoutException: Operation "model loading" timed out)
Without the mmproj, the model loads fine and can process normal text prompts, but it does not process images, which is expected.
To Reproduce:
final modelParams = ModelParams()..nGpuLayers = -1;
final contextParams = ContextParams()
..nPredict = -1
..nCtx = 4096
..nBatch = 1024;
final samplerParams = SamplerParams()
..temp = 0.25
..topP = 0.90;
final loadCommand = LlamaLoad(
path: "PATH_TO_MODEL",
modelParams: modelParams,
contextParams: contextParams,
samplingParams: samplerParams,
mmprojPath: "PATH_TO_MMPROJ_MODEL",
);
final parent = LlamaParent(loadCommand);
await parent.init(); // <- TimeoutException occurs here
The code looks similar to the given examples, so i don't know what I'm doing wrong.
I got the models here
Expected behavior
The model should load successfully with the mmproj file and be able to process image prompts.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels