Skip to content

README.md

Latest commit

 

History

History
207 lines (151 loc) · 6.71 KB

README.md

File metadata and controls

207 lines (151 loc) · 6.71 KB
Raw

Apple M4 Pro 48GB: Local LLM Inference Benchmark

Benchmarking local LLM inference on Apple M4 Pro, comparing LM Studio vs Ollama backends across multiple models and context sizes.

Hardware

Spec Detail
Chip Apple M4 Pro
CPU 14-core (10P + 4E)
GPU 20-core
Memory 48GB Unified
Memory Bandwidth ~273 GB/s
OS macOS Sequoia (Darwin 25.2.0)

Models Tested

Model Type Params (active) Size Backend
Qwen3-Coder-30B-A3B MoE 30B (3B) ~18GB MLX (both)
Qwen3.5-35B-A3B MoE 35B (3B) ~24GB GGUF (llama.cpp)*
Gemma 4 26B-A4B MoE 26B (4B) ~18GB GGUF (llama.cpp)*

* Qwen 3.5 and Gemma 4 MLX backends are not yet supported in Ollama/LM Studio as of April 2026. These models fall back to llama.cpp with Metal acceleration.

Results Summary

Generation Speed (tok/s) — All Models

Model LM Studio Ollama Notes
Qwen3-30B-A3B 79.0 65.9 Both MLX
Gemma 4 26B-A4B — 59.5 (short) / 56.4 (long) GGUF, Ollama only
Qwen3.5-35B-A3B — 27.5 (short) / 28.5 (long) GGUF, Ollama only

Time to First Token — All Models

Model LM Studio Ollama
Qwen3-30B-A3B 0.179s 0.087s
Gemma 4 26B-A4B — 0.210s
Qwen3.5-35B-A3B — 0.410s

Detailed Results

1. Qwen3-Coder-30B-A3B — LM Studio (MLX) vs Ollama (MLX)

Both backends use MLX. This is the purest comparison of LM Studio vs Ollama overhead.

Prompt Metric LM Studio Ollama Winner
Short (~100 tok) Gen tok/s 87.3 64.5 LM Studio +35%
TTFT 0.17s 0.09s Ollama +47%
Medium (~500 tok) Gen tok/s 83.3 68.2 LM Studio +22%
TTFT 0.17s 0.08s Ollama +51%
Long (~2K tok) Gen tok/s 79.3 65.7 LM Studio +21%
TTFT 0.15s 0.08s Ollama +43%
Long ctx (~2K in) Gen tok/s 66.2 65.2 ~Tie
TTFT 0.23s 0.09s Ollama +61%

Verdict: LM Studio generates 20% faster, Ollama responds 51% sooner.

2. Gemma 4 26B-A4B — Ollama (GGUF/llama.cpp)

Prompt Gen tok/s TTFT Prompt eval tok/s Total time
Short (~100 tok) 59.5 0.21s 138 2.73s
Medium (~500 tok) 58.0 0.21s 183 14.00s
Long (~2K tok) 56.4 0.21s 223 36.55s
Context 4K 53.3 0.23s 21,020 9.84s
Context 8K 50.8 0.25s 37,972 10.34s
Context 16K 45.5 0.29s 66,870 11.54s
Context 32K 47.8 70.15s 467 74.23s

Key observations:

  • Very consistent ~56-59 tok/s generation across short-to-long outputs
  • Excellent prompt processing up to 16K context (67K tok/s at 16K!)
  • 32K context causes a dramatic cliff: TTFT explodes to 70s, prompt eval drops from 67K to 467 tok/s. This suggests the KV cache exceeds available fast memory at 32K.
  • Only generated 195/512 tokens at 32K context (model struggles with the filler content)

3. Qwen3.5-35B-A3B — Ollama (GGUF/llama.cpp)

Prompt Gen tok/s TTFT Prompt eval tok/s Total time
Short (~100 tok) 27.5 0.41s 83 5.87s
Medium (~500 tok) 26.7 0.37s 111 30.57s
Long (~2K tok) 28.5 0.42s 119 72.40s
Context 4K 28.0 4.18s 1,053 22.49s
Context 8K 27.8 0.54s 16,410 18.99s
Context 16K 25.6 1.97s 8,995 21.97s
Context 32K 9.3 85.11s 386 140.17s

Key observations:

  • Consistent ~27-28 tok/s generation (about half of Gemma 4, despite similar MoE architecture)
  • Running on GGUF/llama.cpp — Ollama's MLX backend does not yet support Qwen 3.5 architecture
  • 32K context is painful: 85s TTFT, generation drops to 9.3 tok/s
  • 8K prompt eval is surprisingly fast (16K tok/s) — likely KV cache hit from warmup

Context Window Scaling

How TTFT scales with input context size:

Context    Gemma 4 26B        Qwen3.5 35B
   4K      0.23s              4.18s
   8K      0.25s              0.54s *
  16K      0.29s              1.97s
  32K      70.15s !!          85.11s !!

* 8K TTFT anomalously low for Qwen3.5 — likely KV cache reuse from warmup run.

Both models hit a wall at 32K context on 48GB. The KV cache for 32K tokens at these model sizes likely exceeds the available unified memory after model weights are loaded, causing heavy swapping.

Practical context limit on M4 Pro 48GB: ~16K tokens for these 18-24GB models.

Analysis

MLX vs llama.cpp on Apple Silicon

Backend Best for Example
MLX Supported models (Qwen3) 79 tok/s gen, 87ms TTFT
llama.cpp/GGUF Broader model support 28-59 tok/s gen, 200ms+ TTFT

MLX delivers 2-3x better performance when the model architecture is supported. As of April 2026, MLX in Ollama only supports Qwen3/3.5-A3B (coding variant) and Gemma 3.

Model Comparison on Ollama

Gemma 4 26B Qwen3.5 35B
Gen speed 2x faster Slower (GGUF)
TTFT 2x faster Slower
Context scaling Better up to 16K Similar cliff at 32K
Model quality Strong general Latest Qwen reasoning

Recommendations

Use case Best choice
Fast interactive chat Qwen3-30B via LM Studio MLX
Coding assistance Qwen3-Coder-30B via LM Studio MLX
General tasks (no MLX) Gemma 4 26B via Ollama
Large context (up to 16K) Gemma 4 26B — best TTFT scaling
Maximum context (32K+) Needs more RAM or smaller model

Reproduce

Quick start

# Run all benchmarks automatically
./run.sh

# List available model configs
./run.sh --list

# Test specific model
./run.sh --models 0 --runs 3

# Custom context sizes
./run.sh --context "4,8,16,32,64"

# Ollama only (skip LM Studio)
./run.sh --skip-lmstudio

# Quick test
./run.sh --models 0 --prompts short --runs 1

Manual benchmark

python3 benchmark.py \
  --ollama-model "gemma4:26b" \
  --context-sizes "4,8,16,32" \
  --runs 3 \
  --output results/my_test.json

Prerequisites

  • Mac with Apple Silicon (32GB+ unified memory)
  • Ollama (brew install ollama)
  • LM Studio (optional)
  • Python 3.10+ with httpx (pip install httpx)

Files

File Description
benchmark.py Python benchmark engine
run.sh Automated orchestrator
results/ JSON result files per model
results.json Legacy Qwen3 LM Studio vs Ollama results

License

MIT