HIP mixed TurboQuant vec FA on gfx900/gfx906 - #99

Open

2bigO wants to merge 175 commits into

TheTom:feature/turboquant-kv-cachefrom

2bigO:feature/turboquant-kv-cache

Commits on Apr 15, 2026

WIP: add TurboQuant KV cache types (turbo3, turbo4)

TheTom
and
claude
committed
0b37448
feat: Metal kernels for TurboQuant KV cache (turbo3, turbo4) #21

TheTom
and
claude
committed
b37e5f0
feat: full TurboQuant with rotation matrices in Metal kernels #21

TheTom
and
claude
committed
63e25a5
feat: inline rotation matrices in Metal shader + C round-trip test #21

TheTom
and
claude
committed
f40ccae
fix: remove thread static from Metal dequantize, fix stale code #23

TheTom
and
claude
committed
d10ca51
feat: replace dense 128x128 matvec with Fast Walsh-Hadamard rotation #26

TheTom
and
claude
committed
3777275
docs: detailed speed investigation plan for TurboQuant Metal shader #23

TheTom
and
claude
committed
fa52236
docs: log simd_broadcast attempt — no speed improvement #23

TheTom
and
claude
committed
8cf626a
docs: log threadgroup attempt — no speed improvement, rethinking #23

TheTom
and
claude
committed
2448468
docs: CRITICAL — dequant is NOT the bottleneck, no-op still 2.4 tok/s #23

TheTom
and
claude
committed
d675e12
fix: inline turbo-wht.h — was causing CPU fallback, not Metal! #23

TheTom
and
claude
committed
adfbc59
docs: real Metal benchmarks after #include fix — 8× gap not 35× #23

TheTom
and
claude
committed
e8f23eb
docs: final investigation summary + upstream tracking #23 #27

TheTom
and
claude
committed
447ca79
docs: upstream competitive intel — pre-rotate-queries is the key #28

TheTom
and
claude
committed
5787361
docs: speed ceiling test — 49 tok/s without dequant rotation (4.6× gain) #23

TheTom
and
claude
committed
786f0c9
docs: pre-rotate-queries implementation plan + speed ceiling 49 tok/s

TheTom
and
claude
committed
79c08dd
feat: pre-rotate-queries optimization — 51.4 tok/s (5× speedup) #23

TheTom
and
claude
committed
808e171
docs: final investigation summary — 2.4 → 51.4 tok/s journey complete #23

TheTom
and
claude
committed
ac157ee
feat: MSE-only mode — drop QJL, all 3 bits to PolarQuant #23

TheTom
and
claude
committed
d044965
docs: Change 2 not needed — Q rotation overhead is negligible

TheTom
and
claude
committed
068715c
docs: block size is the bottleneck — q4_0 at block 32 = 100% of q8_0

TheTom
and
claude
committed
c551be4
feat: block size 32 — 77.7 tok/s MoE (91% of q8_0), 17.0 Qwopus (97%) 🎉

TheTom
and
claude
committed
fff2092
fix: TURBO_D=128 independent of QK_TURBO3, file turbo4 bugs #29

TheTom
and
claude
committed
6a52a43
docs: final investigation log — 77.7 tok/s, 91% of q8_0

TheTom
and
claude
committed
2462ab3
CRITICAL: turbo3 perplexity is 165.6 vs q8_0 6.1 — quality broken #30

TheTom
and
claude
committed
b4270ec
CRITICAL: found TWO root causes for PPL=165 #30

TheTom
and
claude
committed
8755e26
docs: bisect confirms block size innocent, rotation access is the bug #31

TheTom
and
claude
committed
b0234bd
fix: restore inverse rotation in dequant — PPL 6.19 (1.2% of q8_0) #31 #30

TheTom
and
claude
committed
ebe0a23
docs: perplexity 6.194 confirmed — 1.4% of q8_0 #30

TheTom
and
claude
committed
091e12a
docs: complete quality benchmark summary + lessons learned #30

TheTom
and
claude
committed
538e5af
perf: fp16 WHT dequant + SIMD cooperative dequant — 45% speedup

TheTom
and
claude
committed
7300255
chore: move turboquant docs to turboquant_plus repo

TheTom
and
claude
committed
095aa0b
perf: vectorized half4 WHT butterfly — 31% speedup (1074 → 1411 tok/s)

TheTom
and
claude
committed
557b033
perf: pre-packed half4 sign arrays — minor speedup (1411 → 1424 tok/s)

TheTom
and
claude
committed
cc31405
perf: graph-side WHT rotation — 2095 tok/s (0.78x q8_0, was 0.53x)

TheTom
and
claude
committed
4de7c1c
perf: block-32 + graph WHT — 2747 tok/s (1.02x q8_0!!!)

TheTom
and
claude
committed
e5b7470
feat: layer-adaptive KV cache — q8_0 quality with 80% turbo3 compression

TheTom
and
claude
committed
73f0008
fix: address Codex review on layer-adaptive — thread safety + underflow guard

TheTom
and
claude
committed
701e085
wip: context scaling fix — skip unnecessary ggml_cont + 32x32 rotation data

TheTom
and
claude
committed
6267548
experiment: group-32 rotation FAILED — PPL 7.06 (target 6.19)

TheTom
and
claude
committed
af0cb0a
feat: add GGML_OP_TURBO_WHT — custom O(d log d) Walsh-Hadamard Transform

TheTom
and
claude
committed
012faec
perf: optimized turbo3 dequant — eliminates context scaling regression

TheTom
and
claude
committed
7173be9
ci: quality+speed gate script — PPL + context scaling check before push

TheTom
and
claude
committed
8fa5bff
perf: fp16 centroid LUT — decode +6-14% at long context (#33)

TheTom
and
claude
committed
b696c5d
perf: float norm broadcast in vec dequant — decode +2-3% over fp16 LUT

TheTom
and
claude
committed
e398a4e
fix: add turbo3/turbo4 cache types to llama-bench arg parser

TheTom
and
claude
committed
5e6277b
experiment: split 2x4-entry constant LUT for M1 decode fix

TheTom
and
claude
committed
6c9cfb1
fix: Metal shader comment accuracy per Codex review

TheTom
and
claude
committed
d96703a
cleanup: remove stray diagnostic output files
TheTom
committed
33b3d8a
feat: turbo3 norm correction — PPL 6.211 → 6.176 (free quality win)

committed
440d324
fix: auto-enable flash attention for turbo cache types + fix ggml context overflow

TheTom
and
claude
committed
4bf7ddc
experiment: register centroid LUT tested — register spill on Metal

committed
884da3f
fix: turbo4 SET_ROWS corruption, tail-block truncation, constant coupling (Issue #29)

authored and
TheTom
committed
3eca09a
fix: stack overflow in turbo4 CPU init — 64KB array on worker thread stack

authored and
TheTom
committed
a5a0f7b
experiment: batched byte extraction + explicit bit field pre-extract

committed
06a6b62
experiment: profiling modes for turbo3 decode bottleneck isolation

TheTom
and
claude
committed
86a5bbf
experiment: 4-entry magnitude LUT + branchless sign (XOR trick)
TheTom
committed
baa6116
experiment: force non-vec FA path for turbo3 (nl=2 vs nl=8)

TheTom
and
claude
committed
1c2e558
experiment: zero-LUT select chain — 2-level ternary, no constant memory
TheTom
committed
c527333
feat: auto-detect hardware, use 4-mag LUT on pre-M5 (+38-45% decode)

committed
f19c98c
experiment: 2-pair half2 LUT — only 2 constant addresses per lookup
TheTom
committed
fbd5ec9
experiment: deferred norm multiply (batch float4 * norm at end)
TheTom
committed
7c2d880
revert to proven 4-mag + per-element norm (deferred norm was slower)

TheTom
and
claude
committed
35034f1
experiment: named-register centroid×norm — 4 constant reads upfront, zero divergence, ternary select
TheTom
committed
616c7b9
revert to 4-mag LUT (proven best), document all findings
TheTom
committed
73d512c
experiment: inline block processing — bypass template dequant in FA inner loop
TheTom
committed
bdcd8ec
experiment: inline block WORSE on M2 (-10-15%), reverted to 4-mag

TheTom
and
claude
committed
b596a5c
experiment: FULLY BRANCHLESS FMA decode — zero ternary, zero memory, zero branch
TheTom
committed
34f7c39
final: 12 approaches tested, 4-mag LUT is the hardware limit

TheTom
and
claude
committed
9637c1c
experiment: SIMD SHUFFLE magnitude select — cross-lane LUT replacement
TheTom
committed
5755740
experiment: simd_shuffle 14.7 at 8K — close to 4-mag (15.1) but not better

TheTom
and
claude
committed
927e68a
experiment: fused block dot — per-centroid Q accumulation, 4 constant reads total
TheTom
committed
4b44b2b
experiment: fused block dot 8.1 at 8K — worst result, 64 comparisons kill it

TheTom
and
claude
committed
d9ba9bf
experiment: 4-mag helps M5 at 16K (+2.4%) but hurts at 32K (-7.3%)

TheTom
and
claude
committed
b3cd6a7
experiment: M5 LUT cost grows to 34% at 32K context

TheTom
and
claude
committed
927cfc1
feat: sparse V dequant — +12% decode at 32K, zero quality loss

TheTom
and
claude
committed
26b0bcc
feat: sparse V dequant — +22% decode at 32K on M5, auto-enabled

TheTom
and
claude
committed
de44bfe
Revert "Merge pull request #4 from seanrasch/feature/turboquant-kv-cache"
TheTom
committed
687b184
experiment: dedicated turbo4 SET_ROWS kernel + prefill FA kernels

TheTom
and
claude
committed
3d40ac4
experiment: turbo4 2+1 bit packing — +33% decode, drop QJL

TheTom
and
claude
committed
cb4d495
experiment: direct-extract turbo4 dequant — matches turbo3 speed

TheTom
and
claude
committed
280c242
experiment: 4-bit half-precision centroid LUT for turbo4 vec path

TheTom
and
claude
committed
a56ed21
experiment: fix turbo4 struct for 4-bit — Codex-caught OOB bug

TheTom
and
claude
committed
b4b6a30
experiment: 8-mag LUT tested, reverted — direct 16-LUT faster on M5

TheTom
and
claude
committed
c4e98a5
experiment: add turbo4_dequant_f16 compute shader (prefill prep)

TheTom
and
claude
committed
7053f4d
feat: TURBO4_USE_4BIT ifdef for ABI compatibility

TheTom
and
claude
committed
e7ac14d
feat: complete 4-bit C reference for turbo4 — quantize + dequant

TheTom
and
claude
committed
f4d9d3e
fix: add turbo WHT rotation to ISWA build_attn — fixes Gemma 2

TheTom
and
claude
committed
5e954d4
feat: CUDA port of TurboQuant3 KV cache compression (RTX 5090 / SM 12.0)

authored and
TheTom
committed
4489686
perf: enable MMA/TILE flash attention for turbo3 — 0.97x q8_0 prefill

authored and
TheTom
committed
1127f5b
perf: parallel k_set_rows_turbo3 + optimise KQ/V dequant — +31% decode (71→94 t/s, 98.5% of F16)

authored and
TheTom
committed
9a4afd8
fix: VEC flash-attn Q/K stride mismatch in vec_dot_fattn_vec_KQ_turbo3_0

authored and
TheTom
committed
a17a63a
fix: graceful fallback for turbo3 with non-128-aligned head dims (issue #13)

authored and
TheTom
committed
e9ab045
fix: graceful fallback for turbo3 on non-128-aligned head dims (issue #13)

authored and
TheTom
committed
fb2d86d
feat: 64-element WHT groups + MLA Q rotation fix (issue #13)

authored and
TheTom
committed
d86034d
feat: mixed turbo3/q8_0 KV cache types (-ctk turbo3 -ctv q8_0 and vice versa)

authored and
TheTom
committed
567fadf
fix: implement CPU turbo3 quantize (was a stub that zeroed qs/signs)

authored and
TheTom
committed
f0601a7
feat: GGML_TYPE_TURBO2_0 — 2-bit TurboQuant KV cache (6.4x compression)

authored and
TheTom
committed
25a19f2
fix: MLA inverse WHT group_size derived from K (not V) — fixes GLM-4.7

authored and
TheTom
committed
52178bb
feat: InnerQ per-channel equalization + turbo2 64-group fallback

authored and
TheTom
committed
f3f7c3c
perf: sparse V dequant — skip negligible attention weights in VEC kernel

authored and
TheTom
committed
65a2c69
fix: require head_dim % 128 for turbo KV — fall back to q8_0 otherwise

authored and
TheTom
committed
df33248
feat: Metal support for turbo2 (2-bit KV cache, 6.4x compression)

TheTom
and
claude
committed
e3b87f8
feat: asymmetric K/V quant support for Metal flash attention

TheTom
and
claude
committed
fe6749f
feat: asymmetric K/V support + q8_0 × turbo FA kernel instantiations

TheTom
and
claude
committed
59e2139
feat: zero-pad non-128 heads for full 7-stage WHT (replaces q8_0 fallback)

authored and
TheTom
committed
d158db5
perf: CUDA MMA flash attention for D=640 (GLM-4.7 turbo3: 37→192 t/s)

authored and
TheTom
committed
0836309
fix: add turbo3/turbo2 cross-type VEC FA instances (issue #25 bug 2)

authored and
TheTom
committed
1d5f5c7
feat: CUDA port of turbo4 (4-bit, 3.8x compression) — fixes issue #25 bug 1

authored and
TheTom
committed
821f843
fix: turbo4 on GLM-4.7 — context init check accounts for zero-padding (issue #28)

authored and
TheTom
committed
de03f39
feat: Boundary V (experimental) — layer-aware V compression

TheTom
and
claude
committed
a3e97d8
fix: KV state serialization uses padded tensor width (issue #28 follow-up)

authored and
TheTom
committed
ee82c55
feat: HIP/ROCm porting for TheTom's turbo3/turbo2 warp-cooperative kernels

authored and
TheTom
committed
a30a59e
Increase turbo3/turbo2 block size from 32 to 128

TheTom
and
claude
committed
b9df4c3
fix: CUDA warp-to-block mapping for block_size=128 (turbo3, turbo2)

Chris Grant
authored and
TheTom
committed
f028c66
Enable Sparse V on all Metal, auto-enable Boundary V for turbo2-V

TheTom
and
claude
committed
a6ff538
fix: add missing TurboQuant FA template instances for HIP/ROCm build

terrysimons
authored and
TheTom
committed
d9552b9
Remove unused CENTROIDS_1BIT constant

TheTom
and
claude
committed
d11b426
feat: TQ3_1S + TQ4_1S weight quantization with V2.1 fused Metal kernels

TheTom
and
claude
committed
3a2fad1
fix: add post-unrotate memory barrier for in-layer mixing safety

TheTom
and
claude
committed
7b5094b
fix: disable upstream attn rotation by default (conflicts with TurboQuant)

TheTom
and
claude
committed
c452be6
feat: CUDA port of TQ4_1S/TQ3_1S weight dequant (signalnine)
TheTom
committed
2ebd519
fix: TQ4_1S CUDA — mmvq exclusion for fused path + quantize tool registration

authored and
TheTom
committed
bf9bf31
perf: fused TQ4_1S/TQ3_1S mul_mat_vec — 3.4x decode speedup

authored and
TheTom
committed
8c2e0d8
fix: TQ4_1S on MoE models — disable CUDA graphs for TQ MUL_MAT_ID

authored and
TheTom
committed
ec6b8d4
perf: V12 single-phase fused TQ mmvq — shmem activation, no global scratch

TheTom
and
claude
committed
76ebd26
fix: Windows MSVC build compatibility for TQ weight types

TheTom
and
claude
committed
adda3bc
fix: AMD HIP/ROCm build support for TQ4_1S weight compression

TheTom
and
claude
committed
694ed03
fix: add dk512 Metal FA kernel instances for turbo types (Gemma 4 support)

TheTom
and
claude
committed
2962592
fix: CPU vec_dot heap allocation for turbo/TQ types (n > 4096 models)

TheTom
and
claude
committed
b1a6f79
Fix turbo4 C reference WHT dequant mismatch (#43)
TheTom
committed
fe2ead9
feat: load-time TQ4_1S -> q8_0 conversion for CUDA dp4a speed

committed
e3ce079
fix: remove redundant extern from GGML_API macro (GCC 13.3 hard error)

TheTom
and
claude
committed
71c7a4c
Enhance Metal operations for TQ weights and concurrency handling for Gemma 4

iamwavecut
authored and
TheTom
committed
c29fab6
Update GGMLQuantizationType and LlamaFileType enums to include TQ3_1S and TQ4_1S quantization types with corresponding sizes in GGML_QUANT_SIZES.

iamwavecut
authored and
TheTom
committed
5bad823
fix: GCC double extern in ops.cpp turbo3_cpu_wht_group_size

TheTom
and
claude
committed
e5ac94d
feat: add MoE expert count kernel instantiations + TQ4_1S backend tests

TheTom
and
claude
committed
753f199
fix: cap map0 kernel shmem for 256-expert MoE models

TheTom
and
claude
committed
6571604
perf: TQ4_1S native kernel 3.5× faster — 240 t/s (was 68), smaller VRAM than q8_0

signalnine
authored and
TheTom
committed
51481c3
perf: warp-cooperative TQ4_1S dequant (16× less compute per block)

signalnine
authored and
TheTom
committed
cc1bae2
feat: multi-token TQ4_1S dp4a kernel + multi-GPU fix + static build fix

signalnine
authored and
TheTom
committed
941d456
fix: replace __dp4a with ggml_cuda_dp4a for HIP/ROCm compatibility

signalnine
authored and
TheTom
committed
579db29
fix: AMD/RDNA4 arch dispatch — scalar half path for TQ4_1S on AMD GPUs

TheTom
and
claude
committed
0bf1eef
feat: Vulkan compute shader support for turbo3 KV cache

authored and
TheTom
committed
a494833
metal: add TurboFlash attention kernel for turbo3 KV cache decode

TheTom
and
claude
committed
e596f47
docs: add AMD Instinct MI300X (gfx942) ROCm test results

andyluo7
authored and
TheTom
committed
3b5e148
feat: add CDNA4 (gfx950/MI355X) support + test results

andyluo7
authored and
TheTom
committed
1df4783
vulkan: fix and complete turbo3 KV cache support

Titaniumtown
authored and
TheTom
committed
ff8bb73
vulkan: add turbo3 backend tests

Titaniumtown
authored and
TheTom
committed
88fcb67
Add GitHub Sponsors funding link
TheTom
committed
a4736ff
Fix memory explosion on Apple Silicon

huwprosser
authored and
TheTom
committed
e53f802
Fix GGML_OP_COUNT assertion for RPC

cpburnz
authored and
TheTom
committed
6775542
ci: fix turbo build and test failures

authored and
TheTom
committed
0009301

Commits on Apr 16, 2026

fix: add TURBO2_0 to flash_attn auto-enable check

TheTom
and
claude
committed
1073622

Commits on Apr 17, 2026

fix(cuda): allow f16/bf16 + q8_0 mixed KV without GGML_CUDA_FA_ALL_QUANTS (#82)

TheTom
and
claude
authored
59798f1
fix(cuda): add F16-K + TURBO-V dispatch cases in fattn.cu
dentity007
committed
fa4e8be
fix nix build: Add spirv-headers to vulkanBuildInputs

cguentherTUChemnitz
authored and
Christian Günther
committed
f9ee92b
fix(metal): add turbo2/3/4 types to FLASH_ATTN_EXT and CPY support checks
tomwolfe
committed
93bf21d
vulkan: fix turbo3 build + coopmat FA after April upstream sync
Tuklus
committed
0198d58

Commits on Apr 18, 2026

Commits on Apr 20, 2026

Commits on Apr 21, 2026

ci: add TurboQuant+ release workflow
TheTom
committed
4d24ad8
hip: disable mixed turbo vec FA by default on gfx900/gfx906
2bigO
committed
f15ee99
cmake: stop installing tests by default
2bigO
committed
bc2d9e5