Conversation
Full Metal backend Metal Shaders (voxtral_tts_shaders.metal, 446 lines): - rms_norm: threadgroup reduction with shared memory - silu_mul: fused SwiGLU activation - rope_apply / rope_apply_pos: rotary position embeddings - decoder_attention: online softmax with SIMD cooperative dot products - kv_cache_copy: single-token and batched KV cache writes - causal_softmax: for prefill attention scores - bf16_to_f32, time_embedding, argmax helper kernels Metal Implementation (voxtral_tts_metal.m, 872 lines): - BF16->F16 weight cache (MPS doesn't support bf16 natively) - F32 weight cache for norm weights - Activation buffer pool (reusable scratch buffers) - Zero-copy shared memory KV cache (MTLResourceStorageModeShared) - MPS MatrixMultiplication for GEMM (f32 activations x f16 weights) - Monolithic decoder forward: all 26 layers in ONE command buffer with MPS matmuls + custom compute kernels interleaved - Memory barriers between dependent operations Integration: - #ifdef USE_METAL dispatch in tts_llm_forward() - Metal shared alloc for KV cache - Metal init/shutdown in model load/free - Makefile: make apple-gpu (requires Xcode + Metal frameworks) - Shaders embedded via xxd -i at compile time Build: make apple-gpu Requires: macOS with Apple Silicon (M1+) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Added #ifdef USE_METAL test paths for rms_norm, silu_mul, add_inplace, gemm_bf16 - Added standalone Metal kernel wrappers (rms_norm, silu_mul, add_inplace) for test harness and individual dispatch - Fixed ARC __strong pointer annotation for pipeline state array - Added test-metal Makefile target Metal test results (Apple M4): - add_inplace: PASS (exact match) - silu_mul: PASS (max_rel=3.4e-7) - rms_norm: PASS (max_rel=2.4e-7) - gemm_bf16: PASS (max_rel=8.9e-2 — bf16->f16 lossy conversion) - Attention: PASS (CPU-self, Metal attention wrapper TODO) The high GEMM relative error (8.9% max) from bf16->f16 conversion may cause accumulated divergence in the monolithic 26-layer forward pass. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fixed duplicate function definition from bulk rename - Removed unused f32 weight cache (causes OOM on 16GB Macs) - Using bf16->f16 converted weights for MPS matmul (same as voxtral.c) - Added standalone kernel wrappers for test harness Known limitation: bf16->f16 conversion has ~8.9% max relative error which causes audio quality degradation on Metal. The TTS model is more precision-sensitive than ASR. Macs with >16GB may use f32 weight cache for better quality (toggle in code). Test results (M4, 16GB): - Kernels: 11/11 PASS - Model loads and generates audio - Audio quality: degraded due to f16 precision Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Metal Shaders:
Metal Implementation:
Build: make apple-gpu