Optimize Qwen3-ASR batch decode CPU sync

[hhh2210]

Context

This is a follow-up to the ASR/Qwen3-ASR infrastructure work tracked in ml-explore/mlx-swift#229. Thanks for reviewing and merging that groundwork; this PR keeps the next optimization small, correctness-first, and benchmark-driven.

Summary

• Add a correctness-safe Qwen3-ASR batch decode path that keeps per-row decoder forwards serial while syncing one batched [B, 1] token tensor per decode step.
• Add audio transcribe-batch --batch-size timing fields for Qwen3 batch runs.
• Add fixed-chunk benchmark tooling and a non-interactive Apple GPU power monitor wrapper.
• Document local benchmark results, batch-size cap behavior, and the gated true [B,1,H] experimental path.

Benchmark

Qwen3-ASR 0.6B 4-bit, repeated 10s chunks.

Local benchmark environment: MacBook Pro with an M4 chip, fans set to maximum speed.

Batch size	Files	Audio	Inference	Aggregate RTF	Speedup	Output digest
1	24	240s	3.95s	0.0165	1.000x	56444fd6d4b9
6	24	240s	3.55s	0.0148	1.113x	56444fd6d4b9
8	24	240s	3.68s	0.0153	1.073x	56444fd6d4b9

Batch size 8 was slower than 6 in the local sweeps, so the recommended cap should be benchmark-driven.

Known limitation

The true [B,1,H] MLX decoder forward is still behind QWEN3_ASR_EXPERIMENTAL_BATCH_DECODE=1. It currently fails repeated-chunk row correctness at batch size 2, where row 0 completes and row 1 truncates, so this PR keeps the production path conservative.

Validation

• make build
• swift test --disable-sandbox --filter 'Qwen3ASRBatchedDecodeTests|TranscribeBatchCommandTests'
• swift test --disable-sandbox --filter E2EQwen3ASRBatchedDecodeCorrectnessTests
• make test
• python3 -m py_compile scripts/benchmark_qwen3_batch_decode.py scripts/monitor_apple_gpu.py
• git diff --check

Refs ml-explore/mlx-swift#229
Refs ml-explore/mlx-swift#233

[hhh2210]

@codex review

[Copilot]

Pull request overview

This PR introduces a correctness-safe Qwen3-ASR batch transcription API and CLI support that reduces CPU/GPU synchronization overhead during greedy decoding, plus supporting benchmarks and documentation for evaluating batch-size caps and experimental true-batched decoding.

Changes:

• Add Qwen3ASRModel.transcribeBatch(...) with a production-safe “bulk token sync” greedy path and an env-gated experimental true [B,1,H] decode path.
• Extend audio transcribe-batch with --batch-size grouping and additional per-item/per-batch timing fields (JSONL and human output).
• Add benchmark/power-monitor scripts and documentation for fixed-chunk batch decode sweeps and experimental validation.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
Tests/Qwen3ASRTests/Qwen3ASRBatchedDecodeTests.swift	Unit coverage for option resolution, seq-len bucketing, prompt token boundaries, and batched causal mask.
Tests/Qwen3ASRTests/E2EQwen3ASRBatchedDecodeCorrectnessTests.swift	E2E regression ensuring public batch transcription matches serial greedy output for repeated chunks.
Tests/AudioCLITests/AudioCLITests.swift	CLI parsing tests for transcribe-batch --batch-size and related flags.
Sources/Qwen3ASR/Qwen3ASR.swift	Implements transcribeBatch, seq-len bucketing helpers, bulk-sync greedy batch decode, and env-gated experimental true-batched decode.
Sources/AudioCLILib/TranscribeBatchCommand.swift	Adds --batch-size grouping for Qwen3 runs and emits new per-batch timing fields.
scripts/monitor_apple_gpu.py	Non-interactive powermetrics wrapper for GPU/CPU power sampling during benchmarks.
scripts/benchmark_qwen3_batch_decode.py	Fixed-chunk batch-size sweep harness around audio transcribe-batch --jsonl.
docs/inference/qwen3-asr-inference.md	Documents the batched greedy decode behavior and experimental gating.
docs/benchmarks/qwen3-batch-decode.md	Benchmark methodology, usage, and known limitations for batched decode sweeps.

[ivan-digital]

Thanks for raising this. Traced the experimental [B,1,H] row-1 truncation to MLXNN.RoPE: it produces row-asymmetric output at [B>1, H, T=1, D] even with identical row inputs. Issue: ml-explore/mlx#3496.

Default BulkSync path (+11%) is unaffected. mlx-lm's BatchKVCache pattern doesn't fix it either — the bug is upstream of the cache.

Cleanup follow-up with regression test: ml-explore/mlx-swift#236.

[hhh2210]

Thanks for tracing this down to MLXNN.RoPE and for opening the upstream issue. That explanation makes sense to me, especially since the default BulkSync path is unaffected and BatchKVCache does not address the upstream RoPE behavior.

I agree with keeping the true [B,1,H] path gated until ml-explore/mlx#3496 is fixed, and ml-explore/mlx-swift#236 looks like the right cleanup/regression-test follow-up on top of this PR.