Add Parakeet-TDT 0.6B v3 (FastConformer-TDT ASR) - #111
Merged
Conversation
Initial port of nvidia/parakeet-tdt-0.6b-v3 — a 600M-param
FastConformer-TDT
ASR model supporting 25 European languages with auto-detection.
Files under community_models/parakeet_tdt/:
- assets: config parsing (Transformers-format), tokenizer loading
- weights: NeMo safetensors key mapping, BN fold for 24-layer
depthwise conv
- frontend: Mel filterbank frontend (16kHz, 128 bins, preemphasis=0.97)
- encoder: 24-layer FastConformer, NeMo-style Conv2D subsampling,
relative position attention with untied biases, xscaling
- decoder: 2-layer LSTM predictor + joint (joint_enc(1024→640),
decoder_projector(640→640), joint_head(640→8198))
- session: offline session with prepare/run
- loader: IVoiceModelLoader, family="parakeet_tdt"
- headers: include/engine/community_models/parakeet_tdt/
Wiring: model_specs/parakeet_tdt.json, registry.cpp registration,
CMakeLists.txt engine_runtime sources + parakeet_warm_bench target.
The model loads and runs end-to-end on both CPU and CUDA (GTX 1650:
~180ms vs ~1570ms CPU). Encoder graph builds and produces 93-frame
1024-dim output. Decoder produces all-blank output due to a suspected
safetensors reader bug returning wrong data for tensors at offsets
beyond ~40 MB — BN statistics and depthwise conv weights in the
48 MB region read different values than direct HTTP range fetches,
while embedding weights at 1.6 MB read correctly.
…tend The C++ frontend computed log-mel features (preemphasis -> STFT -> mel filterbank -> log) but never normalized them, emitting raw log-mel values (mean ~-11.8, std ~4.1). NeMo's AudioToMelSpectrogramPreprocessor for this model is configured with normalize: "per_feature", which subtracts the per-mel-bin mean and divides by the per-mel-bin unbiased std (computed over the valid, unpadded time frames only, with a 1e-5 epsilon added to the std) before the features ever reach the encoder. Skipping this step fed the subsampling conv stack input at roughly the wrong scale by orders of magnitude, and that error compounded through the conv/linear layers, producing runaway-magnitude activations by the time they reached the transformer encoder layers. Confirmed against NeMo's own normalize_batch() (features.py): with the fix, our mel features match NeMo's bit-for-bit in direction (cosine similarity 1.0) and to 7 decimal places in scale, and the subsampling stage's linear output now matches NeMo's to 4 significant figures (previously off by ~500x). This does not fully fix decoding on its own — a separate bug remains in the transformer encoder layers — but it eliminates a major, independently-verified source of divergence from the reference model. Bing Bong
The FastConformer encoder layer's depthwise conv was padded causally (all padding on the left, via pad_causal_1d(x, kernel-1)), which is wrong for this model's full-context (non-streaming) configuration. NeMo's ConformerConvolution.depthwise_conv is a CausalConv1D, but for the full-context encoder it is constructed with an explicit integer padding=conv_context_size=(kernel_size-1)//2, and CausalConv1D treats an int padding argument as SYMMETRIC left/right padding, not causal-only left padding (only a `None` or explicit [left, right] argument produces asymmetric/causal padding, and neither is used here). Padding causally instead shifted every frame's receptive field, corrupting the residual stream through all 24 encoder layers. Isolated a single encoder layer's conv module against hook-captured NeMo reference activations (same weights, same input) to localize this: with causal padding the module's output was nearly uncorrelated with NeMo's (cosine similarity 0.22); switching to symmetric padding raised that to 0.92, uniformly across all 93 frames (not concentrated at the sequence edges, which rules out an off-by-one/boundary-only explanation and points at the padding mode itself). Combined with the prior frontend normalization fix, the model now transcribes the reference test clip correctly end to end: "Well, I don't wish to see it any more, observed Phoebe, turning away her eyes. It is certainly very like the old portrait." — matching the NeMo reference transcription exactly. Bing Bong
Two follow-ups from the recent Parakeet TDT correctness review
(tmp/TODO.md items 1-2):
- decoder.cpp: remove an unconditional fprintf(stderr, "DECODE_DIAG ...")
that fired on every single decode() call (frame 0 of every request),
left over from the earlier blank-output debugging session. It also
shadowed the outer `cfg` local with a redundant re-declaration.
- assets.cpp: stop silently swallowing all processor_config.json parse
errors behind a bare `catch (...)` that only restored the preemphasis
default, leaving sample_rate/feature_size/n_fft/win_length/hop_length
silently on their struct defaults if anything about that file failed to
parse. Now: skip parsing (and use defaults) only when the file is
genuinely absent (resources.has_file(...) == false); if it's present but
fails to parse or doesn't match the expected schema, that exception now
propagates instead of being swallowed. Verified against three cases:
file present and well-formed (unchanged behavior), file missing entirely
(still falls back to defaults, though in practice resource-bundle file
matching already makes this case unreachable today), and file present
but malformed (previously silent, now throws
"missing required json key: feature_extractor" as expected).
Rebuilt and reran parakeet_warm_bench against the reference clip: stderr
is now clean of DECODE_DIAG output and the transcription is unchanged
("Well, I don't wish to see it any more, observed Phoebe, turning away
her eyes. It is certainly very like the old portrait.").
Bing Bong
Follow-up from the recent Parakeet TDT correctness review (tmp/TODO.md item 3, "Tier 1"). Neither of the two real bugs fixed in 4a10c48/3bf2d12 (missing frontend normalization, wrong conv padding mode) crashed or produced an obviously malformed result — the model loaded, ran, and produced plausible-shaped tensors throughout — so nothing in the existing test suite would have caught either regression. This adds the cheapest thing that actually would have: run the full offline pipeline against a fixed clip and assert the decoded text matches the known-correct NeMo reference transcription exactly. - tests/parakeet_tdt/assets/2086-149220-0033.wav: a ~7.44s LibriSpeech test-clean clip (CC BY 4.0), with provenance documented in the accompanying assets/README.md. - tests/parakeet_tdt/test_golden_transcription.cpp: loads the model, runs it against that clip, and asserts the output text is exactly "Well, I don't wish to see it any more, observed Phoebe, turning away her eyes. It is certainly very like the old portrait." — verified against the actual NeMo reference model output for this clip, not just eyeballed. Requires the real (multi-GB) model weights, which aren't present in a normal checkout/CI build; the test detects this at startup and exits 125 (SKIP) rather than failing when the model or fixture audio is missing. - CMakeLists.txt: wires the test into the existing ENGINE_BUILD_TESTS/ ctest suite via the add_engine_unittest() helper, with SKIP_RETURN_CODE 125 configured so a missing-model environment shows as skipped rather than failed. This is not a substitute for numerical layer-by-layer parity comparison against NeMo (the "Tier 2" harness described in tmp/todo-03-add-parity-regression-test.md, not yet built) — it only catches regressions large enough to flip the final decoded text for this one clip. It's deliberately cheap enough to run on every change, unlike a NeMo-based numerical comparison which needs a NeMo install on top of the model weights this test already needs. Verified: `ctest -R parakeet_golden_transcription_test` passes (~7.4s on CPU) against the current model checkout, and the skip path was confirmed independently by running the binary against a nonexistent model path (exits 125 as expected). Bing Bong
The depthwise conv in build_fastconformer_conv_module was built with use_bias=false, silently dropping weights.conv_dw_bias — the batch-norm bias term fold_bn() computes and bakes into the depthwise conv (-running_mean*scale + bn_bias). The scale/weight side of the BN fold was applied correctly; only the additive bias term was being discarded. This did not produce obviously wrong output — greedy/argmax decoding turned out to be robust enough to this that the golden-clip transcription was already correct without it — which is exactly why it went unnoticed until a numerical (not just text-output) comparison against the real NeMo model was built. Isolating the depthwise conv in a minimal single-op graph and comparing against a hand-computed reference showed every output value differed from the correct one by exactly the same constant offset, equal in magnitude to the folded bias for that channel — a very clean signature once found, but invisible from the top-level transcription alone. Also exports build_encoder_layer (previously anonymous-namespace) via encoder.h so a layer can be built and numerically verified in isolation without duplicating the real production graph-building code — this is what made isolating the bug to this one line tractable instead of only being visible as an accumulated ~0.78 cosine similarity drift by the final of 24 layers. Verified: chaining 24 isolated single-layer builds — each fed the previous one's real output, exactly like the production encoder does — now stays at cosine similarity >= 0.999999 against the real NeMo model's per-layer output through every one of the 24 layers (previously this drifted down substantially by the later layers). Golden-clip transcription remains correct on both CPU and CUDA. Bing Bong
Adds the "Tier 2" numerical layer-by-layer comparison harness alongside the existing golden-transcription test: a NeMo reference dump script, a C++ dump tool built off the real production entry points (frontend extraction, full encoder, and one isolated encoder layer via the newly exported build_encoder_layer), and a numpy-only comparator that checks cosine similarity and relative scale per stage. This tool is what found and confirmed the encoder conv-module bias bug fixed in 837382e — the golden-transcription test alone did not catch it, since greedy decoding was robust enough to that particular corruption not to change the final transcribed text on the one test clip. Comparing actual activations, not just final decoded text, is a meaningfully different and complementary check. - tests/parakeet_tdt/parity/dump_nemo_reference.py: runs the real NeMo model with forward hooks, dumps mel features, pos_emb, every encoder layer's output, and the final encoder output as .npy files. - tests/parakeet_tdt/parity/npy_io.h: minimal float32 .npy reader/writer (just enough for this tool; not a general-purpose npy library). - tests/parakeet_tdt/parity/dump_cpp_reference.cpp: mirrors the NeMo dump using the real, unmodified C++ production entry points for mel features and the full encoder, plus one isolated single-layer build for granular comparison. Deliberately does not splice extra debug-output taps into the shared production graph — see the file's top comment for why (an extra "dead end" output tensor on that specific cached multi-thousand- node graph read back incorrect values during the original encoder debugging session, for reasons never fully root-caused; small isolated graphs and the unmodified production entry points both proved reliable and are what this tool sticks to). - tests/parakeet_tdt/parity/compare_parity.py: numpy-only comparator, runnable without NeMo/torch installed. - CMakeLists.txt: wires parakeet_parity_dump as a buildable tool under ENGINE_BUILD_TESTS (not registered as an add_test — it's a manual/ pre-release check requiring a NeMo install on the Python side, not a CI gate). Bing Bong
The parakeet_tdt loader was registered and working but had no installable package entry (tools/check_loader_catalog_sync.py warned about this) and wasn't listed anywhere in README.md or docs/community_models/models.md. - tools/model_manager.py: add a parakeet_tdt ModelPackage entry. The repo (nvidia/parakeet-tdt-0.6b-v3) hosts both a Transformers-compatible safetensors checkpoint and a NeMo .nemo archive side by side; the package uses include_prefixes to fetch only the four files this loader actually reads (config.json, processor_config.json, tokenizer.json, model.safetensors), not the redundant multi-GB .nemo file. Verified the entry resolves against the real repo: `python3 tools/model_manager.py install parakeet_tdt` began fetching the correct files (config.json matched the existing local file's exact byte size; model.safetensors started downloading from the right repo) before being stopped partway through given the file size. - docs/model_manager.md: add the package row to the recommended package table (required by the sync checker). - docs/community_models/parakeet_tdt.md: new per-model doc following the existing community-model doc convention — build/run commands, what was validated and how (golden-transcription test + numerical parity harness), the two bugs found and fixed while getting there, and current known limitations (offline only, no streaming yet). - README.md, docs/community_models/models.md: add the community-models table row. python3 tools/check_loader_catalog_sync.py now passes with no warnings. Bing Bong
Investigated adding NeMo cache-aware streaming support (causal conv with a cross-chunk cache_last_time cache, chunked-limited attention with a cache_last_channel KV cache) before writing any code, and found the actual checkpoint doesn't support it in a meaningful way: - nvidia/parakeet-tdt-0.6b-v3's encoder config has att_context_style="regular" and att_context_size=[-1, -1] — unlimited, fully bidirectional attention, not NeMo's "chunked_limited" streaming style. - Calling NeMo's own encoder.setup_streaming_params() on this checkpoint doesn't error, but produces a degenerate configuration (~5.8 second chunks, a ~10000-frame attention cache) that wouldn't provide the latency or bounded-memory benefit real streaming is for. - NVIDIA's own maintainers, asked directly about streaming with this model on its Hugging Face discussion page, pointed users at a different, dedicated cache-aware streaming architecture rather than confirming this checkpoint for the purpose. Building a genuine chunked/causal streaming path against a model trained with an unrestricted attention receptive field risks silently degrading accuracy for a mode that wouldn't provide much practical benefit even working correctly. This needs a different, purpose-trained checkpoint (att_context_style="chunked_limited"), not more implementation effort against this one. This framework already has real streaming ASR: the nemotron_asr family targets nvidia/nemotron-3.5-asr-streaming-0.6b, a same-size-class NVIDIA checkpoint actually trained cache-aware with configurable chunk sizes down to 80ms, and already implements IStreamingVoiceTaskSession. Pointed to it from parakeet_tdt's known-limitations doc for anyone who lands here specifically wanting streaming. Bing Bong
pad_causal_1d, pad_causal_2d, and causal_conv_output_dim were leftover, never-called scaffolding from an earlier attempt at streaming support (compiler already flagged all three as unused). Confirmed streaming isn't a viable follow-up for this specific checkpoint (see the preceding commit) before removing them, so this isn't losing groundwork for later — it's dead code for a direction that doesn't apply to this model. The one function that was actually load-bearing, pad_symmetric_1d (the fix for the encoder conv module's padding mode), is unaffected. Verified: rebuilds warning-free, golden-transcription test and the numerical parity harness both still pass unchanged. Bing Bong
parakeet_warm_bench.cpp set ENGINE_TIMING_ENABLED/ENGINE_TIMING_FILE as
environment variables, but the logging subsystem doesn't read those env
vars anywhere in the codebase — every other *_warm_bench.cpp tool
(nemotron_asr, vibevoice_asr, qwen3_tts, etc.) wires up timing by calling
engine::debug::configure_logging(LoggingConfig{true, timing_path}) directly.
parakeet_warm_bench never did this, so its --timing-file output has been
silently empty since the tool was written — the frontend/encoder/decoder
breakdown was never actually available, only the top-level wall-clock
number.
Also fixed two stale/wrong keys in ordered_keys(): "parakeet.encoder_ms"
never matched what encoder.cpp actually logs
(debug::timing_log_scalar("parakeet_tdt.encoder_ms", ...) — note the
_tdt), and "parakeet.pre_encode_ms" is never emitted anywhere at all
(dead reference, presumably left over from a metric that was renamed or
removed). Replaced with the correct "parakeet_tdt.encoder_ms" and added
"parakeet_tdt.encoder.graph.compute_ms", which is what actually turned
out to matter for performance work (see the following commits): encoder
graph compute is ~93-96% of total wall time in every configuration
measured.
Bing Bong
…ights Compared audio.cpp against the real NeMo/PyTorch reference on the same machine (see docs/community_models/parakeet_tdt.md's new Performance section for the full numbers and methodology). With the timing breakdown now actually working (previous commit), encoder graph compute was clearly ~93-96% of total wall time in every configuration, so that's where optimization effort actually matters — not the frontend or decoder. Two changes, both using infrastructure that already existed (no new inference code): - Thread count: this machine is 6 physical / 12 logical cores; 8 threads (an arbitrary default) oversubscribes, 6 is consistently fastest, 12 is consistently worst. Core-count-dependent, not a universal number — sweep it on your own hardware. - parakeet_tdt.matmul_weight_type=q8_0 (an existing but previously untested session option): ggml's CPU backend has heavily hand-tuned Q8_0 dot-product kernels; measured faster than F16/BF16 here, not just faster than F32. Net: CPU 1247.5ms -> 830.9ms (1.60x), CUDA 167.4ms -> 132.1ms (1.27x), transcription unchanged in every configuration. CPU with q8_0 is now ~3% faster than PyTorch's own CPU number on this clip. Quantified the accuracy cost via the numerical parity harness rather than just trusting "the transcription still matched": enc_out cosine similarity vs the NeMo reference drops from 0.972258 (native F32) to 0.968028 (q8_0) — comparable in size to the float32 accumulation noise already present between an isolated single-layer graph and the full 24-layer production graph, not a cliff, but a real, measured, non-zero cost. Only validated against one clip, so this is documented as a strongly-recommended opt-in session option, not changed as the default — that would need broader validation first. - dump_cpp_reference.cpp: add --matmul-weight-type so the parity harness can numerically compare any weight storage type against the NeMo reference, which is what produced the 0.972258 -> 0.968028 numbers above. - docs/community_models/parakeet_tdt.md: new Performance section. - tests/parakeet_tdt/parity/README.md: document the new flag. Bing Bong
…re it as a net loss Wires the FastConformer encoder's relative-position self-attention onto ggml_flash_attn_ext_with_bias_mask (already used by common_relative_attention.cpp's specialized_flash path, but unused by any production model) via a new parakeet_tdt.encoder_flash_attention session option, off by default. Validated correct via the numerical parity harness (enc_out cosine 0.972325 vs. 0.972258 for the existing path, layer_0 unchanged) but measured consistently slower end to end, not faster, on both CPU and CUDA, on both the 7.4s reference clip and a synthetic 59.5s clip. Documented as an opt-in for other hardware rather than removed, since it's now proven correct infrastructure, but explicitly not recommended based on what was actually measured here. Bing Bong
…uant bucket pointwise_conv1/pointwise_conv2 in the FastConformer conv module are kernel_size=1 Conv1d layers — mathematically Linear layers, and run through LinearModule/mul_mat in this port's encoder, not a real conv op. They were being loaded under conv_weight_type (capped at f16, no q8_0) instead of matmul_weight_type, silently excluding ~3*hidden^2 params/layer across all 24 layers from quantization when matmul_weight_type=q8_0 was requested. Found by cross-referencing CrispStrobe/CrispASR's FastConformer/Parakeet port, which documents hitting and fixing the identical quantizer misclassification for the same class of weights. Verified via the numerical parity harness (no change at the native default; q8_0 accuracy unchanged within noise) and the golden-transcription test (exact match on both CPU and CUDA). ~10% additional CPU wall-time reduction on top of the existing q8_0 + thread-tuning result; no clear CUDA change. Bing Bong
… layer load_encoder_layer now reads the raw F32 q_proj/k_proj/v_proj rows and concatenates them at load time into a single [3*hidden, hidden] weight (row-major Linear layout concatenates directly along the output-feature axis, no interleaving needed), populating the previously-dead qkv_weight field on the shared AttentionWeights struct. build_encoder_layer now does one Linear(hidden, 3*hidden) + slice instead of three separate Linear(hidden, hidden) calls. Inspired by CrispStrobe/CrispASR crediting the same fusion as part of their FastConformer encoder win. Validated via the parity harness to be numerically identical to the pre-fusion path at both native (cosine 0.972258) and q8_0 (cosine 0.967974) precision — a pure reorganization, not a precision trade — and the golden-transcription test still matches exactly on CPU and CUDA. No clear wall-time change measured on this hardware (dispatch overhead isn't the bottleneck at this graph size here), but it's zero-cost to carry and may pay off on other backends or under batched/concurrent serving conditions not exercised by this benchmark. Bing Bong
…graph engine::runtime::optimize_graph (src/framework/runtime/graph_optimizer.cpp) is a real, unit-tested graph rewrite pass — folds broadcast repeats into consuming ops, elides pure-metadata reshape/view/cont chains, elides no-ops — but had zero callers anywhere in the codebase: not graph_executor.cpp, not any model family. Wired into ParakeetEncoderRuntime::ensure_graph, called once per distinct input length right after ggml_build_forward_expand, before allocation, so the cost is amortized across every subsequent encode() call that reuses the cached graph. Effect on the reference clip's encoder graph: 2854 nodes -> 1425 (1323 metadata-only ops elided, 106 broadcast repeats folded). Validated via the parity harness (enc_out cosine 0.972258, bit-identical to before) and the golden-transcription test (exact match, both backends). Speed, measured via an interleaved A/B using the optimizer's existing ENGINE_GRAPH_OPTIMIZER=0/1 env toggle to cancel out this machine's background-load drift: CPU q8_0 encoder time ~812ms -> ~726ms (~10%, ahead in 5/6 paired runs). CUDA: no measurable difference, expected, since GPU options intentionally skip metadata-only elision (the scheduler-based backend still needs those nodes). Bing Bong
Traced parakeet_warm_bench with nsys profile --trace=cuda: 16068 cudaLaunchKernel calls, 4265 cudaStreamSynchronize calls (47% of total CUDA API time), and zero cudaGraphLaunch/cudaGraphInstantiate calls anywhere, despite GGML_CUDA_GRAPHS=ON in this build and the encoder/decoder graphs already satisfying the capture prerequisite (same cached ggml_cgraph* reused across every call at a given shape). Root cause traced into ggml_cuda_graph_set_enabled (external/ggml/src/ggml-cuda/ggml-cuda.cu): ggml unconditionally disables CUDA graph capture below compute capability 8.0 (Ampere). This machine's GTX 1650 is Turing, compute capability 7.5. Not a bug or misconfiguration in this codebase - the build already does everything right - but it retroactively explains why flash attention and fused QKV (both already documented above) measured no CUDA benefit here: they target exactly the per-op dispatch overhead that graph replay exists to hide, which never activates on this card. Documented as a reason to re-measure both on Ampere-or-newer hardware rather than trust this machine's null result. Bing Bong
Three copy-elimination changes in the encoder layer, all bit-exact: - Q/K/V are read out of the fused QKV result as strided 4d views already in per-head order, rather than slicing the feature axis and forcing each slice contiguous. Those conts were not optional once you slice, because reshape_heads goes through ggml_reshape which asserts contiguity — so the old path copied all three projections in full every layer just to satisfy a reshape, and MatMulModule then re-materialized k and v anyway. - The Transformer-XL relative shift is expressed directly as the strided view it mathematically is, replacing the generic concat + cont + slice + cont sequence (~1.1 MB of copying per layer, most of it discarded by the following slice). It also folds the trailing slice in for free. See the derivation above relative_shift_view. - The convolution module's BTC->BCT->BTC transpose pair around pointwise_conv1 cancelled out exactly and is gone; only the depthwise conv actually needs BCT. Encoder graph drops 1425 -> 1257 nodes and cont traffic 123 -> 84 MB. Encoder output is bit-identical across all 95232 enc_out floats. Measured on its own this is only ~0.3% of encode time on a 6-core i7-9750H (drift-cancelling interleaved A/B, median of 6 passes): the encoder is overwhelmingly GEMM-bound, with ~74% of cycles inside tinyBLAS. Kept because it is free, removes work that scales with sequence length, and the node-count reduction helps backends where dispatch is not free. Bing Bong
Three constant scales ran as their own ggml_scale pass over the activations: the 0.5 residual half-step at the end of each feed-forward branch (twice per layer, 48 passes over [seq, 1024]) and RelPositionalEncoding's xscaling by sqrt(d_model) after the subsampling projection. All three are folded into the producing Linear's weights at load time instead, removing all 49 SCALE nodes from the encoder graph (1257 -> 1208). The fold is exact rather than approximate, because both constants are powers of two. Scaling a float by a power of two only adjusts the exponent, and every term of the dot product is scaled identically, so sum(0.5*w_i*x_i) equals 0.5*sum(w_i*x_i) bit for bit. It also survives quantisation: q8_0 derives its block scale from max|w|, so halving every weight in a block halves the block scale and leaves the stored integers untouched. For the subsampling projection both weight and bias are scaled, since the xscaling applied after the linear, not before it. Verified bit-identical across all 95232 enc_out floats. Bing Bong
ensure_graph() reuses any cached graph whose capacity merely exceeds the request, and encode() zero-pads the input up to that capacity. Zero input does not stay zero: every subsampling conv carries a bias, so the padded tail reaches the encoder stack as nonzero garbage. The attention mask was unconditionally all-zeros, so real frames attended to that garbage, and because softmax normalizes across keys it rescaled every real frame's attention. Not theoretical: feeding a 7.4s clip to a graph prepared with a 60s audio contract dropped an entire sentence from the transcription. matched "...turning away her eyes. It is certainly very like the old portrait." oversized "...turning away her eyes." and this commit restores the full, correct text in the oversized case. Masking is by key column only, never by query row, so the padded query rows keep a full set of valid keys and no softmax row degenerates to all -inf and produces NaN. Their outputs are discarded afterwards regardless. Matched-size graphs are unaffected: no mask entries are written when valid == capacity, and enc_out stays bit-identical across all 95232 floats. Bing Bong
…ersized The encoder graph always runs at the capacity it was built for, because encode() zero-pads the input up to that capacity. ensure_graph() reused any cached graph whose capacity merely exceeded the request, so a session prepared with a long audio contract paid the long price on every short clip that followed. Measured on a 6-core i7-9750H, 7.4s clip: matched-capacity graph 1018 ms 60s-capacity graph 10928 ms (10.7x) 60s contract, with this change 1140 ms Rebuilding is cheap by comparison — ~400 ms, and dominated by recomputing the 24 per-layer positional projections; the allocation itself is ~0.4 ms. So rebuilding wins outright on the very first call whenever the mismatch is more than a few percent, and wins by more on every call after that. A 10% oversize tolerance keeps a stream of clips whose lengths wobble slightly from rebuilding every call, and caps the wasted compute at roughly the same fraction. enc_out stays bit-identical when the capacity already matched, and the golden transcription test passes. Bing Bong
…ology Documents where the encoder's CPU time actually goes (~74% in llamafile's tinyBLAS f32 GEMM), the measured GEMM ceiling per weight type at 1 and 6 threads, and the graph-capacity bug — both its 10.7x cost and the dropped sentence it caused. Also records two optimizations that looked obvious and measured as losses, so they don't get re-attempted: - Padding the token axis to clear tinyBLAS's fast-path guards really does speed the attention GEMMs up ~2x, but slows every weight GEMM ~6%, and the weight GEMMs dominate: ~233 ms spent to save ~91 ms. - The single-core GEMM is not memory-bound at all — throughput is flat from 0.25 MB to 32 MB of weights — so it is kernel-bound at ~22% of AVX2 peak inside vendored code, and shrinking weights only helps when it selects a genuinely faster kernel (q8_0 does, f16 does not). Finally, documents the benchmarking method, because this machine drifts 20-30% between cold and thermally saturated. The ABBA-with-mean estimator cancels linear drift; the min() estimator an earlier harness used does not, and produced a confident 5-8% "regression" that reversed once corrected. Bing Bong
… numbers Replaces the earlier hand-wave with per-op measurements at 1 and 6 threads, and separates the two padding variants: padding the token axis (clear loss, ~233 ms spent to save ~91 ms) versus padding only the key axis (net ~+0.5% at 6 threads, ~+1% at 1). Records why neither is worth doing, including that changing AV's k dimension gives up the bit-exactness the other changes keep. Bing Bong
The two existing tiers both run on a single English clip, which is not enough to decide whether a weight-storage type is safe. Greedy decoding absorbs a lot of numerical drift, and cosine similarity on enc_out is not a transcription: q8_0 scores 0.9964 there and is byte-identical on the golden clip, which reads as "free". It is not. Across 35 FLEURS clips in 7 languages, q8_0 matched the f32 transcription exactly on only 91.4% of clips and cost +0.4 points of absolute WER, concentrated in particular languages (Estonian 16.4% -> 18.4%) rather than spread evenly. One clip in one language would have reported that as a clean pass. This harness pulls FLEURS test clips for the 24 European languages the model supports that FLEURS covers, and reports per weight type: exact-match rate against f32, WER against f32, absolute WER against the human reference, and a per-language breakdown so concentrated damage is visible. fetch_fleurs.py deletes each ~150-400 MB parquet right after extracting the few clips it keeps, and records per-language progress, so it is resumable and does not accumulate ~10 GB. compare_weight_types.py runs all clips for a type in one process via --audio-sequence so the quantized weights are built once. Bing Bong
Replaces "only validated against one clip" with an actual measurement: 120 FLEURS clips, 5 per language, across the 24 European languages the model supports that FLEURS covers. The result reframes q8_0. It changes 8.3% of transcripts, which the previous single-clip evidence (byte-identical) completely missed — but absolute WER against the human reference is flat, 0.1331 vs f32's 0.1338. The churn is mostly cosmetic (T Rex -> T-Rex), and where real words change q8_0 is sometimes right where f32 was wrong. So the 1.79x is not paid for in quality on this corpus; it is paid for in reproducibility against an f32 baseline. Also records bf16 as the low-thought option (1.14x, 99.2% identical), f16 as pointless (no speedup, slower than f32 single-core), and corrects the thread count guidance: 12 threads is 3.1% faster than 6, not "consistently worst". That earlier claim was an artifact of sequential before/after timing on a drifting machine and does not survive a controlled A/B; the GEMM microbenchmark agrees with the corrected result. Bing Bong
The weight-precision bullet still asserted that both reduced-precision types measured slower than F32, which the 24-language study contradicts for BF16: it is 1.14x faster. F16 remains a non-speedup (and is genuinely slower than F32 single-core), so the underlying point — that the win tracks which ggml kernels are well optimized rather than bit width — survives, but the specific claim did not. Bing Bong
… GPU caveat Records the actual output rather than only the summary: every clip's reference plus what native/f16/bf16/q8_0 each transcribed, 120 clips across 24 languages. That means the aggregate numbers can be re-derived, or re-scored under a different WER normalization, without re-running the study or re-downloading FLEURS. Includes the per-language table and samples of the 10 clips where q8_0 diverges. Also states the test hardware precisely, and how far each half generalizes. The CPU is an ordinary 6-core i7-9750H, so the CPU findings should carry to comparable AVX2 machines. The GPU does not generalize: it is a GTX 1650 with Max-Q Design — Turing, compute capability 7.5, 4 GB, and a 35 W power limit. That is about the weakest CUDA device this model realistically runs on, and its compute capability is below the 8.0 threshold where ggml enables CUDA graph capture at all. The several optimizations that measured as "no change" on CUDA are statements about this card, not about the optimizations, and the docs now say so wherever those results appear. All of native/f32/f16/bf16/q8_0 verified transcribing correctly end to end; they were already supported, no code change needed. Bing Bong
The model doc had grown to 489 lines, 4.2x the next largest doc in docs/community_models/ (glm_tts 116, outetts 104, vietneu_tts 66), and someone who just wanted to run the model had to scroll past benchmark methodology, rejected experiments, and CUDA-graph hardware analysis to find it. The repo already has the right shape for this: long writeups live in docs/reports/ (glm_tts_validation.md, outetts_validation.md, and the existing *_performance.md reports), with the community_models entry kept short. This follows that split — 162 lines of user-facing doc, 405 lines of report — and moves no content out of the tree, only between files. Also removes references that only resolve on the machine this was developed on, or in projects that are not this one: - a comparison to a locally cloned copy of another Parakeet port - the benchmark harness, which the methodology section cited by a path under the untracked tmp/ directory. Rather than drop the reference, the harness is now checked in under tests/parakeet_tdt/bench/ with a README explaining why the ABBA-with-mean estimator is required here and what breaks without it. - a stale pointer in the golden-transcription test to an untracked planning note, repointed at the parity harness it actually meant Third-party attribution is kept where it is genuine provenance: the conv-pointwise weight misclassification and the fused-QKV idea were both found by cross-referencing a public project, and citing that is honest. What is removed is the passage that leaned on that project's own unmeasured benchmark numbers to justify a decision here; that argument now stands on this repo's own reasoning. Bing Bong
The harness hardcoded BackendType::Cpu, so it could only ever answer "does this match NeMo on CPU". It could not answer the question that actually matters when a change touches tensor layout: does CPU still agree with CUDA. That gap was live. The recent switch to reading attention operands as strided ggml_view_4d views is verified bit-exact on CPU, where MatMulModule materializes the operands — but CUDA runs different kernels over the same graph, and nothing checked it. Transcription matching on both backends is suggestive, not proof. With --backend cuda, it is now checked directly: enc_out agrees between CPU and CUDA to cosine 0.99999988 (relative RMS 3.5e-06, max absolute 2.6e-06), which is float32 accumulation-order noise between different kernel implementations, and mel features are bit-identical. Bing Bong
Records how to use --backend to diff CPU against CUDA directly, what to expect (float32 accumulation noise, not bit-exactness, because different kernels sum in different orders), and the current measured agreement. Bing Bong
Adds the backend/memory section the report was missing. Both backends re-validated on the merged tree; CPU and CUDA agree on enc_out to cosine 0.99999988, which is what confirms the view-based attention rewrite is correct under CUDA's kernels and not only under the CPU path where its bit-exactness was measured. Memory numbers were absent entirely and CONTRIBUTING asks for them. Two are worth knowing: q8_0 cuts peak host RSS 3.6x (3843 -> 1078 MiB), a larger practical difference than its 1.79x speed win on constrained machines; and peak VRAM is 2642 of 4096 MiB, on a card where the PyTorch reference cannot load the model at all. The graph right-sizing fix also carries to CUDA at a similar factor: ~940 ms to ~126 ms, about 7.5x, for the mismatched-capacity case. Bing Bong
…pass The parity evidence in these docs was inherited: bit-exactness measured against an internal pre-change baseline, with the NeMo reference dump itself lost to a reboot partway through. Regenerated it (torch 2.13.0+cu130, nemo 2.7.3) and re-ran against the current tree. All three stages pass on CPU and CUDA. enc_out cosine is 0.972258 on both, identical to the value measured before any of the optimizations in this report — which is the strongest available statement that they changed no math, since it is checked against NeMo directly rather than against our own earlier output. Bing Bong
The quoted CPU tuned figure (~750 ms) was inherited from a session that predated the optimization pass, and this machine drifts 20-30% thermally, so it was neither current nor safely comparable to the rows around it. Re-measured with the drift-cancelling harness (ABBA, 4 passes, median of 5 iterations per run, machine idle): default (8 threads, f32) ~1245 ms CPU / ~169 ms CUDA tuned (12 threads, q8_0) ~715 ms CPU / ~131 ms CUDA ratio 1.72x / 1.31x The default rows land within a couple percent of the historical figures (1247.5 / 170.6), which is a useful check that the older table still means something. The CPU tuned row is the one that moved: ~715 ms rather than ~750 ms, i.e. 10.4x real-time rather than 9.9x. Also labels the historical table as a progression whose rows were each measured in their own session, so it is not read as row-to-row comparable. Bing Bong
Owner
|
@dleiferives Thank you for your contribution! I'd appreciate it if you could update the spec to follow the current model_specs_v1/ format, but without adding a schema version number. The existing code will simply ignore the new v1 fields, so this won’t break anything. This is just to make the eventual migration easier. |
The rows were labelled "default settings" and "q8_0 + tuned threads", which does not make clear that both are *this port* and only the first row is the Python reference — the question "is default the Python one?" came up reading it, so the labelling was doing real damage on the table a reader hits first. Splits implementation and settings into separate columns, and states the comparison the table exists to make: untuned this port is slower than PyTorch on CPU (~1245 ms vs 857.7 ms), tuned it wins by ~1.2x, and on this GPU there is no comparison because PyTorch cannot load the model in 4 GB at all. Also spells out what "untuned" means, so it cannot be read as a deliberately handicapped baseline: --threads defaults to 8 and matmul_weight_type defaults to native, which is f32 for this checkpoint. Bing Bong
Requested in review. parakeet_tdt was the only family present in model_specs/ but missing from model_specs_v1/ (34 of 35), so this closes that gap ahead of the eventual migration. Deliberately carries no version/schema_version field, per the review note — none of the existing 34 v1 specs has one either. Nothing existing changes behaviour: - model_specs/parakeet_tdt.json is left exactly as it was. Every other family keeps its minimal legacy spec (family + sources) alongside the enriched v1 file, and this matches that. - The `sources` block in the v1 spec is byte-identical to the legacy one, so the resource resolution the loader actually performs is unchanged. - Verified empirically rather than assumed: running the loader against a spec file containing the full v1 field set transcribes correctly, confirming the extra fields are ignored. - `tools/model_manager_v2.py --specs-dir model_specs_v1 list` now lists parakeet_tdt, where before it was absent. Modelled on nemotron_asr (the closest analogue — NVIDIA ASR loaded from a safetensors checkpoint) with the community-model package shape used by glm_tts/outetts/vietneu_tts: status "community", no package_defaults, and the Hugging Face snapshot declared per-package. Top-level key set now matches those peers exactly. Also fixes a real gap the migration surfaced: `encoder_flash_attention` is accepted by the session but was never advertised in the loader's option list, so it was undiscoverable. session.cpp, loader.cpp and the v1 spec now declare the identical set of seven session options. Bing Bong
dleiferives
changed the title
Contributor
Author
Yeah, of course. Just implemented & updated pr body to denote work and results more accurately. |
dleiferives
marked this pull request as ready for review
July 27, 2026 17:00
Owner
|
@dleiferives You can check my old impl of https://github.com/0xShug0/audio.cpp/tree/preview/parakeet_tdt_old. That implementation was done before the 0.1 release and isn't very polished, but it may still be useful. |
Owner
|
@dleiferives Please check #128. The example is Confucius4-TTS. |
Contributor
Author
|
Awesome, that is super helpful thank you. |
Reference: 0xShug0/audio.cpp preview/parakeet_tdt_old Adapted: decoder emission-frame capture, persistent predictor-state lifecycle, and token-piece-to-word timestamp merging
Reference: 0xShug0/audio.cpp preview/parakeet_tdt_old Adapted: bounded left-center-right window scheduling, center-only incremental decoding, retained predictor state, partial transcript merging, and long-form window reuse
Reference: 0xShug0/audio.cpp preview/parakeet_tdt_old Adapted: continued bounded-window streaming lifecycle work; framework event delivery and public option naming were rewritten for the current API
Use the schema-v1 audio_chunk_threshold_sec contract from issue 0xShug0#128 and enforce the spec's duration minima in the runtime parser.
# Conflicts: # CMakeLists.txt # src/framework/runtime/registry.cpp
Contributor
Author
|
Implemented the requested Parakeet follow-up and aligned it with the schema-v1 flow described in #128, using Confucius4-TTS as the reference shape. What changed:
I reviewed Validation on the merged tree:
The main schema/streaming work is in |
Owner
|
@dleiferives Merged! Thank you for contributing the new model! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Native port of
nvidia/parakeet-tdt-0.6b-v3,loaded directly from the Transformers-compatible
model.safetensorscheckpointrather than the
.nemoarchive. Offline (full-context) transcription, CPU andCUDA.
Validation
Tier 1 — golden transcription (CI)
ctest -R parakeet_golden_transcription_test, wired intoENGINE_BUILD_TESTS,skips cleanly when the model isn't installed. Asserts the decoded text of a
checked-in LibriSpeech test-clean clip matches NeMo's exactly.
Tier 2 — numerical parity against NeMo (manual)
tests/parakeet_tdt/parity/compares mel features, an isolated encoder layer,and full encoder output against real NeMo activations captured via forward
hooks (torch 2.13.0+cu130, nemo 2.7.3). Current results, on both backends:
mel_featureslayer_0enc_outenc_out's looser bound is float32 summation order across a ~2M-node graph,not a correctness gap — the README explains why, and it was verified by
chaining 24 isolated single-layer builds, which holds cosine >= 0.999999 the
whole way. That 0.972258 is identical to the value measured before the
optimization pass, which is the strongest available statement that none of
the performance work changed the math: it is checked against NeMo directly
rather than against our own earlier output.
This tier found true bugs that the golden tests missed
Tier 3 — multilingual accuracy (new)
tests/parakeet_tdt/multilingual/— 120 FLEURS clips across 24 of the 25supported languages, 21.8 min of audio. Raw transcripts for every clip and
weight type are checked in under
multilingual/results/so the numbers can bere-derived without re-running or re-downloading.
This exists because one clip in one language is not enough to clear a
quantization:
q8_0is byte-identical on the golden clip, and across 24languages it changes 8.3% of transcripts.
Backends
Both, on the same machine, agreeing numerically:
enc_outagrees between CPU and CUDA to cosine 0.99999988 (rel RMS3.5e-06, mel bit-identical) — float32 accumulation noise between different
kernels. Checked with the parity dump's
--backendflag.Worth noting:
nemo_asrcannot load this model on this card at all —PyTorch's overhead on top of the weights exhausts the 4 GB budget. The native
path runs in 2642 MiB.
Performance
Reference clip (7.435s), Intel i7-9750H (6C/12T) + GTX 1650 Max-Q:
q8_0Untuned, this port is slower than PyTorch on CPU (~1245 ms vs 857.7 ms).
Tuned it wins by ~1.2x, and on this GPU there is no comparison to make: the
PyTorch reference cannot load the model in 4 GB, while the native path runs it
in 2642 MiB. "Untuned" is just the two defaults —
--threads8, andmatmul_weight_typenative, which is f32 for this checkpoint.Measured with a drift-cancelling harness (ABBA ordering, 4 passes, median of 5
iterations per run, machine idle) — this laptop swings 20-30% between cold and
thermally saturated, so sequential before/after timing is not trustworthy at
this scale. Tuned is 1.72x default on CPU, 1.31x on CUDA.
The largest single win was not a kernel or precision change — it was a
correctness bug. The encoder graph runs at whatever capacity it was built for,
and
encode()zero-pads up to that capacity, so a session prepared with a longaudio contract paid the long price on every short clip:
And it was simultaneously wrong. Zero input doesn't stay zero (the subsampling
convs have biases), and the attention mask was unconditionally all-zeros, so
real frames attended to padded garbage. On the reference clip that silently
dropped an entire sentence:
Fixed in two commits (mask padded keys; rebuild rather than run oversized), and
the same fix carries to CUDA at ~7.5x.
Three further CPU optimizations are bit-exact — verified identical across
all 95,232
enc_outfloats: attention operands read as strided views insteadof copies, constant scales folded into weights (exact, both are powers of two),
and a redundant transpose pair removed.
Weight precision
parakeet_tdt.matmul_weight_typeacceptsnative|f32|f16|bf16|q8_0.native(f32)f16bf16q8_0q8_0changes 8.3% of transcripts but absolute WER does not move. The churn ismostly formatting (
T Rex→T-Rex), and where words change it goes bothdirections — it is sometimes right where f32 was wrong. It costs
reproducibility against an f32 baseline, not quality. It also cuts peak RSS
3.6x, which may matter more than the speed on constrained machines.
nativestays the default as the only setting that reproduces exactly.Build and run
Package id
parakeet_tdt, model dirmodels/parakeet-tdt-0.6b-v3.tools/check_loader_catalog_sync.pyreports catalog and loaders in sync.Known limitations
att_context_style="regular",att_context_size=[-1,-1]— unlimited bidirectional attention, not NeMo'schunked_limitedstreaming style.setup_streaming_params()doesn't errorbut produces a degenerate config (~5.8s chunks, ~10000-frame cache) with none
of the real benefit. NVIDIA's maintainers point at a different architecture
for streaming. Real support needs a purpose-trained checkpoint; this repo
already has
nemotron_asrfor that.falls off a cliff under quantization, not publishable per-language figures.
is below the 8.0 threshold where ggml enables CUDA graph capture. Several
optimizations measured as no change on CUDA; that is this card, not the
optimization. Worth re-measuring on Ampere or newer.
Docs
docs/community_models/parakeet_tdt.md— user-facing (162 lines)docs/reports/parakeet_tdt_performance.md— full methodology, profiling, andthe optimizations tried and rejected with numbers, so they don't get
re-attempted
tests/parakeet_tdt/{parity,multilingual,bench}/README.md— harnesses