Model capability registry: descriptors, dictation, synthesis hardening#39
Merged
Conversation
added 3 commits
July 4, 2026 10:02
Ship the remaining pieces on top of the capability registry: Dictation: Parakeet ASR through the sidecar (transcribe_parakeet), a save-and-transcribe IPC path with its own cache dir, an ASR model registry, and a DictationPanel front end. Model plumbing: pass the registry-selected modelId through init_model and the synthesize commands so holders reload when the artifact changes; default VoxCPM2 to the bf16 bundle; registry-driven per-engine seed ladders, token budgets, and language normalization. Synthesis output conditioning: every engine's WAV now gets a silent lead-in, edge fades, and a post-roll (conditionAudioEdges). Qwen3 ICL and OmniVoice additionally open some renders (reference-dependent) with a short low-level codec artifact before the first word — quiet hiss on the Marek reference, a low hum on Anna — followed by a silence gap. trimLeadingJunk removes it via energy-island analysis: leading islands that are short, early, gap-separated, and either ≥20 dB below the loudest island or 8-20 dB below with a pure low-frequency ZCR signature. Clean renders pass through bit-exact. A fixed pre-trim could not work here: the residue runs past 150 ms on affected renders while unaffected ones start voiced within 25 ms. Indic-Mio markers: map the Studio's inline emotion markers onto the model's closed suffix-tag vocabulary. The model reads unknown parentheticals aloud, so unmapped markers are dropped rather than passed through. Demo: lazy clip seeding with stable cache ids; exact bundled reference transcripts (a stale Anna transcript leaked prompt words into takes); Qwen goes single-shot at a fixed seed; the Hindi demo gains a second, female FLEURS speaker (the existing reference is the male recording of its sentence) with lines alternating across both voices. Tests: sidecar e2e for onset cleanliness on both affected engines and marker handling on Indic-Mio (ignored, model-backed), fast protocol tests for engine mismatch and missing fields, and frontend unit tests for the new synth, clip-audio, media-src, and language helpers.
ivan-digital
changed the title
added 3 commits
July 5, 2026 12:35
Pins buildHindiDemoProject against the two-speaker seed: one track per voice, alternating lines routed to the matching track. Guards the demo shape that a stale saved project can otherwise mask during manual testing.
… editing UX Synthesis: every engine now renders in a single pass. Removed the seed/cfg retry ladder and the ASR grading gate entirely — the 16-bit models produce intelligible speech in one shot (validated by a synth→ASR roundtrip across all engines), and the macOS grader (Parakeet) could not read non-Latin scripts, so it scored good Hindi/CJK takes near 0% and burned the whole ladder for nothing. The seed now varies per synthesis (derived from the per-call invocation salt), so a Regenerate rolls a fresh take — the manual escape hatch for an unlucky render, instead of an automatic gate. Indic-Mio: cache the WavLM speaker embedding per reference so it is computed once per voice instead of once per attempt; combined with single-shot this takes warm synthesis from multiple laddered attempts to ~0.7 RTF. Map inline emotion markers onto the model's closed suffix-tag vocabulary so it stops reading the marker aloud. Onset cleanup: a shared conditionSynthOutput helper strips leading codec/reference-boundary junk (a short sub-speech blip before the first word on some reference/seed combinations) and edge-conditions, wired into every cloning engine. Detects genuine junk by energy islands well below the speech body; passes clean audio through unchanged (never trims a real quiet onset word). Editing UX: explicit Save button in the top bar (autosave still runs); Add script line from the track inspector and track list so a clip can be created without dictation; a microphone in the script editor to dictate a line directly into the field (shared useDictationRecorder hook with the dictation panel); responsive top bar that no longer wraps or overlaps at narrow widths.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Registry-driven engine rollout, end to end:
model-registry.jsonbecomes the single source of truth for engine/ASR capabilities, and the Studio, Rust shell, and Swift sidecar all consume it. On top of that: Parakeet dictation, synthesis output conditioning with onset-artifact removal, Indic-Mio emotion-marker support, and a two-voice Hindi demo.What changed
modelIdflows throughinit_modeland every synthesize command so sidecar holders reload on artifact switches; VoxCPM2 defaults to the bf16 bundle.transcribe_parakeetin the sidecar, save-and-transcribe IPC with its own cache dir, ASR model registry,DictationPanelUI.conditionAudioEdges(silent lead-in, edge fades, post-roll) for every engine, plustrimLeadingJunkfor a reference-dependent leading codec artifact on Qwen3 ICL (quiet hiss, ~100 ms, Marek reference) and OmniVoice (low hum, ~200 ms, Anna reference). Energy-island analysis removes short, early, gap-separated islands ≥20 dB below the loudest island (or 8–20 dB below with a pure low-frequency ZCR signature); clean renders pass through bit-exact. A fixed pre-trim can't cover this: residue runs past 150 ms on affected renders while unaffected ones start voiced within 25 ms.happy/sad/angry/disgust/fear/surprise); unmapped markers are dropped — the model reads unknown parentheticals aloud.Test plan
cargo test --lib— 38 passedcargo test --test sidecar— 6 fast protocol tests passed (incl. new engine-mismatch / missing-field rejections)cargo test --test sidecar -- --ignored qwen_onset_is_speech_not_junk omnivoice_onset_is_speech_not_junk indic_mio_does_not_speak_emotion_marker— 3 passed against real models: trims fire at 610 ms / 390 ms on the previously affected renders, first island is speech-loud, Parakeet coverage ≥ 60 %, marker never spokenpnpm test— 42 passed;tsc --noEmitclean