CosyVoice: seeded long-form synthesis + upstream parity fixes#378
Merged
Conversation
ivan-digital
force-pushed
the
feat/cosyvoice-seeded-longform
branch
from
8cc289c to
ba1b954
Compare
Thread an optional random seed through the speak CLI into the CosyVoice long-form voice-cloning path so multi-segment synthesis is reproducible, with tests. Work in progress preserved from the local worktree.
ivan-digital
force-pushed
the
feat/cosyvoice-seeded-longform
branch
from
ba1b954 to
ac6cb4f
Compare
Eight inference divergences from upstream CosyVoice caused reverby, voice-unstable cloning output with onset clicks and occasional recitation of the reference transcript: - PreLookaheadLayer: restore leaky_relu(0.01) and the residual connection the DiT was trained with - Pass the CAM++ x-vector into the voice profile so the flow runs speaker-conditioned - Fixed flow noise: load the upstream seed-0 rand_noise buffer from the bundle (flow_noise.bin) or COSY_FIXED_NOISE, fall back to a keyed draw; ODE solver state and CFG combine now run float32 - Add a compiled DiT fast path for the spks+cond (cloning) inputs - Rewrite the NSF source to SineGen2-causal semantics: %1 phase wrap, frame-rate cumsum, nearest staircase upsample, unvoiced excitation at sine_amp/3 (uniform), no extra noise on the merged source; clip the ISTFT magnitude at 1e2 - Slice the mel at the prompt boundary before vocoding instead of cutting rendered audio; drop the 300 ms prompt-leak trim - Zero-shot LLM text layout: "You are a helpful assistant.<|endofprompt|>" + transcript + content; the previous transcript-first layout made the model recite the reference transcript mid-utterance - ISTFT: trim n_fft/2 edge samples (center=True semantics) and fade the first stitched segment head — both were audible clicks at render start Validation: ASR round-trip is word-exact across seeds and segment paths; speaker-embedding cosine to the reference improves 0.59 -> 0.81; RTF 0.45 -> 0.39. Adds FlowVocoderParityTests and updates the long-form stitch tests to the every-edge-fades contract.
ivan-digital
changed the title
The fixed flow-noise buffer now ships in the HF bundles. Add it to the bundle download list and fetch it best-effort into caches created before the file existed (the main download is skipped once weights are present). Bundles without the file match nothing in the glob snapshot, so loading still falls back to the keyed deterministic draw.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thread an optional random seed from the
speakCLI through the CosyVoice long-form voice-cloning path so multi-segment synthesis is reproducible run-to-run, and fix eight inference divergences from upstream CosyVoice that degraded cloning quality (reverby, voice-unstable output with onset clicks and occasional recitation of the reference transcript).Parity fixes
PreLookaheadLayer: leaky_relu(0.01) + the residual connection the DiT was trained withflow_noise.binorCOSY_FIXED_NOISE, keyed deterministic fallback; ODE solver state and CFG combine run float32; compiled DiT fast path for the cloning inputs%1phase wrap, frame-rate cumsum, nearest staircase upsample, unvoiced excitation atsine_amp/3(uniform), no extra noise on the merged source; ISTFT magnitude clipped at1e2feat[:, :, mel_len1:]) instead of cutting rendered audio; the 300 ms prompt-leak trim is removed — the leak it papered over no longer exists"You are a helpful assistant.<|endofprompt|>" + transcript + content; the previous transcript-first layout made the model recite the reference transcript mid-utterancen_fft/2edge samples (torchcenter=Truesemantics) and the long-form stitcher fades every segment edge including the first head — both were audible clicks at render startValidation
FlowVocoderParityTests(7 regression tests, each fails on the pre-fix code) plus updated long-form stitch tests; full unit suite greenRegression risk: cloning output changes by design; text-only TTS is touched only by the PreLookaheadLayer and solver-precision corrections, both of which move it toward upstream. The seeded reproducibility contract is preserved — flow noise is now fixed by default and LLM sampling still follows
--seed.