Skip to content

Fix QuickTalk silence gating - #137

Open
cwang0810 wants to merge 1 commit into
datascale-ai:mainfrom
cwang0810:codex/quicktalk-silence-gate
Open

Fix QuickTalk silence gating#137
cwang0810 wants to merge 1 commit into
datascale-ai:mainfrom
cwang0810:codex/quicktalk-silence-gate

Conversation

@cwang0810

Copy link
Copy Markdown
Contributor

Summary

  • Add QuickTalk PCM silence gating so fully silent chunks render idle frames without running HuBERT or the QuickTalk model.
  • Add per-frame silence masking inside mixed audio chunks, so silent frames fall back to idle while voiced frames still render normally.
  • Keep QuickTalk warmup able to run through the model by bypassing the silence gate for warmup only.

Validation

  • pytest tests/unit/test_quicktalk_adapter.py tests/unit/test_render_pipeline.py tests/frontend/test_quicktalk_send_path.py -q
  • python -m py_compile opentalking/models/quicktalk/adapter.py opentalking/pipeline/speak/render_pipeline.py tests/unit/test_quicktalk_adapter.py
  • Verified on a GPU host with real quicktalk.pth and real avatars:
    • female-host-transparent pure silence: HuBERT/model calls dropped from 1/1 to 0/0, idle-frame pixel diff became 0.
    • mixed silence/voice chunk: silent mask became [true,false,true,false], only voiced frames ran model inference.
    • anime-handsome-guy pure silence: HuBERT/model calls 0/0, all output frames matched idle frames exactly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant