[RNE Rewrite] feat: add voice activity detection pipeline#1298
Draft
msluszniak wants to merge 5 commits into
Draft
[RNE Rewrite] feat: add voice activity detection pipeline#1298msluszniak wants to merge 5 commits into
msluszniak wants to merge 5 commits into
Conversation
msluszniak
force-pushed
the
@ms/rewrite-vad
branch
from
43021d3 to
6321295
Compare
Port the VAD feature to the rewrite as a pure-TypeScript pipeline on top of the core model.execute primitive (no new C++): - src/extensions/speech/tasks/vad.ts: createVAD runner replicating the native FSMN-VAD algorithm (framing + Hann window + pre-emphasis, chunked inference, thresholding / min-duration / padding / merge). Segments are returned in seconds. Relies on the get_dynamic_dims relaxed input validation for the dynamic frame dimension; the fsmn-vad model is re-exported with it. - src/extensions/speech/vadStreamer.ts: pure streaming state machine driving onSpeechBegin / onSpeechEnd over an accumulating buffer. - src/hooks/useVAD.ts: hook wrapping createVAD + streamer lifecycle. - Register models.vad.FSMN_VAD and export the speech extension. - apps/speech: expo-router demo (mirrors apps/nlp) with a real-time mic VAD screen via react-native-audio-api.
…arams into model config Frame geometry (sample rate, window/hop, FFT size, pre-emphasis, min frames) is FSMN-specific and now lives on VADModel.featureConfig (supplied by the models registry) instead of hardcoded constants in the task. The pipeline and streamer are parameterized by it; detection thresholds remain generic VADOptions.
Framing (windowing, mean-removal, pre-emphasis, Hann) was ~86% of a detect() call on device — ~40ms of Hermes vs ~6ms for the model forward pass. Port it to a native `speech.frameWaveform` op that writes directly into the pre-allocated model-input tensor, fusing mean-removal + pre-emphasis + Hann into a single dependency-free (vectorizable) pass. Framing drops to ~3.4ms on device (~12x), leaving it below the model's own inference cost.
msluszniak
force-pushed
the
@ms/rewrite-vad
branch
from
f76a5c0 to
ae3f0da
Compare
The streamer re-ran the model over the whole growing buffer every tick (up to ~8ms on a 10s buffer on device), even though only ~100ms is new and the streaming decision only needs recent context. Cap the sliding window to 2.5s so per-tick detect stays flat and low (~3x cheaper, ~2.7ms) — still well above FSMN's receptive field and the 250ms min-speech-duration, so detection is unaffected. On-device benchmark showed the alternative 'O(1) insert' buffer only saved ~6us/tick (0.2%), so it was dropped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a Voice Activity Detection (VAD) task pipeline and a corresponding
speechexample app. Chunked inference, segment postprocessing and streaming run in TypeScript on top of the coremodel.executeprimitive. The per-frame feature extraction (framing, mean-removal, pre-emphasis, Hann window) is a nativespeech.frameWaveformC++ op: on device it dominated adetect()call (~86%, ~40 ms of Hermes vs ~6 ms for the model forward pass), so per the extension guidelines it lives in C++. It writes straight into the pre-allocated model-input tensor, fusing mean-removal + pre-emphasis + Hann into one dependency-free (vectorizable) pass; framing drops to ~3.4 ms (~12×), below the model's own inference cost.Introduces a breaking change?
Type of change
Tested on
Testing instructions
speechapp on iOS and AndroidScreenshots
Related issues
Closes #1249
Checklist
Additional notes
get_dynamic_dims_forwardinput validation from the embeddings PR ([RNE Rewrite] Add text and image embeddings pipelines #1292): VAD feeds a variable-length[frames, 512]input tensor per chunk. Outputs are still validated exactly, so the output tensor is pre-allocated at the model-declared shape. Requires [RNE Rewrite] Add text and image embeddings pipelines #1292 to land. Thefsmn-vadmodel is re-exported (tagv0.10.0) with aget_dynamic_dims_forwardmethod returning int32[rank, 3]bounds per input.[1, frames, classes]with class 0 = non-speech (speech = 1 - p0), matching the current native implementation.