From da1657d04db403a8dd897a2b8bedaa31fa285ba4 Mon Sep 17 00:00:00 2001 From: Ivan Date: Sun, 5 Jul 2026 11:56:28 +0200 Subject: [PATCH] Add Audio2Face-3D avatar motion runtime MLX Swift port of NVIDIA Audio2Face-3D: a hand-written graph runs the official model forward pass and emits timestamped coefficient frames for speech-driven avatars. - Audio2Face3D library target; fromPretrained downloads the published MLX bundles (Mark v2.3 default, Claire and James v2.3.1) from Hugging Face through the shared Hub path. - speech avatar-motion CLI command: WAV in, JSONL coefficient frames out. - Deterministic unit tests, ONNX parity fixtures, and E2E tests that download each published bundle and run real inference. - Model doc covering the network architecture and identity layouts; catalogue entries in README.md and all 13 translations. --- .gitignore | 3 + AGENTS.md | 1 + Package.swift | 19 + README.md | 6 +- README_ar.md | 6 +- README_de.md | 6 +- README_es.md | 6 +- README_fr.md | 6 +- README_hi.md | 6 +- README_ja.md | 6 +- README_ko.md | 6 +- README_pt.md | 6 +- README_ru.md | 6 +- README_th.md | 6 +- README_tr.md | 6 +- README_vi.md | 6 +- README_zh.md | 6 +- .../Audio2Face3D/Audio2Face3DDownloader.swift | 135 ++++ Sources/Audio2Face3D/Audio2Face3DFrame.swift | 50 ++ .../Audio2Face3D/Audio2Face3DMLXRuntime.swift | 696 ++++++++++++++++++ Sources/Audio2Face3D/Audio2Face3DModel.swift | 101 +++ Sources/Audio2Face3D/Configuration.swift | 91 +++ Sources/AudioCLILib/AudioCLI.swift | 1 + Sources/AudioCLILib/AvatarMotionCommand.swift | 76 ++ .../Audio2Face3DParityTests.swift | 39 + .../Audio2Face3DTests/Audio2Face3DTests.swift | 114 +++ .../E2EAudio2Face3DTests.swift | 60 ++ docs/models/audio2face3d.md | 67 ++ 28 files changed, 1509 insertions(+), 28 deletions(-) create mode 100644 Sources/Audio2Face3D/Audio2Face3DDownloader.swift create mode 100644 Sources/Audio2Face3D/Audio2Face3DFrame.swift create mode 100644 Sources/Audio2Face3D/Audio2Face3DMLXRuntime.swift create mode 100644 Sources/Audio2Face3D/Audio2Face3DModel.swift create mode 100644 Sources/Audio2Face3D/Configuration.swift create mode 100644 Sources/AudioCLILib/AvatarMotionCommand.swift create mode 100644 Tests/Audio2Face3DTests/Audio2Face3DParityTests.swift create mode 100644 Tests/Audio2Face3DTests/Audio2Face3DTests.swift create mode 100644 Tests/Audio2Face3DTests/E2EAudio2Face3DTests.swift create mode 100644 docs/models/audio2face3d.md diff --git a/.gitignore b/.gitignore index 3306814e..4aa45490 100644 --- a/.gitignore +++ b/.gitignore @@ -10,6 +10,9 @@ DerivedData/ __pycache__/ *.pyc +# Model export outputs (generated by speech-models export scripts) +out/ + # Model weights cosyvoice3-mlx-4bit/ personaplex-mlx-4bit/ diff --git a/AGENTS.md b/AGENTS.md index 0687b024..cb5be226 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -15,6 +15,7 @@ AI speech models for Apple Silicon (MLX Swift). ASR, TTS, speech-to-speech, VAD, - **Always work in a separate git worktree** so concurrent agents don't fight over the same working directory. Create one with `git worktree add ../speech-swift- `, do all edits there, push from there. Multiple agents may be running against this repo at the same time — checking out branches in the shared working copy clobbers their state and silently loses WIP files. Delete the worktree (`git worktree remove`) when the task is done. - **Never commit, push, or comment on GitHub without explicit user confirmation.** Draft first, ask to confirm, then execute. - **Every README.md change must update all 10 translations** (`README_zh.md`, `README_ja.md`, `README_ko.md`, `README_es.md`, `README_de.md`, `README_fr.md`, `README_hi.md`, `README_pt.md`, `README_ru.md`, `README_ar.md`). No exceptions. +- **Keep docs and comments scoped to this package.** Model docs, code comments, and PR descriptions describe this package's models, APIs, and formats only — never downstream consumer apps or their integration rules. ## Git Conventions diff --git a/Package.swift b/Package.swift index bb259a5d..4708278a 100644 --- a/Package.swift +++ b/Package.swift @@ -148,6 +148,10 @@ let package = Package( name: "SpeechWakeWord", targets: ["SpeechWakeWord"] ), + .library( + name: "Audio2Face3D", + targets: ["Audio2Face3D"] + ), .executable( name: "speech", targets: ["AudioCLI"] @@ -531,6 +535,13 @@ let package = Package( "AudioCommon" ] ), + .target( + name: "Audio2Face3D", + dependencies: [ + "AudioCommon", + .product(name: "MLX", package: "mlx-swift") + ] + ), .target( name: "AudioCLILib", dependencies: [ @@ -560,6 +571,7 @@ let package = Package( "MagpieTTSCoreML", "MADLADTranslation", "SpeechWakeWord", + "Audio2Face3D", "AudioCommon", .product(name: "MLX", package: "mlx-swift"), .product(name: "MLXRandom", package: "mlx-swift"), @@ -935,6 +947,13 @@ let package = Package( .copy("Resources/kws_lovely_child.wav"), .copy("Resources/ref_encoder_light_up.bin") ] + ), + .testTarget( + name: "Audio2Face3DTests", + dependencies: [ + "Audio2Face3D", + "AudioCommon" + ] ) ] ) diff --git a/README.md b/README.md index 59139b35..9e9c239b 100644 --- a/README.md +++ b/README.md @@ -60,6 +60,7 @@ On-device speech recognition, synthesis, and understanding for Mac and iOS. Runs - **[Hibiki Zero-3B](https://soniqo.audio/guides/audio-translate)** — Streaming speech-to-speech translation (FR/ES/PT/DE → EN, MLX INT4 + INT8, Kyutai Moshi/Mimi stack, CC-BY-4.0) - **[PersonaPlex](https://soniqo.audio/guides/respond)** — Full-duplex speech-to-speech (7B, audio in → audio out, 18 voice presets) +- **[Audio2Face-3D](docs/models/audio2face3d.md)** — Speech-driven facial animation for avatars (NVIDIA Audio2Face-3D v2.3 Mark, 301 facial coefficients, MLX) **Enhancement, Separation & Audio Generation** @@ -144,7 +145,7 @@ struct DictateView: View { `SpeechUI` ships only `TranscriptionView` (finals + partials) and `TranscriptionStore` (streaming ASR adapter). Use AVFoundation for audio visualization and playback. -Available SPM products: `Qwen3ASR`, `WhisperASR`, `Qwen3TTS`, `Qwen3TTSCoreML`, `ParakeetASR`, `ParakeetStreamingASR`, `NemotronStreamingASR`, `OmnilingualASR`, `KokoroTTS`, `SupertonicTTS`, `VibeVoiceTTS`, `CosyVoiceTTS`, `VoxCPM2TTS`, `ChatterboxTTS`, `OmniVoiceTTS`, `IndicMioTTS`, `FishAudioTTS`, `MagpieTTS`, `MagpieTTSCoreML`, `MAGNeTMusicGen`, `StableAudio3MusicGen`, `FlashSR`, `PersonaPlex`, `HibikiTranslate`, `MADLADTranslation`, `SpeechVAD`, `SpeechWakeWord`, `SpeechEnhancement`, `SpeechRestoration`, `SourceSeparation`, `Qwen3Chat`, `FunctionGemma`, `SpeechCore`, `SpeechUI`, `AudioCommon`. +Available SPM products: `Qwen3ASR`, `WhisperASR`, `Qwen3TTS`, `Qwen3TTSCoreML`, `ParakeetASR`, `ParakeetStreamingASR`, `NemotronStreamingASR`, `OmnilingualASR`, `KokoroTTS`, `SupertonicTTS`, `VibeVoiceTTS`, `CosyVoiceTTS`, `VoxCPM2TTS`, `ChatterboxTTS`, `OmniVoiceTTS`, `IndicMioTTS`, `FishAudioTTS`, `MagpieTTS`, `MagpieTTSCoreML`, `MAGNeTMusicGen`, `StableAudio3MusicGen`, `FlashSR`, `PersonaPlex`, `Audio2Face3D`, `HibikiTranslate`, `MADLADTranslation`, `SpeechVAD`, `SpeechWakeWord`, `SpeechEnhancement`, `SpeechRestoration`, `SourceSeparation`, `Qwen3Chat`, `FunctionGemma`, `SpeechCore`, `SpeechUI`, `AudioCommon`. ## Models @@ -179,6 +180,7 @@ Compact view below. **[Full model catalogue with sizes, quantisations, download | [MADLAD-400](https://soniqo.audio/guides/translate) | Text → Text (Translation) | MLX | 3B | **400+** | | [Hibiki Zero-3B](https://soniqo.audio/guides/audio-translate) | Speech → Speech (Translation) | MLX | 3B | FR/ES/PT/DE → EN | | [PersonaPlex](https://soniqo.audio/guides/respond) | Speech → Speech | MLX | 7B | EN | +| [Audio2Face-3D](docs/models/audio2face3d.md) | Speech → Facial animation | MLX | v2.3 Mark | Agnostic | | [Silero VAD](https://soniqo.audio/guides/vad) | Voice Activity Detection | MLX, CoreML | 309K | Agnostic | | [KWS Zipformer](docs/models/kws-zipformer.md) | Audio → Wake word | CoreML (ANE) | 3M | EN/custom keywords | | [Pyannote](https://soniqo.audio/guides/diarize) | VAD + Diarization | MLX | 1.5M | Agnostic | @@ -456,7 +458,7 @@ speech-swift is split into one SPM target per model so consumers only pay for wh **[Full architecture diagram with backends, memory tables, and module map → soniqo.audio/architecture](https://soniqo.audio/architecture)** · **[API reference → soniqo.audio/api](https://soniqo.audio/api)** · **[Benchmarks → soniqo.audio/benchmarks](https://soniqo.audio/benchmarks)** Local docs (repo): -- **Models:** [Qwen3-ASR](docs/models/asr-model.md) · [WhisperASR](docs/models/whisper-asr.md) · [Qwen3-TTS](docs/models/tts-model.md) · [CosyVoice](docs/models/cosyvoice-tts.md) · [Kokoro](docs/models/kokoro-tts.md) · [VoxCPM2](docs/models/voxcpm2-tts.md) · [VibeVoice](docs/models/vibevoice.md) · [Supertonic](docs/models/supertonic-tts.md) · [Chatterbox](docs/models/chatterbox-tts.md) · [Indic-Mio](docs/models/indic-mio-tts.md) · [Fish Audio S2 Pro](docs/models/fish-audio-s2-pro.md) · [Magpie TTS](docs/models/magpie-tts.md) · [Parakeet TDT](docs/models/parakeet-asr.md) · [Parakeet Streaming](docs/models/parakeet-streaming-asr.md) · [Nemotron Streaming](docs/models/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/models/omnilingual-asr.md) · [PersonaPlex](docs/models/personaplex.md) · [Hibiki](docs/models/hibiki.md) · [MADLAD-400](docs/models/madlad-translation.md) · [FunctionGemma](docs/models/function-gemma.md) · [Qwen3.5 Chat](docs/models/qwen35-chat.md) · [Gemma 4 Chat](docs/models/gemma4-chat.md) · [Qwen3 Dense Chat](docs/models/qwen3-dense-chat.md) · [FireRedVAD](docs/models/fireredvad.md) · [KWS Zipformer](docs/models/kws-zipformer.md) · [Sidon](docs/models/sidon.md) · [Source Separation](docs/models/source-separation.md) · [HTDemucs](docs/models/htdemucs.md) · [MAGNeT](docs/models/magnet-music-gen.md) · [Stable Audio 3](docs/models/stable-audio-3.md) · [FlashSR](docs/models/flashsr.md) +- **Models:** [Qwen3-ASR](docs/models/asr-model.md) · [WhisperASR](docs/models/whisper-asr.md) · [Qwen3-TTS](docs/models/tts-model.md) · [CosyVoice](docs/models/cosyvoice-tts.md) · [Kokoro](docs/models/kokoro-tts.md) · [VoxCPM2](docs/models/voxcpm2-tts.md) · [VibeVoice](docs/models/vibevoice.md) · [Supertonic](docs/models/supertonic-tts.md) · [Chatterbox](docs/models/chatterbox-tts.md) · [Indic-Mio](docs/models/indic-mio-tts.md) · [Fish Audio S2 Pro](docs/models/fish-audio-s2-pro.md) · [Magpie TTS](docs/models/magpie-tts.md) · [Parakeet TDT](docs/models/parakeet-asr.md) · [Parakeet Streaming](docs/models/parakeet-streaming-asr.md) · [Nemotron Streaming](docs/models/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/models/omnilingual-asr.md) · [PersonaPlex](docs/models/personaplex.md) · [Hibiki](docs/models/hibiki.md) · [MADLAD-400](docs/models/madlad-translation.md) · [FunctionGemma](docs/models/function-gemma.md) · [Qwen3.5 Chat](docs/models/qwen35-chat.md) · [Gemma 4 Chat](docs/models/gemma4-chat.md) · [Qwen3 Dense Chat](docs/models/qwen3-dense-chat.md) · [FireRedVAD](docs/models/fireredvad.md) · [KWS Zipformer](docs/models/kws-zipformer.md) · [Sidon](docs/models/sidon.md) · [Source Separation](docs/models/source-separation.md) · [HTDemucs](docs/models/htdemucs.md) · [MAGNeT](docs/models/magnet-music-gen.md) · [Stable Audio 3](docs/models/stable-audio-3.md) · [FlashSR](docs/models/flashsr.md) · [Audio2Face-3D](docs/models/audio2face3d.md) - **Inference:** [Qwen3-ASR](docs/inference/qwen3-asr-inference.md) · [WhisperASR](docs/inference/whisper-asr-inference.md) · [Parakeet TDT](docs/inference/parakeet-asr-inference.md) · [Parakeet Streaming](docs/inference/parakeet-streaming-asr-inference.md) · [Nemotron Streaming](docs/inference/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/inference/omnilingual-asr-inference.md) · [TTS](docs/inference/qwen3-tts-inference.md) · [VoxCPM2](docs/inference/voxcpm2-inference.md) · [VibeVoice](docs/inference/vibevoice-inference.md) · [Fish Audio S2 Pro](docs/inference/fish-audio-s2-pro.md) · [Magpie TTS](docs/inference/magpie-tts.md) · [Hibiki](docs/inference/hibiki-inference.md) · [MADLAD-400](docs/inference/madlad-translation.md) · [MAGNeT](docs/inference/magnet-music-gen.md) · [Stable Audio 3](docs/inference/stable-audio-3.md) · [FlashSR](docs/inference/flashsr.md) · [Forced Aligner](docs/inference/forced-aligner.md) · [Silero VAD](docs/inference/silero-vad.md) · [FireRedVAD](docs/inference/fireredvad.md) · [Wake-word](docs/inference/wake-word.md) · [Speaker Diarization](docs/inference/speaker-diarization.md) · [Speech Enhancement](docs/inference/speech-enhancement.md) · [Sidon](docs/inference/sidon.md) · [Cache/offline](docs/inference/cache-and-offline.md) - **Reference:** [Shared Protocols](docs/shared-protocols.md) diff --git a/README_ar.md b/README_ar.md index 4f4c1527..6fb171f8 100644 --- a/README_ar.md +++ b/README_ar.md @@ -70,6 +70,7 @@ - **[Hibiki Zero-3B](https://soniqo.audio/guides/audio-translate)** — ترجمة تدفقية من كلام إلى كلام (FR/ES/PT/DE → EN، MLX INT4 + INT8، حزمة Kyutai Moshi/Mimi، CC-BY-4.0) - **[PersonaPlex](https://soniqo.audio/ar/guides/respond)** — تحويل صوت إلى صوت ثنائي الاتجاه الكامل (7B، صوت داخل → صوت خارج، 18 إعداداً صوتياً مسبقاً) +- **[Audio2Face-3D](docs/models/audio2face3d.md)** — تحريك وجه الأفاتار انطلاقًا من الكلام (NVIDIA Audio2Face-3D v2.3 Mark, 301 معامل وجه, MLX) **التحسين والفصل وتوليد الصوت** @@ -182,7 +183,7 @@ struct DictateView: View { تشحن `SpeechUI` فقط `TranscriptionView` (النهائيات + الجزئيات) و `TranscriptionStore` (محول ASR تدفقي). استخدم AVFoundation لتصور الصوت وتشغيله. -منتجات SPM المتاحة: `Qwen3ASR`, `WhisperASR`, `Qwen3TTS`, `Qwen3TTSCoreML`, `ParakeetASR`, `ParakeetStreamingASR`, `NemotronStreamingASR`, `OmnilingualASR`, `KokoroTTS`, `SupertonicTTS`, `VibeVoiceTTS`, `CosyVoiceTTS`, `VoxCPM2TTS`, `ChatterboxTTS`, `OmniVoiceTTS`, `IndicMioTTS`, `FishAudioTTS`, `MagpieTTS`, `MagpieTTSCoreML`, `MAGNeTMusicGen`, `StableAudio3MusicGen`, `FlashSR`, `PersonaPlex`, `HibikiTranslate`, `MADLADTranslation`, `SpeechVAD`, `SpeechWakeWord`, `SpeechEnhancement`, `SpeechRestoration`, `SourceSeparation`, `Qwen3Chat`, `FunctionGemma`, `SpeechCore`, `SpeechUI`, `AudioCommon`. +منتجات SPM المتاحة: `Qwen3ASR`, `WhisperASR`, `Qwen3TTS`, `Qwen3TTSCoreML`, `ParakeetASR`, `ParakeetStreamingASR`, `NemotronStreamingASR`, `OmnilingualASR`, `KokoroTTS`, `SupertonicTTS`, `VibeVoiceTTS`, `CosyVoiceTTS`, `VoxCPM2TTS`, `ChatterboxTTS`, `OmniVoiceTTS`, `IndicMioTTS`, `FishAudioTTS`, `MagpieTTS`, `MagpieTTSCoreML`, `MAGNeTMusicGen`, `StableAudio3MusicGen`, `FlashSR`, `PersonaPlex`, `Audio2Face3D`, `HibikiTranslate`, `MADLADTranslation`, `SpeechVAD`, `SpeechWakeWord`, `SpeechEnhancement`, `SpeechRestoration`, `SourceSeparation`, `Qwen3Chat`, `FunctionGemma`, `SpeechCore`, `SpeechUI`, `AudioCommon`. @@ -223,6 +224,7 @@ struct DictateView: View { | [MADLAD-400](https://soniqo.audio/ar/guides/translate) | نص → نص (ترجمة) | MLX | 3B | **أكثر من 400** | | [Hibiki Zero-3B](https://soniqo.audio/guides/audio-translate) | كلام → كلام (ترجمة) | MLX | 3B | FR/ES/PT/DE → EN | | [PersonaPlex](https://soniqo.audio/ar/guides/respond) | كلام → كلام | MLX | 7B | EN | +| [Audio2Face-3D](docs/models/audio2face3d.md) | كلام → تحريك الوجه | MLX | v2.3 Mark | محايد للغة | | [Silero VAD](https://soniqo.audio/ar/guides/vad) | اكتشاف النشاط الصوتي | MLX, CoreML | 309K | محايد للغة | | [KWS Zipformer](docs/models/kws-zipformer.md) | Audio → Wake word | CoreML (ANE) | 3M | EN/custom keywords | | [Pyannote](https://soniqo.audio/ar/guides/diarize) | VAD + تمييز | MLX | 1.5M | محايد للغة | @@ -547,7 +549,7 @@ speech-swift مقسم إلى هدف SPM واحد لكل نموذج بحيث يد **[مخطط الهيكلة الكامل مع الواجهات الخلفية وجداول الذاكرة وخريطة الوحدات → soniqo.audio/architecture](https://soniqo.audio/ar/architecture)** · **[مرجع API → soniqo.audio/api](https://soniqo.audio/ar/api)** · **[الاختبارات → soniqo.audio/benchmarks](https://soniqo.audio/ar/benchmarks)** الوثائق المحلية (المستودع): -- **النماذج:** [Qwen3-ASR](docs/models/asr-model.md) · [WhisperASR](docs/models/whisper-asr.md) · [Qwen3-TTS](docs/models/tts-model.md) · [CosyVoice](docs/models/cosyvoice-tts.md) · [Kokoro](docs/models/kokoro-tts.md) · [VoxCPM2](docs/models/voxcpm2-tts.md) · [VibeVoice](docs/models/vibevoice.md) · [Supertonic](docs/models/supertonic-tts.md) · [Chatterbox](docs/models/chatterbox-tts.md) · [Indic-Mio](docs/models/indic-mio-tts.md) · [Fish Audio S2 Pro](docs/models/fish-audio-s2-pro.md) · [Magpie TTS](docs/models/magpie-tts.md) · [Parakeet TDT](docs/models/parakeet-asr.md) · [Parakeet Streaming](docs/models/parakeet-streaming-asr.md) · [Nemotron Streaming](docs/models/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/models/omnilingual-asr.md) · [PersonaPlex](docs/models/personaplex.md) · [Hibiki](docs/models/hibiki.md) · [MADLAD-400](docs/models/madlad-translation.md) · [FunctionGemma](docs/models/function-gemma.md) · [Qwen3.5 Chat](docs/models/qwen35-chat.md) · [Gemma 4 Chat](docs/models/gemma4-chat.md) · [Qwen3 Dense Chat](docs/models/qwen3-dense-chat.md) · [FireRedVAD](docs/models/fireredvad.md) · [KWS Zipformer](docs/models/kws-zipformer.md) · [Sidon](docs/models/sidon.md) · [Source Separation](docs/models/source-separation.md) · [HTDemucs](docs/models/htdemucs.md) · [MAGNeT](docs/models/magnet-music-gen.md) · [Stable Audio 3](docs/models/stable-audio-3.md) · [FlashSR](docs/models/flashsr.md) +- **النماذج:** [Qwen3-ASR](docs/models/asr-model.md) · [WhisperASR](docs/models/whisper-asr.md) · [Qwen3-TTS](docs/models/tts-model.md) · [CosyVoice](docs/models/cosyvoice-tts.md) · [Kokoro](docs/models/kokoro-tts.md) · [VoxCPM2](docs/models/voxcpm2-tts.md) · [VibeVoice](docs/models/vibevoice.md) · [Supertonic](docs/models/supertonic-tts.md) · [Chatterbox](docs/models/chatterbox-tts.md) · [Indic-Mio](docs/models/indic-mio-tts.md) · [Fish Audio S2 Pro](docs/models/fish-audio-s2-pro.md) · [Magpie TTS](docs/models/magpie-tts.md) · [Parakeet TDT](docs/models/parakeet-asr.md) · [Parakeet Streaming](docs/models/parakeet-streaming-asr.md) · [Nemotron Streaming](docs/models/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/models/omnilingual-asr.md) · [PersonaPlex](docs/models/personaplex.md) · [Hibiki](docs/models/hibiki.md) · [MADLAD-400](docs/models/madlad-translation.md) · [FunctionGemma](docs/models/function-gemma.md) · [Qwen3.5 Chat](docs/models/qwen35-chat.md) · [Gemma 4 Chat](docs/models/gemma4-chat.md) · [Qwen3 Dense Chat](docs/models/qwen3-dense-chat.md) · [FireRedVAD](docs/models/fireredvad.md) · [KWS Zipformer](docs/models/kws-zipformer.md) · [Sidon](docs/models/sidon.md) · [Source Separation](docs/models/source-separation.md) · [HTDemucs](docs/models/htdemucs.md) · [MAGNeT](docs/models/magnet-music-gen.md) · [Stable Audio 3](docs/models/stable-audio-3.md) · [FlashSR](docs/models/flashsr.md) · [Audio2Face-3D](docs/models/audio2face3d.md) - **الاستدلال:** [Qwen3-ASR](docs/inference/qwen3-asr-inference.md) · [WhisperASR](docs/inference/whisper-asr-inference.md) · [Parakeet TDT](docs/inference/parakeet-asr-inference.md) · [Parakeet Streaming](docs/inference/parakeet-streaming-asr-inference.md) · [Nemotron Streaming](docs/inference/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/inference/omnilingual-asr-inference.md) · [TTS](docs/inference/qwen3-tts-inference.md) · [VoxCPM2](docs/inference/voxcpm2-inference.md) · [VibeVoice](docs/inference/vibevoice-inference.md) · [Fish Audio S2 Pro](docs/inference/fish-audio-s2-pro.md) · [Magpie TTS](docs/inference/magpie-tts.md) · [Hibiki](docs/inference/hibiki-inference.md) · [MADLAD-400](docs/inference/madlad-translation.md) · [MAGNeT](docs/inference/magnet-music-gen.md) · [Stable Audio 3](docs/inference/stable-audio-3.md) · [FlashSR](docs/inference/flashsr.md) · [Forced Aligner](docs/inference/forced-aligner.md) · [Silero VAD](docs/inference/silero-vad.md) · [FireRedVAD](docs/inference/fireredvad.md) · [Wake-word](docs/inference/wake-word.md) · [Speaker Diarization](docs/inference/speaker-diarization.md) · [Speech Enhancement](docs/inference/speech-enhancement.md) · [Sidon](docs/inference/sidon.md) · [Cache/offline](docs/inference/cache-and-offline.md) - **المرجع:** [البروتوكولات المشتركة](docs/shared-protocols.md) diff --git a/README_de.md b/README_de.md index 856b77b7..eed93688 100644 --- a/README_de.md +++ b/README_de.md @@ -60,6 +60,7 @@ Spracherkennung, -synthese und -verständnis auf dem Gerät für Mac und iOS. L - **[Hibiki Zero-3B](https://soniqo.audio/guides/audio-translate)** — Streaming-Sprache-zu-Sprache-Übersetzung (FR/ES/PT/DE → EN, MLX INT4 + INT8, Kyutai Moshi/Mimi-Stack, CC-BY-4.0) - **[PersonaPlex](https://soniqo.audio/de/guides/respond)** — Vollduplex-Sprache-zu-Sprache (7B, Audio rein → Audio raus, 18 Stimmvoreinstellungen) +- **[Audio2Face-3D](docs/models/audio2face3d.md)** — Sprachgesteuerte Gesichtsanimation für Avatare (NVIDIA Audio2Face-3D v2.3 Mark, 301 Gesichtskoeffizienten, MLX) **Verbesserung, Trennung und Audiogenerierung** @@ -144,7 +145,7 @@ struct DictateView: View { `SpeechUI` liefert nur `TranscriptionView` (finale + partielle Ergebnisse) und `TranscriptionStore` (Streaming-ASR-Adapter). Verwende AVFoundation für Audio-Visualisierung und Wiedergabe. -Verfügbare SPM-Produkte: `Qwen3ASR`, `WhisperASR`, `Qwen3TTS`, `Qwen3TTSCoreML`, `ParakeetASR`, `ParakeetStreamingASR`, `NemotronStreamingASR`, `OmnilingualASR`, `KokoroTTS`, `SupertonicTTS`, `VibeVoiceTTS`, `CosyVoiceTTS`, `VoxCPM2TTS`, `ChatterboxTTS`, `OmniVoiceTTS`, `IndicMioTTS`, `FishAudioTTS`, `MagpieTTS`, `MagpieTTSCoreML`, `MAGNeTMusicGen`, `StableAudio3MusicGen`, `FlashSR`, `PersonaPlex`, `HibikiTranslate`, `MADLADTranslation`, `SpeechVAD`, `SpeechWakeWord`, `SpeechEnhancement`, `SpeechRestoration`, `SourceSeparation`, `Qwen3Chat`, `FunctionGemma`, `SpeechCore`, `SpeechUI`, `AudioCommon`. +Verfügbare SPM-Produkte: `Qwen3ASR`, `WhisperASR`, `Qwen3TTS`, `Qwen3TTSCoreML`, `ParakeetASR`, `ParakeetStreamingASR`, `NemotronStreamingASR`, `OmnilingualASR`, `KokoroTTS`, `SupertonicTTS`, `VibeVoiceTTS`, `CosyVoiceTTS`, `VoxCPM2TTS`, `ChatterboxTTS`, `OmniVoiceTTS`, `IndicMioTTS`, `FishAudioTTS`, `MagpieTTS`, `MagpieTTSCoreML`, `MAGNeTMusicGen`, `StableAudio3MusicGen`, `FlashSR`, `PersonaPlex`, `Audio2Face3D`, `HibikiTranslate`, `MADLADTranslation`, `SpeechVAD`, `SpeechWakeWord`, `SpeechEnhancement`, `SpeechRestoration`, `SourceSeparation`, `Qwen3Chat`, `FunctionGemma`, `SpeechCore`, `SpeechUI`, `AudioCommon`. ## Modelle @@ -179,6 +180,7 @@ Kompakte Übersicht unten. **[Vollständiger Modellkatalog mit Größen, Quantis | [MADLAD-400](https://soniqo.audio/de/guides/translate) | Text → Text (Übersetzung) | MLX | 3B | **400+** | | [Hibiki Zero-3B](https://soniqo.audio/guides/audio-translate) | Sprache → Sprache (Übersetzung) | MLX | 3B | FR/ES/PT/DE → EN | | [PersonaPlex](https://soniqo.audio/de/guides/respond) | Sprache → Sprache | MLX | 7B | EN | +| [Audio2Face-3D](docs/models/audio2face3d.md) | Sprache → Gesichtsanimation | MLX | v2.3 Mark | Sprachunabhängig | | [Silero VAD](https://soniqo.audio/de/guides/vad) | Sprachaktivitätserkennung | MLX, CoreML | 309K | Sprachunabhängig | | [KWS Zipformer](docs/models/kws-zipformer.md) | Audio → Wake word | CoreML (ANE) | 3M | EN/custom keywords | | [Pyannote](https://soniqo.audio/de/guides/diarize) | VAD + Diarisierung | MLX | 1.5M | Sprachunabhängig | @@ -455,7 +457,7 @@ speech-swift ist in ein SPM-Target pro Modell aufgeteilt, sodass Konsumenten nur **[Vollständiges Architekturdiagramm mit Backends, Speichertabellen und Modulkarte → soniqo.audio/architecture](https://soniqo.audio/de/architecture)** · **[API-Referenz → soniqo.audio/api](https://soniqo.audio/de/api)** · **[Benchmarks → soniqo.audio/benchmarks](https://soniqo.audio/de/benchmarks)** Lokale Docs (Repo): -- **Modelle:** [Qwen3-ASR](docs/models/asr-model.md) · [WhisperASR](docs/models/whisper-asr.md) · [Qwen3-TTS](docs/models/tts-model.md) · [CosyVoice](docs/models/cosyvoice-tts.md) · [Kokoro](docs/models/kokoro-tts.md) · [VoxCPM2](docs/models/voxcpm2-tts.md) · [VibeVoice](docs/models/vibevoice.md) · [Supertonic](docs/models/supertonic-tts.md) · [Chatterbox](docs/models/chatterbox-tts.md) · [Indic-Mio](docs/models/indic-mio-tts.md) · [Fish Audio S2 Pro](docs/models/fish-audio-s2-pro.md) · [Magpie TTS](docs/models/magpie-tts.md) · [Parakeet TDT](docs/models/parakeet-asr.md) · [Parakeet Streaming](docs/models/parakeet-streaming-asr.md) · [Nemotron Streaming](docs/models/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/models/omnilingual-asr.md) · [PersonaPlex](docs/models/personaplex.md) · [Hibiki](docs/models/hibiki.md) · [MADLAD-400](docs/models/madlad-translation.md) · [FunctionGemma](docs/models/function-gemma.md) · [Qwen3.5 Chat](docs/models/qwen35-chat.md) · [Gemma 4 Chat](docs/models/gemma4-chat.md) · [Qwen3 Dense Chat](docs/models/qwen3-dense-chat.md) · [FireRedVAD](docs/models/fireredvad.md) · [KWS Zipformer](docs/models/kws-zipformer.md) · [Sidon](docs/models/sidon.md) · [Source Separation](docs/models/source-separation.md) · [HTDemucs](docs/models/htdemucs.md) · [MAGNeT](docs/models/magnet-music-gen.md) · [Stable Audio 3](docs/models/stable-audio-3.md) · [FlashSR](docs/models/flashsr.md) +- **Modelle:** [Qwen3-ASR](docs/models/asr-model.md) · [WhisperASR](docs/models/whisper-asr.md) · [Qwen3-TTS](docs/models/tts-model.md) · [CosyVoice](docs/models/cosyvoice-tts.md) · [Kokoro](docs/models/kokoro-tts.md) · [VoxCPM2](docs/models/voxcpm2-tts.md) · [VibeVoice](docs/models/vibevoice.md) · [Supertonic](docs/models/supertonic-tts.md) · [Chatterbox](docs/models/chatterbox-tts.md) · [Indic-Mio](docs/models/indic-mio-tts.md) · [Fish Audio S2 Pro](docs/models/fish-audio-s2-pro.md) · [Magpie TTS](docs/models/magpie-tts.md) · [Parakeet TDT](docs/models/parakeet-asr.md) · [Parakeet Streaming](docs/models/parakeet-streaming-asr.md) · [Nemotron Streaming](docs/models/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/models/omnilingual-asr.md) · [PersonaPlex](docs/models/personaplex.md) · [Hibiki](docs/models/hibiki.md) · [MADLAD-400](docs/models/madlad-translation.md) · [FunctionGemma](docs/models/function-gemma.md) · [Qwen3.5 Chat](docs/models/qwen35-chat.md) · [Gemma 4 Chat](docs/models/gemma4-chat.md) · [Qwen3 Dense Chat](docs/models/qwen3-dense-chat.md) · [FireRedVAD](docs/models/fireredvad.md) · [KWS Zipformer](docs/models/kws-zipformer.md) · [Sidon](docs/models/sidon.md) · [Source Separation](docs/models/source-separation.md) · [HTDemucs](docs/models/htdemucs.md) · [MAGNeT](docs/models/magnet-music-gen.md) · [Stable Audio 3](docs/models/stable-audio-3.md) · [FlashSR](docs/models/flashsr.md) · [Audio2Face-3D](docs/models/audio2face3d.md) - **Inferenz:** [Qwen3-ASR](docs/inference/qwen3-asr-inference.md) · [WhisperASR](docs/inference/whisper-asr-inference.md) · [Parakeet TDT](docs/inference/parakeet-asr-inference.md) · [Parakeet Streaming](docs/inference/parakeet-streaming-asr-inference.md) · [Nemotron Streaming](docs/inference/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/inference/omnilingual-asr-inference.md) · [TTS](docs/inference/qwen3-tts-inference.md) · [VoxCPM2](docs/inference/voxcpm2-inference.md) · [VibeVoice](docs/inference/vibevoice-inference.md) · [Fish Audio S2 Pro](docs/inference/fish-audio-s2-pro.md) · [Magpie TTS](docs/inference/magpie-tts.md) · [Hibiki](docs/inference/hibiki-inference.md) · [MADLAD-400](docs/inference/madlad-translation.md) · [MAGNeT](docs/inference/magnet-music-gen.md) · [Stable Audio 3](docs/inference/stable-audio-3.md) · [FlashSR](docs/inference/flashsr.md) · [Forced Aligner](docs/inference/forced-aligner.md) · [Silero VAD](docs/inference/silero-vad.md) · [FireRedVAD](docs/inference/fireredvad.md) · [Wake-word](docs/inference/wake-word.md) · [Speaker Diarization](docs/inference/speaker-diarization.md) · [Speech Enhancement](docs/inference/speech-enhancement.md) · [Sidon](docs/inference/sidon.md) · [Cache/offline](docs/inference/cache-and-offline.md) - **Referenz:** [Geteilte Protokolle](docs/shared-protocols.md) diff --git a/README_es.md b/README_es.md index 1ab39769..fa067690 100644 --- a/README_es.md +++ b/README_es.md @@ -60,6 +60,7 @@ Reconocimiento, síntesis y comprensión de voz en el dispositivo para Mac e iOS - **[Hibiki Zero-3B](https://soniqo.audio/guides/audio-translate)** — Traducción voz a voz en streaming (FR/ES/PT/DE → EN, MLX INT4 + INT8, pila Kyutai Moshi/Mimi, CC-BY-4.0) - **[PersonaPlex](https://soniqo.audio/es/guides/respond)** — Voz a voz full-duplex (7B, audio de entrada → audio de salida, 18 presets de voz) +- **[Audio2Face-3D](docs/models/audio2face3d.md)** — Animación facial de avatares impulsada por voz (NVIDIA Audio2Face-3D v2.3 Mark, 301 coeficientes faciales, MLX) **Mejora, separación y generación de audio** @@ -144,7 +145,7 @@ struct DictateView: View { `SpeechUI` solo incluye `TranscriptionView` (finales + parciales) y `TranscriptionStore` (adaptador de ASR en streaming). Usa AVFoundation para la visualización y reproducción de audio. -Productos SPM disponibles: `Qwen3ASR`, `WhisperASR`, `Qwen3TTS`, `Qwen3TTSCoreML`, `ParakeetASR`, `ParakeetStreamingASR`, `NemotronStreamingASR`, `OmnilingualASR`, `KokoroTTS`, `SupertonicTTS`, `VibeVoiceTTS`, `CosyVoiceTTS`, `VoxCPM2TTS`, `ChatterboxTTS`, `OmniVoiceTTS`, `IndicMioTTS`, `FishAudioTTS`, `MagpieTTS`, `MagpieTTSCoreML`, `MAGNeTMusicGen`, `StableAudio3MusicGen`, `FlashSR`, `PersonaPlex`, `HibikiTranslate`, `MADLADTranslation`, `SpeechVAD`, `SpeechWakeWord`, `SpeechEnhancement`, `SpeechRestoration`, `SourceSeparation`, `Qwen3Chat`, `FunctionGemma`, `SpeechCore`, `SpeechUI`, `AudioCommon`. +Productos SPM disponibles: `Qwen3ASR`, `WhisperASR`, `Qwen3TTS`, `Qwen3TTSCoreML`, `ParakeetASR`, `ParakeetStreamingASR`, `NemotronStreamingASR`, `OmnilingualASR`, `KokoroTTS`, `SupertonicTTS`, `VibeVoiceTTS`, `CosyVoiceTTS`, `VoxCPM2TTS`, `ChatterboxTTS`, `OmniVoiceTTS`, `IndicMioTTS`, `FishAudioTTS`, `MagpieTTS`, `MagpieTTSCoreML`, `MAGNeTMusicGen`, `StableAudio3MusicGen`, `FlashSR`, `PersonaPlex`, `Audio2Face3D`, `HibikiTranslate`, `MADLADTranslation`, `SpeechVAD`, `SpeechWakeWord`, `SpeechEnhancement`, `SpeechRestoration`, `SourceSeparation`, `Qwen3Chat`, `FunctionGemma`, `SpeechCore`, `SpeechUI`, `AudioCommon`. ## Modelos @@ -179,6 +180,7 @@ Vista compacta a continuación. **[Catálogo completo de modelos con tamaños, c | [MADLAD-400](https://soniqo.audio/es/guides/translate) | Texto → Texto (Traducción) | MLX | 3B | **400+** | | [Hibiki Zero-3B](https://soniqo.audio/guides/audio-translate) | Voz → Voz (Traducción) | MLX | 3B | FR/ES/PT/DE → EN | | [PersonaPlex](https://soniqo.audio/es/guides/respond) | Voz → Voz | MLX | 7B | EN | +| [Audio2Face-3D](docs/models/audio2face3d.md) | Voz → Animación facial | MLX | v2.3 Mark | Agnóstico | | [Silero VAD](https://soniqo.audio/es/guides/vad) | Detección de actividad vocal | MLX, CoreML | 309K | Agnóstico | | [KWS Zipformer](docs/models/kws-zipformer.md) | Audio → Wake word | CoreML (ANE) | 3M | EN/custom keywords | | [Pyannote](https://soniqo.audio/es/guides/diarize) | VAD + Diarización | MLX | 1.5M | Agnóstico | @@ -455,7 +457,7 @@ speech-swift está dividido en un target SPM por modelo para que los consumidore **[Diagrama completo de arquitectura con backends, tablas de memoria y mapa de módulos → soniqo.audio/architecture](https://soniqo.audio/es/architecture)** · **[Referencia de API → soniqo.audio/api](https://soniqo.audio/es/api)** · **[Benchmarks → soniqo.audio/benchmarks](https://soniqo.audio/es/benchmarks)** Docs locales (repositorio): -- **Modelos:** [Qwen3-ASR](docs/models/asr-model.md) · [WhisperASR](docs/models/whisper-asr.md) · [Qwen3-TTS](docs/models/tts-model.md) · [CosyVoice](docs/models/cosyvoice-tts.md) · [Kokoro](docs/models/kokoro-tts.md) · [VoxCPM2](docs/models/voxcpm2-tts.md) · [VibeVoice](docs/models/vibevoice.md) · [Supertonic](docs/models/supertonic-tts.md) · [Chatterbox](docs/models/chatterbox-tts.md) · [Indic-Mio](docs/models/indic-mio-tts.md) · [Fish Audio S2 Pro](docs/models/fish-audio-s2-pro.md) · [Magpie TTS](docs/models/magpie-tts.md) · [Parakeet TDT](docs/models/parakeet-asr.md) · [Parakeet Streaming](docs/models/parakeet-streaming-asr.md) · [Nemotron Streaming](docs/models/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/models/omnilingual-asr.md) · [PersonaPlex](docs/models/personaplex.md) · [Hibiki](docs/models/hibiki.md) · [MADLAD-400](docs/models/madlad-translation.md) · [FunctionGemma](docs/models/function-gemma.md) · [Qwen3.5 Chat](docs/models/qwen35-chat.md) · [Gemma 4 Chat](docs/models/gemma4-chat.md) · [Qwen3 Dense Chat](docs/models/qwen3-dense-chat.md) · [FireRedVAD](docs/models/fireredvad.md) · [KWS Zipformer](docs/models/kws-zipformer.md) · [Sidon](docs/models/sidon.md) · [Source Separation](docs/models/source-separation.md) · [HTDemucs](docs/models/htdemucs.md) · [MAGNeT](docs/models/magnet-music-gen.md) · [Stable Audio 3](docs/models/stable-audio-3.md) · [FlashSR](docs/models/flashsr.md) +- **Modelos:** [Qwen3-ASR](docs/models/asr-model.md) · [WhisperASR](docs/models/whisper-asr.md) · [Qwen3-TTS](docs/models/tts-model.md) · [CosyVoice](docs/models/cosyvoice-tts.md) · [Kokoro](docs/models/kokoro-tts.md) · [VoxCPM2](docs/models/voxcpm2-tts.md) · [VibeVoice](docs/models/vibevoice.md) · [Supertonic](docs/models/supertonic-tts.md) · [Chatterbox](docs/models/chatterbox-tts.md) · [Indic-Mio](docs/models/indic-mio-tts.md) · [Fish Audio S2 Pro](docs/models/fish-audio-s2-pro.md) · [Magpie TTS](docs/models/magpie-tts.md) · [Parakeet TDT](docs/models/parakeet-asr.md) · [Parakeet Streaming](docs/models/parakeet-streaming-asr.md) · [Nemotron Streaming](docs/models/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/models/omnilingual-asr.md) · [PersonaPlex](docs/models/personaplex.md) · [Hibiki](docs/models/hibiki.md) · [MADLAD-400](docs/models/madlad-translation.md) · [FunctionGemma](docs/models/function-gemma.md) · [Qwen3.5 Chat](docs/models/qwen35-chat.md) · [Gemma 4 Chat](docs/models/gemma4-chat.md) · [Qwen3 Dense Chat](docs/models/qwen3-dense-chat.md) · [FireRedVAD](docs/models/fireredvad.md) · [KWS Zipformer](docs/models/kws-zipformer.md) · [Sidon](docs/models/sidon.md) · [Source Separation](docs/models/source-separation.md) · [HTDemucs](docs/models/htdemucs.md) · [MAGNeT](docs/models/magnet-music-gen.md) · [Stable Audio 3](docs/models/stable-audio-3.md) · [FlashSR](docs/models/flashsr.md) · [Audio2Face-3D](docs/models/audio2face3d.md) - **Inferencia:** [Qwen3-ASR](docs/inference/qwen3-asr-inference.md) · [WhisperASR](docs/inference/whisper-asr-inference.md) · [Parakeet TDT](docs/inference/parakeet-asr-inference.md) · [Parakeet Streaming](docs/inference/parakeet-streaming-asr-inference.md) · [Nemotron Streaming](docs/inference/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/inference/omnilingual-asr-inference.md) · [TTS](docs/inference/qwen3-tts-inference.md) · [VoxCPM2](docs/inference/voxcpm2-inference.md) · [VibeVoice](docs/inference/vibevoice-inference.md) · [Fish Audio S2 Pro](docs/inference/fish-audio-s2-pro.md) · [Magpie TTS](docs/inference/magpie-tts.md) · [Hibiki](docs/inference/hibiki-inference.md) · [MADLAD-400](docs/inference/madlad-translation.md) · [MAGNeT](docs/inference/magnet-music-gen.md) · [Stable Audio 3](docs/inference/stable-audio-3.md) · [FlashSR](docs/inference/flashsr.md) · [Forced Aligner](docs/inference/forced-aligner.md) · [Silero VAD](docs/inference/silero-vad.md) · [FireRedVAD](docs/inference/fireredvad.md) · [Wake-word](docs/inference/wake-word.md) · [Speaker Diarization](docs/inference/speaker-diarization.md) · [Speech Enhancement](docs/inference/speech-enhancement.md) · [Sidon](docs/inference/sidon.md) · [Cache/offline](docs/inference/cache-and-offline.md) - **Referencia:** [Protocolos compartidos](docs/shared-protocols.md) diff --git a/README_fr.md b/README_fr.md index 84bdafad..a125907d 100644 --- a/README_fr.md +++ b/README_fr.md @@ -60,6 +60,7 @@ Reconnaissance, synthese et comprehension vocale embarquees pour Mac et iOS. S'e - **[Hibiki Zero-3B](https://soniqo.audio/guides/audio-translate)** — Traduction parole-a-parole en streaming (FR/ES/PT/DE → EN, MLX INT4 + INT8, pile Kyutai Moshi/Mimi, CC-BY-4.0) - **[PersonaPlex](https://soniqo.audio/fr/guides/respond)** -- Parole-a-parole en full-duplex (7B, audio entrant → audio sortant, 18 preselections de voix) +- **[Audio2Face-3D](docs/models/audio2face3d.md)** — Animation faciale d'avatar pilotée par la voix (NVIDIA Audio2Face-3D v2.3 Mark, 301 coefficients faciaux, MLX) **Amélioration, séparation et génération audio** @@ -144,7 +145,7 @@ struct DictateView: View { `SpeechUI` ne fournit que `TranscriptionView` (finaux + partiels) et `TranscriptionStore` (adaptateur ASR en streaming). Utilisez AVFoundation pour la visualisation et la lecture audio. -Produits SPM disponibles : `Qwen3ASR`, `WhisperASR`, `Qwen3TTS`, `Qwen3TTSCoreML`, `ParakeetASR`, `ParakeetStreamingASR`, `NemotronStreamingASR`, `OmnilingualASR`, `KokoroTTS`, `SupertonicTTS`, `VibeVoiceTTS`, `CosyVoiceTTS`, `VoxCPM2TTS`, `ChatterboxTTS`, `OmniVoiceTTS`, `IndicMioTTS`, `FishAudioTTS`, `MagpieTTS`, `MagpieTTSCoreML`, `MAGNeTMusicGen`, `StableAudio3MusicGen`, `FlashSR`, `PersonaPlex`, `HibikiTranslate`, `MADLADTranslation`, `SpeechVAD`, `SpeechWakeWord`, `SpeechEnhancement`, `SpeechRestoration`, `SourceSeparation`, `Qwen3Chat`, `FunctionGemma`, `SpeechCore`, `SpeechUI`, `AudioCommon`. +Produits SPM disponibles : `Qwen3ASR`, `WhisperASR`, `Qwen3TTS`, `Qwen3TTSCoreML`, `ParakeetASR`, `ParakeetStreamingASR`, `NemotronStreamingASR`, `OmnilingualASR`, `KokoroTTS`, `SupertonicTTS`, `VibeVoiceTTS`, `CosyVoiceTTS`, `VoxCPM2TTS`, `ChatterboxTTS`, `OmniVoiceTTS`, `IndicMioTTS`, `FishAudioTTS`, `MagpieTTS`, `MagpieTTSCoreML`, `MAGNeTMusicGen`, `StableAudio3MusicGen`, `FlashSR`, `PersonaPlex`, `Audio2Face3D`, `HibikiTranslate`, `MADLADTranslation`, `SpeechVAD`, `SpeechWakeWord`, `SpeechEnhancement`, `SpeechRestoration`, `SourceSeparation`, `Qwen3Chat`, `FunctionGemma`, `SpeechCore`, `SpeechUI`, `AudioCommon`. ## Modeles @@ -179,6 +180,7 @@ Vue compacte ci-dessous. **[Catalogue complet des modeles avec tailles, quantifi | [MADLAD-400](https://soniqo.audio/fr/guides/translate) | Texte → Texte (Traduction) | MLX | 3B | **400+** | | [Hibiki Zero-3B](https://soniqo.audio/guides/audio-translate) | Parole → Parole (Traduction) | MLX | 3B | FR/ES/PT/DE → EN | | [PersonaPlex](https://soniqo.audio/fr/guides/respond) | Parole → Parole | MLX | 7B | EN | +| [Audio2Face-3D](docs/models/audio2face3d.md) | Parole → Animation faciale | MLX | v2.3 Mark | Agnostique | | [Silero VAD](https://soniqo.audio/fr/guides/vad) | Detection d'activite vocale | MLX, CoreML | 309K | Agnostique | | [KWS Zipformer](docs/models/kws-zipformer.md) | Audio → Wake word | CoreML (ANE) | 3M | EN/custom keywords | | [Pyannote](https://soniqo.audio/fr/guides/diarize) | VAD + Diarisation | MLX | 1.5M | Agnostique | @@ -455,7 +457,7 @@ speech-swift est decoupe en une cible SPM par modele, de sorte que les consommat **[Diagramme d'architecture complet avec backends, tableaux de memoire et carte des modules → soniqo.audio/architecture](https://soniqo.audio/fr/architecture)** · **[Reference d'API → soniqo.audio/api](https://soniqo.audio/fr/api)** · **[Benchmarks → soniqo.audio/benchmarks](https://soniqo.audio/fr/benchmarks)** Docs locales (depot) : -- **Modeles :** [Qwen3-ASR](docs/models/asr-model.md) · [WhisperASR](docs/models/whisper-asr.md) · [Qwen3-TTS](docs/models/tts-model.md) · [CosyVoice](docs/models/cosyvoice-tts.md) · [Kokoro](docs/models/kokoro-tts.md) · [VoxCPM2](docs/models/voxcpm2-tts.md) · [VibeVoice](docs/models/vibevoice.md) · [Supertonic](docs/models/supertonic-tts.md) · [Chatterbox](docs/models/chatterbox-tts.md) · [Indic-Mio](docs/models/indic-mio-tts.md) · [Fish Audio S2 Pro](docs/models/fish-audio-s2-pro.md) · [Magpie TTS](docs/models/magpie-tts.md) · [Parakeet TDT](docs/models/parakeet-asr.md) · [Parakeet Streaming](docs/models/parakeet-streaming-asr.md) · [Nemotron Streaming](docs/models/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/models/omnilingual-asr.md) · [PersonaPlex](docs/models/personaplex.md) · [Hibiki](docs/models/hibiki.md) · [MADLAD-400](docs/models/madlad-translation.md) · [FunctionGemma](docs/models/function-gemma.md) · [Qwen3.5 Chat](docs/models/qwen35-chat.md) · [Gemma 4 Chat](docs/models/gemma4-chat.md) · [Qwen3 Dense Chat](docs/models/qwen3-dense-chat.md) · [FireRedVAD](docs/models/fireredvad.md) · [KWS Zipformer](docs/models/kws-zipformer.md) · [Sidon](docs/models/sidon.md) · [Source Separation](docs/models/source-separation.md) · [HTDemucs](docs/models/htdemucs.md) · [MAGNeT](docs/models/magnet-music-gen.md) · [Stable Audio 3](docs/models/stable-audio-3.md) · [FlashSR](docs/models/flashsr.md) +- **Modeles :** [Qwen3-ASR](docs/models/asr-model.md) · [WhisperASR](docs/models/whisper-asr.md) · [Qwen3-TTS](docs/models/tts-model.md) · [CosyVoice](docs/models/cosyvoice-tts.md) · [Kokoro](docs/models/kokoro-tts.md) · [VoxCPM2](docs/models/voxcpm2-tts.md) · [VibeVoice](docs/models/vibevoice.md) · [Supertonic](docs/models/supertonic-tts.md) · [Chatterbox](docs/models/chatterbox-tts.md) · [Indic-Mio](docs/models/indic-mio-tts.md) · [Fish Audio S2 Pro](docs/models/fish-audio-s2-pro.md) · [Magpie TTS](docs/models/magpie-tts.md) · [Parakeet TDT](docs/models/parakeet-asr.md) · [Parakeet Streaming](docs/models/parakeet-streaming-asr.md) · [Nemotron Streaming](docs/models/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/models/omnilingual-asr.md) · [PersonaPlex](docs/models/personaplex.md) · [Hibiki](docs/models/hibiki.md) · [MADLAD-400](docs/models/madlad-translation.md) · [FunctionGemma](docs/models/function-gemma.md) · [Qwen3.5 Chat](docs/models/qwen35-chat.md) · [Gemma 4 Chat](docs/models/gemma4-chat.md) · [Qwen3 Dense Chat](docs/models/qwen3-dense-chat.md) · [FireRedVAD](docs/models/fireredvad.md) · [KWS Zipformer](docs/models/kws-zipformer.md) · [Sidon](docs/models/sidon.md) · [Source Separation](docs/models/source-separation.md) · [HTDemucs](docs/models/htdemucs.md) · [MAGNeT](docs/models/magnet-music-gen.md) · [Stable Audio 3](docs/models/stable-audio-3.md) · [FlashSR](docs/models/flashsr.md) · [Audio2Face-3D](docs/models/audio2face3d.md) - **Inference :** [Qwen3-ASR](docs/inference/qwen3-asr-inference.md) · [WhisperASR](docs/inference/whisper-asr-inference.md) · [Parakeet TDT](docs/inference/parakeet-asr-inference.md) · [Parakeet Streaming](docs/inference/parakeet-streaming-asr-inference.md) · [Nemotron Streaming](docs/inference/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/inference/omnilingual-asr-inference.md) · [TTS](docs/inference/qwen3-tts-inference.md) · [VoxCPM2](docs/inference/voxcpm2-inference.md) · [VibeVoice](docs/inference/vibevoice-inference.md) · [Fish Audio S2 Pro](docs/inference/fish-audio-s2-pro.md) · [Magpie TTS](docs/inference/magpie-tts.md) · [Hibiki](docs/inference/hibiki-inference.md) · [MADLAD-400](docs/inference/madlad-translation.md) · [MAGNeT](docs/inference/magnet-music-gen.md) · [Stable Audio 3](docs/inference/stable-audio-3.md) · [FlashSR](docs/inference/flashsr.md) · [Forced Aligner](docs/inference/forced-aligner.md) · [Silero VAD](docs/inference/silero-vad.md) · [FireRedVAD](docs/inference/fireredvad.md) · [Wake-word](docs/inference/wake-word.md) · [Speaker Diarization](docs/inference/speaker-diarization.md) · [Speech Enhancement](docs/inference/speech-enhancement.md) · [Sidon](docs/inference/sidon.md) · [Cache/offline](docs/inference/cache-and-offline.md) - **Reference :** [Protocoles partages](docs/shared-protocols.md) diff --git a/README_hi.md b/README_hi.md index 30828bd0..1cff61f9 100644 --- a/README_hi.md +++ b/README_hi.md @@ -60,6 +60,7 @@ Mac और iOS के लिए ऑन-डिवाइस स्पीच रि - **[Hibiki Zero-3B](https://soniqo.audio/guides/audio-translate)** — स्ट्रीमिंग स्पीच-टू-स्पीच अनुवाद (FR/ES/PT/DE → EN, MLX INT4 + INT8, Kyutai Moshi/Mimi स्टैक, CC-BY-4.0) - **[PersonaPlex](https://soniqo.audio/hi/guides/respond)** — फुल-डुप्लेक्स स्पीच-टू-स्पीच (7B, ऑडियो इन → ऑडियो आउट, 18 वॉयस प्रीसेट) +- **[Audio2Face-3D](docs/models/audio2face3d.md)** — वाक्-चालित अवतार चेहरा एनीमेशन (NVIDIA Audio2Face-3D v2.3 Mark, 301 फेशियल कोएफ़िशिएंट, MLX) **एन्हांसमेंट, सेपरेशन और ऑडियो जनरेशन** @@ -144,7 +145,7 @@ struct DictateView: View { `SpeechUI` केवल `TranscriptionView` (finals + partials) और `TranscriptionStore` (स्ट्रीमिंग ASR एडाप्टर) प्रदान करता है। ऑडियो विज़ुअलाइज़ेशन और प्लेबैक के लिए AVFoundation का उपयोग करें। -उपलब्ध SPM उत्पाद: `Qwen3ASR`, `WhisperASR`, `Qwen3TTS`, `Qwen3TTSCoreML`, `ParakeetASR`, `ParakeetStreamingASR`, `NemotronStreamingASR`, `OmnilingualASR`, `KokoroTTS`, `SupertonicTTS`, `VibeVoiceTTS`, `CosyVoiceTTS`, `VoxCPM2TTS`, `ChatterboxTTS`, `OmniVoiceTTS`, `IndicMioTTS`, `FishAudioTTS`, `MagpieTTS`, `MagpieTTSCoreML`, `MAGNeTMusicGen`, `StableAudio3MusicGen`, `FlashSR`, `PersonaPlex`, `HibikiTranslate`, `MADLADTranslation`, `SpeechVAD`, `SpeechWakeWord`, `SpeechEnhancement`, `SpeechRestoration`, `SourceSeparation`, `Qwen3Chat`, `FunctionGemma`, `SpeechCore`, `SpeechUI`, `AudioCommon`. +उपलब्ध SPM उत्पाद: `Qwen3ASR`, `WhisperASR`, `Qwen3TTS`, `Qwen3TTSCoreML`, `ParakeetASR`, `ParakeetStreamingASR`, `NemotronStreamingASR`, `OmnilingualASR`, `KokoroTTS`, `SupertonicTTS`, `VibeVoiceTTS`, `CosyVoiceTTS`, `VoxCPM2TTS`, `ChatterboxTTS`, `OmniVoiceTTS`, `IndicMioTTS`, `FishAudioTTS`, `MagpieTTS`, `MagpieTTSCoreML`, `MAGNeTMusicGen`, `StableAudio3MusicGen`, `FlashSR`, `PersonaPlex`, `Audio2Face3D`, `HibikiTranslate`, `MADLADTranslation`, `SpeechVAD`, `SpeechWakeWord`, `SpeechEnhancement`, `SpeechRestoration`, `SourceSeparation`, `Qwen3Chat`, `FunctionGemma`, `SpeechCore`, `SpeechUI`, `AudioCommon`. ## मॉडल @@ -179,6 +180,7 @@ struct DictateView: View { | [MADLAD-400](https://soniqo.audio/hi/guides/translate) | टेक्स्ट → टेक्स्ट (अनुवाद) | MLX | 3B | **400+** | | [Hibiki Zero-3B](https://soniqo.audio/guides/audio-translate) | स्पीच → स्पीच (अनुवाद) | MLX | 3B | FR/ES/PT/DE → EN | | [PersonaPlex](https://soniqo.audio/hi/guides/respond) | स्पीच → स्पीच | MLX | 7B | EN | +| [Audio2Face-3D](docs/models/audio2face3d.md) | स्पीच → चेहरा एनीमेशन | MLX | v2.3 Mark | भाषा-तटस्थ | | [Silero VAD](https://soniqo.audio/hi/guides/vad) | वॉयस एक्टिविटी डिटेक्शन | MLX, CoreML | 309K | भाषा-तटस्थ | | [KWS Zipformer](docs/models/kws-zipformer.md) | Audio → Wake word | CoreML (ANE) | 3M | EN/custom keywords | | [Pyannote](https://soniqo.audio/hi/guides/diarize) | VAD + Diarization | MLX | 1.5M | भाषा-तटस्थ | @@ -455,7 +457,7 @@ speech-swift प्रति मॉडल एक SPM टारगेट मे **[बैकएंड, मेमोरी टेबल्स, और मॉड्यूल मैप के साथ पूर्ण आर्किटेक्चर डायग्राम → soniqo.audio/architecture](https://soniqo.audio/hi/architecture)** · **[API संदर्भ → soniqo.audio/api](https://soniqo.audio/hi/api)** · **[बेंचमार्क → soniqo.audio/benchmarks](https://soniqo.audio/hi/benchmarks)** स्थानीय डॉक्स (रिपॉज़िटरी): -- **मॉडल:** [Qwen3-ASR](docs/models/asr-model.md) · [WhisperASR](docs/models/whisper-asr.md) · [Qwen3-TTS](docs/models/tts-model.md) · [CosyVoice](docs/models/cosyvoice-tts.md) · [Kokoro](docs/models/kokoro-tts.md) · [VoxCPM2](docs/models/voxcpm2-tts.md) · [VibeVoice](docs/models/vibevoice.md) · [Supertonic](docs/models/supertonic-tts.md) · [Chatterbox](docs/models/chatterbox-tts.md) · [Indic-Mio](docs/models/indic-mio-tts.md) · [Fish Audio S2 Pro](docs/models/fish-audio-s2-pro.md) · [Magpie TTS](docs/models/magpie-tts.md) · [Parakeet TDT](docs/models/parakeet-asr.md) · [Parakeet Streaming](docs/models/parakeet-streaming-asr.md) · [Nemotron Streaming](docs/models/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/models/omnilingual-asr.md) · [PersonaPlex](docs/models/personaplex.md) · [Hibiki](docs/models/hibiki.md) · [MADLAD-400](docs/models/madlad-translation.md) · [FunctionGemma](docs/models/function-gemma.md) · [Qwen3.5 Chat](docs/models/qwen35-chat.md) · [Gemma 4 Chat](docs/models/gemma4-chat.md) · [Qwen3 Dense Chat](docs/models/qwen3-dense-chat.md) · [FireRedVAD](docs/models/fireredvad.md) · [KWS Zipformer](docs/models/kws-zipformer.md) · [Sidon](docs/models/sidon.md) · [Source Separation](docs/models/source-separation.md) · [HTDemucs](docs/models/htdemucs.md) · [MAGNeT](docs/models/magnet-music-gen.md) · [Stable Audio 3](docs/models/stable-audio-3.md) · [FlashSR](docs/models/flashsr.md) +- **मॉडल:** [Qwen3-ASR](docs/models/asr-model.md) · [WhisperASR](docs/models/whisper-asr.md) · [Qwen3-TTS](docs/models/tts-model.md) · [CosyVoice](docs/models/cosyvoice-tts.md) · [Kokoro](docs/models/kokoro-tts.md) · [VoxCPM2](docs/models/voxcpm2-tts.md) · [VibeVoice](docs/models/vibevoice.md) · [Supertonic](docs/models/supertonic-tts.md) · [Chatterbox](docs/models/chatterbox-tts.md) · [Indic-Mio](docs/models/indic-mio-tts.md) · [Fish Audio S2 Pro](docs/models/fish-audio-s2-pro.md) · [Magpie TTS](docs/models/magpie-tts.md) · [Parakeet TDT](docs/models/parakeet-asr.md) · [Parakeet Streaming](docs/models/parakeet-streaming-asr.md) · [Nemotron Streaming](docs/models/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/models/omnilingual-asr.md) · [PersonaPlex](docs/models/personaplex.md) · [Hibiki](docs/models/hibiki.md) · [MADLAD-400](docs/models/madlad-translation.md) · [FunctionGemma](docs/models/function-gemma.md) · [Qwen3.5 Chat](docs/models/qwen35-chat.md) · [Gemma 4 Chat](docs/models/gemma4-chat.md) · [Qwen3 Dense Chat](docs/models/qwen3-dense-chat.md) · [FireRedVAD](docs/models/fireredvad.md) · [KWS Zipformer](docs/models/kws-zipformer.md) · [Sidon](docs/models/sidon.md) · [Source Separation](docs/models/source-separation.md) · [HTDemucs](docs/models/htdemucs.md) · [MAGNeT](docs/models/magnet-music-gen.md) · [Stable Audio 3](docs/models/stable-audio-3.md) · [FlashSR](docs/models/flashsr.md) · [Audio2Face-3D](docs/models/audio2face3d.md) - **इन्फ़रेंस:** [Qwen3-ASR](docs/inference/qwen3-asr-inference.md) · [WhisperASR](docs/inference/whisper-asr-inference.md) · [Parakeet TDT](docs/inference/parakeet-asr-inference.md) · [Parakeet Streaming](docs/inference/parakeet-streaming-asr-inference.md) · [Nemotron Streaming](docs/inference/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/inference/omnilingual-asr-inference.md) · [TTS](docs/inference/qwen3-tts-inference.md) · [VoxCPM2](docs/inference/voxcpm2-inference.md) · [VibeVoice](docs/inference/vibevoice-inference.md) · [Fish Audio S2 Pro](docs/inference/fish-audio-s2-pro.md) · [Magpie TTS](docs/inference/magpie-tts.md) · [Hibiki](docs/inference/hibiki-inference.md) · [MADLAD-400](docs/inference/madlad-translation.md) · [MAGNeT](docs/inference/magnet-music-gen.md) · [Stable Audio 3](docs/inference/stable-audio-3.md) · [FlashSR](docs/inference/flashsr.md) · [Forced Aligner](docs/inference/forced-aligner.md) · [Silero VAD](docs/inference/silero-vad.md) · [FireRedVAD](docs/inference/fireredvad.md) · [Wake-word](docs/inference/wake-word.md) · [Speaker Diarization](docs/inference/speaker-diarization.md) · [Speech Enhancement](docs/inference/speech-enhancement.md) · [Sidon](docs/inference/sidon.md) · [Cache/offline](docs/inference/cache-and-offline.md) - **संदर्भ:** [शेयर्ड प्रोटोकॉल](docs/shared-protocols.md) diff --git a/README_ja.md b/README_ja.md index 5d19a360..a17fa56a 100644 --- a/README_ja.md +++ b/README_ja.md @@ -60,6 +60,7 @@ Mac・iOS向けのオンデバイス音声認識・合成・理解。Apple Silic - **[Hibiki Zero-3B](https://soniqo.audio/guides/audio-translate)** — ストリーミング音声間翻訳（FR/ES/PT/DE → EN、MLX INT4 + INT8、Kyutai Moshi/Mimi スタック、CC-BY-4.0） - **[PersonaPlex](https://soniqo.audio/ja/guides/respond)** — 全二重音声間会話（7B、音声入力 → 音声出力、18種類のボイスプリセット） +- **[Audio2Face-3D](docs/models/audio2face3d.md)** — 音声駆動のアバター表情アニメーション（NVIDIA Audio2Face-3D v2.3 Mark、顔係数 301 次元、MLX） **強化、分離、オーディオ生成** @@ -144,7 +145,7 @@ struct DictateView: View { `SpeechUI` には `TranscriptionView`（確定 + 部分）と `TranscriptionStore`（ストリーミングASRアダプター）のみが含まれます。音声の可視化や再生には AVFoundation をお使いください。 -利用可能なSPMプロダクト：`Qwen3ASR`, `WhisperASR`, `Qwen3TTS`, `Qwen3TTSCoreML`, `ParakeetASR`, `ParakeetStreamingASR`, `NemotronStreamingASR`, `OmnilingualASR`, `KokoroTTS`, `SupertonicTTS`, `VibeVoiceTTS`, `CosyVoiceTTS`, `VoxCPM2TTS`, `ChatterboxTTS`, `OmniVoiceTTS`, `IndicMioTTS`, `FishAudioTTS`, `MagpieTTS`, `MagpieTTSCoreML`, `MAGNeTMusicGen`, `StableAudio3MusicGen`, `FlashSR`, `PersonaPlex`, `HibikiTranslate`, `MADLADTranslation`, `SpeechVAD`, `SpeechWakeWord`, `SpeechEnhancement`, `SpeechRestoration`, `SourceSeparation`, `Qwen3Chat`, `FunctionGemma`, `SpeechCore`, `SpeechUI`, `AudioCommon`。 +利用可能なSPMプロダクト：`Qwen3ASR`, `WhisperASR`, `Qwen3TTS`, `Qwen3TTSCoreML`, `ParakeetASR`, `ParakeetStreamingASR`, `NemotronStreamingASR`, `OmnilingualASR`, `KokoroTTS`, `SupertonicTTS`, `VibeVoiceTTS`, `CosyVoiceTTS`, `VoxCPM2TTS`, `ChatterboxTTS`, `OmniVoiceTTS`, `IndicMioTTS`, `FishAudioTTS`, `MagpieTTS`, `MagpieTTSCoreML`, `MAGNeTMusicGen`, `StableAudio3MusicGen`, `FlashSR`, `PersonaPlex`, `Audio2Face3D`, `HibikiTranslate`, `MADLADTranslation`, `SpeechVAD`, `SpeechWakeWord`, `SpeechEnhancement`, `SpeechRestoration`, `SourceSeparation`, `Qwen3Chat`, `FunctionGemma`, `SpeechCore`, `SpeechUI`, `AudioCommon`。 ## モデル @@ -179,6 +180,7 @@ struct DictateView: View { | [MADLAD-400](https://soniqo.audio/ja/guides/translate) | テキスト → テキスト（翻訳） | MLX | 3B | **400+** | | [Hibiki Zero-3B](https://soniqo.audio/guides/audio-translate) | 音声 → 音声（翻訳） | MLX | 3B | FR/ES/PT/DE → EN | | [PersonaPlex](https://soniqo.audio/ja/guides/respond) | 音声 → 音声 | MLX | 7B | EN | +| [Audio2Face-3D](docs/models/audio2face3d.md) | 音声 → 表情アニメーション | MLX | v2.3 Mark | 言語非依存 | | [Silero VAD](https://soniqo.audio/ja/guides/vad) | 音声区間検出 | MLX、CoreML | 309K | 言語非依存 | | [KWS Zipformer](docs/models/kws-zipformer.md) | Audio → Wake word | CoreML (ANE) | 3M | EN/custom keywords | | [Pyannote](https://soniqo.audio/ja/guides/diarize) | VAD + ダイアライゼーション | MLX | 1.5M | 言語非依存 | @@ -455,7 +457,7 @@ speech-swift はモデルごとに1つのSPMターゲットに分割されてお **[バックエンド、メモリ表、モジュールマップ付きの完全なアーキテクチャ図 → soniqo.audio/architecture](https://soniqo.audio/ja/architecture)** · **[APIリファレンス → soniqo.audio/api](https://soniqo.audio/ja/api)** · **[ベンチマーク → soniqo.audio/benchmarks](https://soniqo.audio/ja/benchmarks)** ローカルドキュメント（リポジトリ内）： -- **モデル：** [Qwen3-ASR](docs/models/asr-model.md) · [WhisperASR](docs/models/whisper-asr.md) · [Qwen3-TTS](docs/models/tts-model.md) · [CosyVoice](docs/models/cosyvoice-tts.md) · [Kokoro](docs/models/kokoro-tts.md) · [VoxCPM2](docs/models/voxcpm2-tts.md) · [VibeVoice](docs/models/vibevoice.md) · [Supertonic](docs/models/supertonic-tts.md) · [Chatterbox](docs/models/chatterbox-tts.md) · [Indic-Mio](docs/models/indic-mio-tts.md) · [Fish Audio S2 Pro](docs/models/fish-audio-s2-pro.md) · [Magpie TTS](docs/models/magpie-tts.md) · [Parakeet TDT](docs/models/parakeet-asr.md) · [Parakeet Streaming](docs/models/parakeet-streaming-asr.md) · [Nemotron Streaming](docs/models/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/models/omnilingual-asr.md) · [PersonaPlex](docs/models/personaplex.md) · [Hibiki](docs/models/hibiki.md) · [MADLAD-400](docs/models/madlad-translation.md) · [FunctionGemma](docs/models/function-gemma.md) · [Qwen3.5 Chat](docs/models/qwen35-chat.md) · [Gemma 4 Chat](docs/models/gemma4-chat.md) · [Qwen3 Dense Chat](docs/models/qwen3-dense-chat.md) · [FireRedVAD](docs/models/fireredvad.md) · [KWS Zipformer](docs/models/kws-zipformer.md) · [Sidon](docs/models/sidon.md) · [Source Separation](docs/models/source-separation.md) · [HTDemucs](docs/models/htdemucs.md) · [MAGNeT](docs/models/magnet-music-gen.md) · [Stable Audio 3](docs/models/stable-audio-3.md) · [FlashSR](docs/models/flashsr.md) +- **モデル：** [Qwen3-ASR](docs/models/asr-model.md) · [WhisperASR](docs/models/whisper-asr.md) · [Qwen3-TTS](docs/models/tts-model.md) · [CosyVoice](docs/models/cosyvoice-tts.md) · [Kokoro](docs/models/kokoro-tts.md) · [VoxCPM2](docs/models/voxcpm2-tts.md) · [VibeVoice](docs/models/vibevoice.md) · [Supertonic](docs/models/supertonic-tts.md) · [Chatterbox](docs/models/chatterbox-tts.md) · [Indic-Mio](docs/models/indic-mio-tts.md) · [Fish Audio S2 Pro](docs/models/fish-audio-s2-pro.md) · [Magpie TTS](docs/models/magpie-tts.md) · [Parakeet TDT](docs/models/parakeet-asr.md) · [Parakeet Streaming](docs/models/parakeet-streaming-asr.md) · [Nemotron Streaming](docs/models/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/models/omnilingual-asr.md) · [PersonaPlex](docs/models/personaplex.md) · [Hibiki](docs/models/hibiki.md) · [MADLAD-400](docs/models/madlad-translation.md) · [FunctionGemma](docs/models/function-gemma.md) · [Qwen3.5 Chat](docs/models/qwen35-chat.md) · [Gemma 4 Chat](docs/models/gemma4-chat.md) · [Qwen3 Dense Chat](docs/models/qwen3-dense-chat.md) · [FireRedVAD](docs/models/fireredvad.md) · [KWS Zipformer](docs/models/kws-zipformer.md) · [Sidon](docs/models/sidon.md) · [Source Separation](docs/models/source-separation.md) · [HTDemucs](docs/models/htdemucs.md) · [MAGNeT](docs/models/magnet-music-gen.md) · [Stable Audio 3](docs/models/stable-audio-3.md) · [FlashSR](docs/models/flashsr.md) · [Audio2Face-3D](docs/models/audio2face3d.md) - **推論：** [Qwen3-ASR](docs/inference/qwen3-asr-inference.md) · [WhisperASR](docs/inference/whisper-asr-inference.md) · [Parakeet TDT](docs/inference/parakeet-asr-inference.md) · [Parakeet Streaming](docs/inference/parakeet-streaming-asr-inference.md) · [Nemotron Streaming](docs/inference/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/inference/omnilingual-asr-inference.md) · [TTS](docs/inference/qwen3-tts-inference.md) · [VoxCPM2](docs/inference/voxcpm2-inference.md) · [VibeVoice](docs/inference/vibevoice-inference.md) · [Fish Audio S2 Pro](docs/inference/fish-audio-s2-pro.md) · [Magpie TTS](docs/inference/magpie-tts.md) · [Hibiki](docs/inference/hibiki-inference.md) · [MADLAD-400](docs/inference/madlad-translation.md) · [MAGNeT](docs/inference/magnet-music-gen.md) · [Stable Audio 3](docs/inference/stable-audio-3.md) · [FlashSR](docs/inference/flashsr.md) · [Forced Aligner](docs/inference/forced-aligner.md) · [Silero VAD](docs/inference/silero-vad.md) · [FireRedVAD](docs/inference/fireredvad.md) · [Wake-word](docs/inference/wake-word.md) · [Speaker Diarization](docs/inference/speaker-diarization.md) · [Speech Enhancement](docs/inference/speech-enhancement.md) · [Sidon](docs/inference/sidon.md) · [Cache/offline](docs/inference/cache-and-offline.md) - **リファレンス：** [共有プロトコル](docs/shared-protocols.md) diff --git a/README_ko.md b/README_ko.md index 7a80838a..32bda02a 100644 --- a/README_ko.md +++ b/README_ko.md @@ -60,6 +60,7 @@ Mac과 iOS를 위한 온디바이스 음성 인식, 합성 및 이해. Apple Sil - **[Hibiki Zero-3B](https://soniqo.audio/guides/audio-translate)** — 스트리밍 음성-음성 번역 (FR/ES/PT/DE → EN, MLX INT4 + INT8, Kyutai Moshi/Mimi 스택, CC-BY-4.0) - **[PersonaPlex](https://soniqo.audio/ko/guides/respond)** — 전이중 음성-음성 대화 (7B, 오디오 입력 → 오디오 출력, 18개 음색 프리셋) +- **[Audio2Face-3D](docs/models/audio2face3d.md)** — 음성 기반 아바타 얼굴 애니메이션 (NVIDIA Audio2Face-3D v2.3 Mark, 얼굴 계수 301개, MLX) **향상, 분리 및 오디오 생성** @@ -144,7 +145,7 @@ struct DictateView: View { `SpeechUI`에는 `TranscriptionView`(파이널 + 파셜)와 `TranscriptionStore`(스트리밍 ASR 어댑터)만 포함됩니다. 오디오 시각화와 재생에는 AVFoundation을 사용하세요. -사용 가능한 SPM 프로덕트: `Qwen3ASR`, `WhisperASR`, `Qwen3TTS`, `Qwen3TTSCoreML`, `ParakeetASR`, `ParakeetStreamingASR`, `NemotronStreamingASR`, `OmnilingualASR`, `KokoroTTS`, `SupertonicTTS`, `VibeVoiceTTS`, `CosyVoiceTTS`, `VoxCPM2TTS`, `ChatterboxTTS`, `OmniVoiceTTS`, `IndicMioTTS`, `FishAudioTTS`, `MagpieTTS`, `MagpieTTSCoreML`, `MAGNeTMusicGen`, `StableAudio3MusicGen`, `FlashSR`, `PersonaPlex`, `HibikiTranslate`, `MADLADTranslation`, `SpeechVAD`, `SpeechWakeWord`, `SpeechEnhancement`, `SpeechRestoration`, `SourceSeparation`, `Qwen3Chat`, `FunctionGemma`, `SpeechCore`, `SpeechUI`, `AudioCommon`. +사용 가능한 SPM 프로덕트: `Qwen3ASR`, `WhisperASR`, `Qwen3TTS`, `Qwen3TTSCoreML`, `ParakeetASR`, `ParakeetStreamingASR`, `NemotronStreamingASR`, `OmnilingualASR`, `KokoroTTS`, `SupertonicTTS`, `VibeVoiceTTS`, `CosyVoiceTTS`, `VoxCPM2TTS`, `ChatterboxTTS`, `OmniVoiceTTS`, `IndicMioTTS`, `FishAudioTTS`, `MagpieTTS`, `MagpieTTSCoreML`, `MAGNeTMusicGen`, `StableAudio3MusicGen`, `FlashSR`, `PersonaPlex`, `Audio2Face3D`, `HibikiTranslate`, `MADLADTranslation`, `SpeechVAD`, `SpeechWakeWord`, `SpeechEnhancement`, `SpeechRestoration`, `SourceSeparation`, `Qwen3Chat`, `FunctionGemma`, `SpeechCore`, `SpeechUI`, `AudioCommon`. ## 모델 @@ -179,6 +180,7 @@ struct DictateView: View { | [MADLAD-400](https://soniqo.audio/ko/guides/translate) | 텍스트 → 텍스트 (번역) | MLX | 3B | **400+** | | [Hibiki Zero-3B](https://soniqo.audio/guides/audio-translate) | 음성 → 음성 (번역) | MLX | 3B | FR/ES/PT/DE → EN | | [PersonaPlex](https://soniqo.audio/ko/guides/respond) | 음성 → 음성 | MLX | 7B | EN | +| [Audio2Face-3D](docs/models/audio2face3d.md) | 음성 → 얼굴 애니메이션 | MLX | v2.3 Mark | 언어 무관 | | [Silero VAD](https://soniqo.audio/ko/guides/vad) | 음성 활동 감지 | MLX, CoreML | 309K | 언어 무관 | | [KWS Zipformer](docs/models/kws-zipformer.md) | Audio → Wake word | CoreML (ANE) | 3M | EN/custom keywords | | [Pyannote](https://soniqo.audio/ko/guides/diarize) | VAD + 화자 분리 | MLX | 1.5M | 언어 무관 | @@ -455,7 +457,7 @@ speech-swift는 모델당 하나의 SPM 타겟으로 분리되어 있어 사용 **[백엔드, 메모리 테이블, 모듈 맵이 포함된 전체 아키텍처 다이어그램 → soniqo.audio/architecture](https://soniqo.audio/ko/architecture)** · **[API 레퍼런스 → soniqo.audio/api](https://soniqo.audio/ko/api)** · **[벤치마크 → soniqo.audio/benchmarks](https://soniqo.audio/ko/benchmarks)** 로컬 문서 (리포지토리): -- **모델:** [Qwen3-ASR](docs/models/asr-model.md) · [WhisperASR](docs/models/whisper-asr.md) · [Qwen3-TTS](docs/models/tts-model.md) · [CosyVoice](docs/models/cosyvoice-tts.md) · [Kokoro](docs/models/kokoro-tts.md) · [VoxCPM2](docs/models/voxcpm2-tts.md) · [VibeVoice](docs/models/vibevoice.md) · [Supertonic](docs/models/supertonic-tts.md) · [Chatterbox](docs/models/chatterbox-tts.md) · [Indic-Mio](docs/models/indic-mio-tts.md) · [Fish Audio S2 Pro](docs/models/fish-audio-s2-pro.md) · [Magpie TTS](docs/models/magpie-tts.md) · [Parakeet TDT](docs/models/parakeet-asr.md) · [Parakeet Streaming](docs/models/parakeet-streaming-asr.md) · [Nemotron Streaming](docs/models/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/models/omnilingual-asr.md) · [PersonaPlex](docs/models/personaplex.md) · [Hibiki](docs/models/hibiki.md) · [MADLAD-400](docs/models/madlad-translation.md) · [FunctionGemma](docs/models/function-gemma.md) · [Qwen3.5 Chat](docs/models/qwen35-chat.md) · [Gemma 4 Chat](docs/models/gemma4-chat.md) · [Qwen3 Dense Chat](docs/models/qwen3-dense-chat.md) · [FireRedVAD](docs/models/fireredvad.md) · [KWS Zipformer](docs/models/kws-zipformer.md) · [Sidon](docs/models/sidon.md) · [Source Separation](docs/models/source-separation.md) · [HTDemucs](docs/models/htdemucs.md) · [MAGNeT](docs/models/magnet-music-gen.md) · [Stable Audio 3](docs/models/stable-audio-3.md) · [FlashSR](docs/models/flashsr.md) +- **모델:** [Qwen3-ASR](docs/models/asr-model.md) · [WhisperASR](docs/models/whisper-asr.md) · [Qwen3-TTS](docs/models/tts-model.md) · [CosyVoice](docs/models/cosyvoice-tts.md) · [Kokoro](docs/models/kokoro-tts.md) · [VoxCPM2](docs/models/voxcpm2-tts.md) · [VibeVoice](docs/models/vibevoice.md) · [Supertonic](docs/models/supertonic-tts.md) · [Chatterbox](docs/models/chatterbox-tts.md) · [Indic-Mio](docs/models/indic-mio-tts.md) · [Fish Audio S2 Pro](docs/models/fish-audio-s2-pro.md) · [Magpie TTS](docs/models/magpie-tts.md) · [Parakeet TDT](docs/models/parakeet-asr.md) · [Parakeet Streaming](docs/models/parakeet-streaming-asr.md) · [Nemotron Streaming](docs/models/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/models/omnilingual-asr.md) · [PersonaPlex](docs/models/personaplex.md) · [Hibiki](docs/models/hibiki.md) · [MADLAD-400](docs/models/madlad-translation.md) · [FunctionGemma](docs/models/function-gemma.md) · [Qwen3.5 Chat](docs/models/qwen35-chat.md) · [Gemma 4 Chat](docs/models/gemma4-chat.md) · [Qwen3 Dense Chat](docs/models/qwen3-dense-chat.md) · [FireRedVAD](docs/models/fireredvad.md) · [KWS Zipformer](docs/models/kws-zipformer.md) · [Sidon](docs/models/sidon.md) · [Source Separation](docs/models/source-separation.md) · [HTDemucs](docs/models/htdemucs.md) · [MAGNeT](docs/models/magnet-music-gen.md) · [Stable Audio 3](docs/models/stable-audio-3.md) · [FlashSR](docs/models/flashsr.md) · [Audio2Face-3D](docs/models/audio2face3d.md) - **추론:** [Qwen3-ASR](docs/inference/qwen3-asr-inference.md) · [WhisperASR](docs/inference/whisper-asr-inference.md) · [Parakeet TDT](docs/inference/parakeet-asr-inference.md) · [Parakeet Streaming](docs/inference/parakeet-streaming-asr-inference.md) · [Nemotron Streaming](docs/inference/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/inference/omnilingual-asr-inference.md) · [TTS](docs/inference/qwen3-tts-inference.md) · [VoxCPM2](docs/inference/voxcpm2-inference.md) · [VibeVoice](docs/inference/vibevoice-inference.md) · [Fish Audio S2 Pro](docs/inference/fish-audio-s2-pro.md) · [Magpie TTS](docs/inference/magpie-tts.md) · [Hibiki](docs/inference/hibiki-inference.md) · [MADLAD-400](docs/inference/madlad-translation.md) · [MAGNeT](docs/inference/magnet-music-gen.md) · [Stable Audio 3](docs/inference/stable-audio-3.md) · [FlashSR](docs/inference/flashsr.md) · [Forced Aligner](docs/inference/forced-aligner.md) · [Silero VAD](docs/inference/silero-vad.md) · [FireRedVAD](docs/inference/fireredvad.md) · [Wake-word](docs/inference/wake-word.md) · [Speaker Diarization](docs/inference/speaker-diarization.md) · [Speech Enhancement](docs/inference/speech-enhancement.md) · [Sidon](docs/inference/sidon.md) · [Cache/offline](docs/inference/cache-and-offline.md) - **레퍼런스:** [공유 프로토콜](docs/shared-protocols.md) diff --git a/README_pt.md b/README_pt.md index 2bc76a7b..aa5b231a 100644 --- a/README_pt.md +++ b/README_pt.md @@ -60,6 +60,7 @@ Reconhecimento, sintese e compreensao de fala no dispositivo para Mac e iOS. Exe - **[Hibiki Zero-3B](https://soniqo.audio/guides/audio-translate)** — Tradução de fala para fala em streaming (FR/ES/PT/DE → EN, MLX INT4 + INT8, stack Kyutai Moshi/Mimi, CC-BY-4.0) - **[PersonaPlex](https://soniqo.audio/pt/guides/respond)** — Fala-a-fala full-duplex (7B, audio de entrada → audio de saida, 18 presets de voz) +- **[Audio2Face-3D](docs/models/audio2face3d.md)** — Animação facial de avatares guiada por fala (NVIDIA Audio2Face-3D v2.3 Mark, 301 coeficientes faciais, MLX) **Aprimoramento, separação e geração de áudio** @@ -144,7 +145,7 @@ struct DictateView: View { `SpeechUI` inclui apenas `TranscriptionView` (finais + parciais) e `TranscriptionStore` (adaptador de ASR em streaming). Use AVFoundation para visualizacao e reproducao de audio. -Produtos SPM disponiveis: `Qwen3ASR`, `WhisperASR`, `Qwen3TTS`, `Qwen3TTSCoreML`, `ParakeetASR`, `ParakeetStreamingASR`, `NemotronStreamingASR`, `OmnilingualASR`, `KokoroTTS`, `SupertonicTTS`, `VibeVoiceTTS`, `CosyVoiceTTS`, `VoxCPM2TTS`, `ChatterboxTTS`, `OmniVoiceTTS`, `IndicMioTTS`, `FishAudioTTS`, `MagpieTTS`, `MagpieTTSCoreML`, `MAGNeTMusicGen`, `StableAudio3MusicGen`, `FlashSR`, `PersonaPlex`, `HibikiTranslate`, `MADLADTranslation`, `SpeechVAD`, `SpeechWakeWord`, `SpeechEnhancement`, `SpeechRestoration`, `SourceSeparation`, `Qwen3Chat`, `FunctionGemma`, `SpeechCore`, `SpeechUI`, `AudioCommon`. +Produtos SPM disponiveis: `Qwen3ASR`, `WhisperASR`, `Qwen3TTS`, `Qwen3TTSCoreML`, `ParakeetASR`, `ParakeetStreamingASR`, `NemotronStreamingASR`, `OmnilingualASR`, `KokoroTTS`, `SupertonicTTS`, `VibeVoiceTTS`, `CosyVoiceTTS`, `VoxCPM2TTS`, `ChatterboxTTS`, `OmniVoiceTTS`, `IndicMioTTS`, `FishAudioTTS`, `MagpieTTS`, `MagpieTTSCoreML`, `MAGNeTMusicGen`, `StableAudio3MusicGen`, `FlashSR`, `PersonaPlex`, `Audio2Face3D`, `HibikiTranslate`, `MADLADTranslation`, `SpeechVAD`, `SpeechWakeWord`, `SpeechEnhancement`, `SpeechRestoration`, `SourceSeparation`, `Qwen3Chat`, `FunctionGemma`, `SpeechCore`, `SpeechUI`, `AudioCommon`. ## Modelos @@ -179,6 +180,7 @@ Vista compacta abaixo. **[Catalogo completo de modelos com tamanhos, quantizacoe | [MADLAD-400](https://soniqo.audio/pt/guides/translate) | Texto → Texto (Tradução) | MLX | 3B | **400+** | | [Hibiki Zero-3B](https://soniqo.audio/guides/audio-translate) | Fala → Fala (Tradução) | MLX | 3B | FR/ES/PT/DE → EN | | [PersonaPlex](https://soniqo.audio/pt/guides/respond) | Fala → Fala | MLX | 7B | EN | +| [Audio2Face-3D](docs/models/audio2face3d.md) | Fala → Animação facial | MLX | v2.3 Mark | Agnostico | | [Silero VAD](https://soniqo.audio/pt/guides/vad) | Deteccao de atividade de voz | MLX, CoreML | 309K | Agnostico | | [KWS Zipformer](docs/models/kws-zipformer.md) | Audio → Wake word | CoreML (ANE) | 3M | EN/custom keywords | | [Pyannote](https://soniqo.audio/pt/guides/diarize) | VAD + Diarizacao | MLX | 1.5M | Agnostico | @@ -455,7 +457,7 @@ speech-swift e dividido em um target SPM por modelo para que os consumidores pag **[Diagrama completo de arquitetura com backends, tabelas de memoria e mapa de modulos → soniqo.audio/architecture](https://soniqo.audio/pt/architecture)** · **[Referencia de API → soniqo.audio/api](https://soniqo.audio/pt/api)** · **[Benchmarks → soniqo.audio/benchmarks](https://soniqo.audio/pt/benchmarks)** Docs locais (repositorio): -- **Modelos:** [Qwen3-ASR](docs/models/asr-model.md) · [WhisperASR](docs/models/whisper-asr.md) · [Qwen3-TTS](docs/models/tts-model.md) · [CosyVoice](docs/models/cosyvoice-tts.md) · [Kokoro](docs/models/kokoro-tts.md) · [VoxCPM2](docs/models/voxcpm2-tts.md) · [VibeVoice](docs/models/vibevoice.md) · [Supertonic](docs/models/supertonic-tts.md) · [Chatterbox](docs/models/chatterbox-tts.md) · [Indic-Mio](docs/models/indic-mio-tts.md) · [Fish Audio S2 Pro](docs/models/fish-audio-s2-pro.md) · [Magpie TTS](docs/models/magpie-tts.md) · [Parakeet TDT](docs/models/parakeet-asr.md) · [Parakeet Streaming](docs/models/parakeet-streaming-asr.md) · [Nemotron Streaming](docs/models/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/models/omnilingual-asr.md) · [PersonaPlex](docs/models/personaplex.md) · [Hibiki](docs/models/hibiki.md) · [MADLAD-400](docs/models/madlad-translation.md) · [FunctionGemma](docs/models/function-gemma.md) · [Qwen3.5 Chat](docs/models/qwen35-chat.md) · [Gemma 4 Chat](docs/models/gemma4-chat.md) · [Qwen3 Dense Chat](docs/models/qwen3-dense-chat.md) · [FireRedVAD](docs/models/fireredvad.md) · [KWS Zipformer](docs/models/kws-zipformer.md) · [Sidon](docs/models/sidon.md) · [Source Separation](docs/models/source-separation.md) · [HTDemucs](docs/models/htdemucs.md) · [MAGNeT](docs/models/magnet-music-gen.md) · [Stable Audio 3](docs/models/stable-audio-3.md) · [FlashSR](docs/models/flashsr.md) +- **Modelos:** [Qwen3-ASR](docs/models/asr-model.md) · [WhisperASR](docs/models/whisper-asr.md) · [Qwen3-TTS](docs/models/tts-model.md) · [CosyVoice](docs/models/cosyvoice-tts.md) · [Kokoro](docs/models/kokoro-tts.md) · [VoxCPM2](docs/models/voxcpm2-tts.md) · [VibeVoice](docs/models/vibevoice.md) · [Supertonic](docs/models/supertonic-tts.md) · [Chatterbox](docs/models/chatterbox-tts.md) · [Indic-Mio](docs/models/indic-mio-tts.md) · [Fish Audio S2 Pro](docs/models/fish-audio-s2-pro.md) · [Magpie TTS](docs/models/magpie-tts.md) · [Parakeet TDT](docs/models/parakeet-asr.md) · [Parakeet Streaming](docs/models/parakeet-streaming-asr.md) · [Nemotron Streaming](docs/models/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/models/omnilingual-asr.md) · [PersonaPlex](docs/models/personaplex.md) · [Hibiki](docs/models/hibiki.md) · [MADLAD-400](docs/models/madlad-translation.md) · [FunctionGemma](docs/models/function-gemma.md) · [Qwen3.5 Chat](docs/models/qwen35-chat.md) · [Gemma 4 Chat](docs/models/gemma4-chat.md) · [Qwen3 Dense Chat](docs/models/qwen3-dense-chat.md) · [FireRedVAD](docs/models/fireredvad.md) · [KWS Zipformer](docs/models/kws-zipformer.md) · [Sidon](docs/models/sidon.md) · [Source Separation](docs/models/source-separation.md) · [HTDemucs](docs/models/htdemucs.md) · [MAGNeT](docs/models/magnet-music-gen.md) · [Stable Audio 3](docs/models/stable-audio-3.md) · [FlashSR](docs/models/flashsr.md) · [Audio2Face-3D](docs/models/audio2face3d.md) - **Inferencia:** [Qwen3-ASR](docs/inference/qwen3-asr-inference.md) · [WhisperASR](docs/inference/whisper-asr-inference.md) · [Parakeet TDT](docs/inference/parakeet-asr-inference.md) · [Parakeet Streaming](docs/inference/parakeet-streaming-asr-inference.md) · [Nemotron Streaming](docs/inference/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/inference/omnilingual-asr-inference.md) · [TTS](docs/inference/qwen3-tts-inference.md) · [VoxCPM2](docs/inference/voxcpm2-inference.md) · [VibeVoice](docs/inference/vibevoice-inference.md) · [Fish Audio S2 Pro](docs/inference/fish-audio-s2-pro.md) · [Magpie TTS](docs/inference/magpie-tts.md) · [Hibiki](docs/inference/hibiki-inference.md) · [MADLAD-400](docs/inference/madlad-translation.md) · [MAGNeT](docs/inference/magnet-music-gen.md) · [Stable Audio 3](docs/inference/stable-audio-3.md) · [FlashSR](docs/inference/flashsr.md) · [Forced Aligner](docs/inference/forced-aligner.md) · [Silero VAD](docs/inference/silero-vad.md) · [FireRedVAD](docs/inference/fireredvad.md) · [Wake-word](docs/inference/wake-word.md) · [Speaker Diarization](docs/inference/speaker-diarization.md) · [Speech Enhancement](docs/inference/speech-enhancement.md) · [Sidon](docs/inference/sidon.md) · [Cache/offline](docs/inference/cache-and-offline.md) - **Referencia:** [Protocolos compartilhados](docs/shared-protocols.md) diff --git a/README_ru.md b/README_ru.md index 4582a68a..30f5d7ca 100644 --- a/README_ru.md +++ b/README_ru.md @@ -60,6 +60,7 @@ - **[Hibiki Zero-3B](https://soniqo.audio/guides/audio-translate)** — Потоковый перевод речи в речь (FR/ES/PT/DE → EN, MLX INT4 + INT8, стек Kyutai Moshi/Mimi, CC-BY-4.0) - **[PersonaPlex](https://soniqo.audio/ru/guides/respond)** — Полнодуплексная генерация речи из речи (7B, аудио на входе → аудио на выходе, 18 голосовых пресетов) +- **[Audio2Face-3D](docs/models/audio2face3d.md)** — Анимация лица аватара по речи (NVIDIA Audio2Face-3D v2.3 Mark, 301 лицевой коэффициент, MLX) **Улучшение, разделение и генерация аудио** @@ -144,7 +145,7 @@ struct DictateView: View { `SpeechUI` предоставляет только `TranscriptionView` (финальные + частичные результаты) и `TranscriptionStore` (адаптер для потокового ASR). Для визуализации и воспроизведения аудио используйте AVFoundation. -Доступные SPM-продукты: `Qwen3ASR`, `WhisperASR`, `Qwen3TTS`, `Qwen3TTSCoreML`, `ParakeetASR`, `ParakeetStreamingASR`, `NemotronStreamingASR`, `OmnilingualASR`, `KokoroTTS`, `SupertonicTTS`, `VibeVoiceTTS`, `CosyVoiceTTS`, `VoxCPM2TTS`, `ChatterboxTTS`, `OmniVoiceTTS`, `IndicMioTTS`, `FishAudioTTS`, `MagpieTTS`, `MagpieTTSCoreML`, `MAGNeTMusicGen`, `StableAudio3MusicGen`, `FlashSR`, `PersonaPlex`, `HibikiTranslate`, `MADLADTranslation`, `SpeechVAD`, `SpeechWakeWord`, `SpeechEnhancement`, `SpeechRestoration`, `SourceSeparation`, `Qwen3Chat`, `FunctionGemma`, `SpeechCore`, `SpeechUI`, `AudioCommon`. +Доступные SPM-продукты: `Qwen3ASR`, `WhisperASR`, `Qwen3TTS`, `Qwen3TTSCoreML`, `ParakeetASR`, `ParakeetStreamingASR`, `NemotronStreamingASR`, `OmnilingualASR`, `KokoroTTS`, `SupertonicTTS`, `VibeVoiceTTS`, `CosyVoiceTTS`, `VoxCPM2TTS`, `ChatterboxTTS`, `OmniVoiceTTS`, `IndicMioTTS`, `FishAudioTTS`, `MagpieTTS`, `MagpieTTSCoreML`, `MAGNeTMusicGen`, `StableAudio3MusicGen`, `FlashSR`, `PersonaPlex`, `Audio2Face3D`, `HibikiTranslate`, `MADLADTranslation`, `SpeechVAD`, `SpeechWakeWord`, `SpeechEnhancement`, `SpeechRestoration`, `SourceSeparation`, `Qwen3Chat`, `FunctionGemma`, `SpeechCore`, `SpeechUI`, `AudioCommon`. ## Модели @@ -179,6 +180,7 @@ struct DictateView: View { | [MADLAD-400](https://soniqo.audio/ru/guides/translate) | Текст → Текст (Перевод) | MLX | 3B | **400+** | | [Hibiki Zero-3B](https://soniqo.audio/guides/audio-translate) | Речь → Речь (Перевод) | MLX | 3B | FR/ES/PT/DE → EN | | [PersonaPlex](https://soniqo.audio/ru/guides/respond) | Речь → Речь | MLX | 7B | EN | +| [Audio2Face-3D](docs/models/audio2face3d.md) | Речь → Анимация лица | MLX | v2.3 Mark | Универсальный | | [Silero VAD](https://soniqo.audio/ru/guides/vad) | Детектор речи | MLX, CoreML | 309K | Универсальный | | [KWS Zipformer](docs/models/kws-zipformer.md) | Audio → Wake word | CoreML (ANE) | 3M | EN/custom keywords | | [Pyannote](https://soniqo.audio/ru/guides/diarize) | VAD + Диаризация | MLX | 1.5M | Универсальный | @@ -455,7 +457,7 @@ speech-swift разделён на отдельные SPM-таргеты для **[Полная архитектурная диаграмма с бэкендами, таблицами памяти и картой модулей → soniqo.audio/architecture](https://soniqo.audio/ru/architecture)** · **[Справочник API → soniqo.audio/api](https://soniqo.audio/ru/api)** · **[Бенчмарки → soniqo.audio/benchmarks](https://soniqo.audio/ru/benchmarks)** Локальная документация (в репозитории): -- **Модели:** [Qwen3-ASR](docs/models/asr-model.md) · [WhisperASR](docs/models/whisper-asr.md) · [Qwen3-TTS](docs/models/tts-model.md) · [CosyVoice](docs/models/cosyvoice-tts.md) · [Kokoro](docs/models/kokoro-tts.md) · [VoxCPM2](docs/models/voxcpm2-tts.md) · [VibeVoice](docs/models/vibevoice.md) · [Supertonic](docs/models/supertonic-tts.md) · [Chatterbox](docs/models/chatterbox-tts.md) · [Indic-Mio](docs/models/indic-mio-tts.md) · [Fish Audio S2 Pro](docs/models/fish-audio-s2-pro.md) · [Magpie TTS](docs/models/magpie-tts.md) · [Parakeet TDT](docs/models/parakeet-asr.md) · [Parakeet Streaming](docs/models/parakeet-streaming-asr.md) · [Nemotron Streaming](docs/models/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/models/omnilingual-asr.md) · [PersonaPlex](docs/models/personaplex.md) · [Hibiki](docs/models/hibiki.md) · [MADLAD-400](docs/models/madlad-translation.md) · [FunctionGemma](docs/models/function-gemma.md) · [Qwen3.5 Chat](docs/models/qwen35-chat.md) · [Gemma 4 Chat](docs/models/gemma4-chat.md) · [Qwen3 Dense Chat](docs/models/qwen3-dense-chat.md) · [FireRedVAD](docs/models/fireredvad.md) · [KWS Zipformer](docs/models/kws-zipformer.md) · [Sidon](docs/models/sidon.md) · [Source Separation](docs/models/source-separation.md) · [HTDemucs](docs/models/htdemucs.md) · [MAGNeT](docs/models/magnet-music-gen.md) · [Stable Audio 3](docs/models/stable-audio-3.md) · [FlashSR](docs/models/flashsr.md) +- **Модели:** [Qwen3-ASR](docs/models/asr-model.md) · [WhisperASR](docs/models/whisper-asr.md) · [Qwen3-TTS](docs/models/tts-model.md) · [CosyVoice](docs/models/cosyvoice-tts.md) · [Kokoro](docs/models/kokoro-tts.md) · [VoxCPM2](docs/models/voxcpm2-tts.md) · [VibeVoice](docs/models/vibevoice.md) · [Supertonic](docs/models/supertonic-tts.md) · [Chatterbox](docs/models/chatterbox-tts.md) · [Indic-Mio](docs/models/indic-mio-tts.md) · [Fish Audio S2 Pro](docs/models/fish-audio-s2-pro.md) · [Magpie TTS](docs/models/magpie-tts.md) · [Parakeet TDT](docs/models/parakeet-asr.md) · [Parakeet Streaming](docs/models/parakeet-streaming-asr.md) · [Nemotron Streaming](docs/models/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/models/omnilingual-asr.md) · [PersonaPlex](docs/models/personaplex.md) · [Hibiki](docs/models/hibiki.md) · [MADLAD-400](docs/models/madlad-translation.md) · [FunctionGemma](docs/models/function-gemma.md) · [Qwen3.5 Chat](docs/models/qwen35-chat.md) · [Gemma 4 Chat](docs/models/gemma4-chat.md) · [Qwen3 Dense Chat](docs/models/qwen3-dense-chat.md) · [FireRedVAD](docs/models/fireredvad.md) · [KWS Zipformer](docs/models/kws-zipformer.md) · [Sidon](docs/models/sidon.md) · [Source Separation](docs/models/source-separation.md) · [HTDemucs](docs/models/htdemucs.md) · [MAGNeT](docs/models/magnet-music-gen.md) · [Stable Audio 3](docs/models/stable-audio-3.md) · [FlashSR](docs/models/flashsr.md) · [Audio2Face-3D](docs/models/audio2face3d.md) - **Инференс:** [Qwen3-ASR](docs/inference/qwen3-asr-inference.md) · [WhisperASR](docs/inference/whisper-asr-inference.md) · [Parakeet TDT](docs/inference/parakeet-asr-inference.md) · [Parakeet Streaming](docs/inference/parakeet-streaming-asr-inference.md) · [Nemotron Streaming](docs/inference/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/inference/omnilingual-asr-inference.md) · [TTS](docs/inference/qwen3-tts-inference.md) · [VoxCPM2](docs/inference/voxcpm2-inference.md) · [VibeVoice](docs/inference/vibevoice-inference.md) · [Fish Audio S2 Pro](docs/inference/fish-audio-s2-pro.md) · [Magpie TTS](docs/inference/magpie-tts.md) · [Hibiki](docs/inference/hibiki-inference.md) · [MADLAD-400](docs/inference/madlad-translation.md) · [MAGNeT](docs/inference/magnet-music-gen.md) · [Stable Audio 3](docs/inference/stable-audio-3.md) · [FlashSR](docs/inference/flashsr.md) · [Forced Aligner](docs/inference/forced-aligner.md) · [Silero VAD](docs/inference/silero-vad.md) · [FireRedVAD](docs/inference/fireredvad.md) · [Wake-word](docs/inference/wake-word.md) · [Speaker Diarization](docs/inference/speaker-diarization.md) · [Speech Enhancement](docs/inference/speech-enhancement.md) · [Sidon](docs/inference/sidon.md) · [Cache/offline](docs/inference/cache-and-offline.md) - **Справочник:** [Общие протоколы](docs/shared-protocols.md) diff --git a/README_th.md b/README_th.md index 1a335ed9..86029b49 100644 --- a/README_th.md +++ b/README_th.md @@ -60,6 +60,7 @@ - **[Hibiki Zero-3B](https://soniqo.audio/guides/audio-translate)** — การแปลเสียงพูดสู่เสียงพูดแบบสตรีมมิ่ง (FR/ES/PT/DE → EN, MLX INT4 + INT8, สแต็ก Kyutai Moshi/Mimi, CC-BY-4.0) - **[PersonaPlex](https://soniqo.audio/guides/respond)** — เสียงพูดสู่เสียงพูดแบบ full-duplex (7B, เสียงเข้า → เสียงออก, 18 พรีเซ็ตเสียง) +- **[Audio2Face-3D](docs/models/audio2face3d.md)** — แอนิเมชันใบหน้าอวาตาร์ขับเคลื่อนด้วยเสียงพูด (NVIDIA Audio2Face-3D v2.3 Mark, ค่าสัมประสิทธิ์ใบหน้า 301 ค่า, MLX) **การปรับปรุง การแยก และการสร้างเสียง** @@ -144,7 +145,7 @@ struct DictateView: View { `SpeechUI` มาพร้อมเพียง `TranscriptionView` (ผลลัพธ์สุดท้าย + บางส่วน) และ `TranscriptionStore` (อะแดปเตอร์สำหรับ ASR แบบสตรีมมิ่ง) ใช้ AVFoundation สำหรับการแสดงผลภาพเสียงและการเล่นเสียง -ผลิตภัณฑ์ SPM ที่มีให้: `Qwen3ASR`, `WhisperASR`, `Qwen3TTS`, `Qwen3TTSCoreML`, `ParakeetASR`, `ParakeetStreamingASR`, `NemotronStreamingASR`, `OmnilingualASR`, `KokoroTTS`, `SupertonicTTS`, `VibeVoiceTTS`, `CosyVoiceTTS`, `VoxCPM2TTS`, `ChatterboxTTS`, `OmniVoiceTTS`, `IndicMioTTS`, `FishAudioTTS`, `MagpieTTS`, `MagpieTTSCoreML`, `MAGNeTMusicGen`, `StableAudio3MusicGen`, `FlashSR`, `PersonaPlex`, `HibikiTranslate`, `MADLADTranslation`, `SpeechVAD`, `SpeechWakeWord`, `SpeechEnhancement`, `SpeechRestoration`, `SourceSeparation`, `Qwen3Chat`, `FunctionGemma`, `SpeechCore`, `SpeechUI`, `AudioCommon` +ผลิตภัณฑ์ SPM ที่มีให้: `Qwen3ASR`, `WhisperASR`, `Qwen3TTS`, `Qwen3TTSCoreML`, `ParakeetASR`, `ParakeetStreamingASR`, `NemotronStreamingASR`, `OmnilingualASR`, `KokoroTTS`, `SupertonicTTS`, `VibeVoiceTTS`, `CosyVoiceTTS`, `VoxCPM2TTS`, `ChatterboxTTS`, `OmniVoiceTTS`, `IndicMioTTS`, `FishAudioTTS`, `MagpieTTS`, `MagpieTTSCoreML`, `MAGNeTMusicGen`, `StableAudio3MusicGen`, `FlashSR`, `PersonaPlex`, `Audio2Face3D`, `HibikiTranslate`, `MADLADTranslation`, `SpeechVAD`, `SpeechWakeWord`, `SpeechEnhancement`, `SpeechRestoration`, `SourceSeparation`, `Qwen3Chat`, `FunctionGemma`, `SpeechCore`, `SpeechUI`, `AudioCommon` ## โมเดล @@ -179,6 +180,7 @@ struct DictateView: View { | [MADLAD-400](https://soniqo.audio/guides/translate) | ข้อความ → ข้อความ (การแปล) | MLX | 3B | **400+** | | [Hibiki Zero-3B](https://soniqo.audio/guides/audio-translate) | เสียงพูด → เสียงพูด (การแปล) | MLX | 3B | FR/ES/PT/DE → EN | | [PersonaPlex](https://soniqo.audio/guides/respond) | เสียงพูด → เสียงพูด | MLX | 7B | EN | +| [Audio2Face-3D](docs/models/audio2face3d.md) | เสียงพูด → แอนิเมชันใบหน้า | MLX | v2.3 Mark | ไม่จำกัดภาษา | | [Silero VAD](https://soniqo.audio/guides/vad) | การตรวจจับเสียงพูด | MLX, CoreML | 309K | ไม่จำกัดภาษา | | [KWS Zipformer](docs/models/kws-zipformer.md) | Audio → Wake word | CoreML (ANE) | 3M | EN/custom keywords | | [Pyannote](https://soniqo.audio/guides/diarize) | VAD + การแยกผู้พูด | MLX | 1.5M | ไม่จำกัดภาษา | @@ -453,7 +455,7 @@ speech-swift ถูกแบ่งเป็นหนึ่ง SPM target ต่ **[แผนภาพสถาปัตยกรรมฉบับเต็มพร้อม backends ตารางหน่วยความจำ และแผนผังโมดูล → soniqo.audio/architecture](https://soniqo.audio/architecture)** · **[API reference → soniqo.audio/api](https://soniqo.audio/api)** · **[Benchmarks → soniqo.audio/benchmarks](https://soniqo.audio/benchmarks)** เอกสารในเครื่อง (repo): -- **โมเดล:** [Qwen3-ASR](docs/models/asr-model.md) · [WhisperASR](docs/models/whisper-asr.md) · [Qwen3-TTS](docs/models/tts-model.md) · [CosyVoice](docs/models/cosyvoice-tts.md) · [Kokoro](docs/models/kokoro-tts.md) · [VoxCPM2](docs/models/voxcpm2-tts.md) · [VibeVoice](docs/models/vibevoice.md) · [Supertonic](docs/models/supertonic-tts.md) · [Chatterbox](docs/models/chatterbox-tts.md) · [Indic-Mio](docs/models/indic-mio-tts.md) · [Fish Audio S2 Pro](docs/models/fish-audio-s2-pro.md) · [Magpie TTS](docs/models/magpie-tts.md) · [Parakeet TDT](docs/models/parakeet-asr.md) · [Parakeet Streaming](docs/models/parakeet-streaming-asr.md) · [Nemotron Streaming](docs/models/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/models/omnilingual-asr.md) · [PersonaPlex](docs/models/personaplex.md) · [Hibiki](docs/models/hibiki.md) · [MADLAD-400](docs/models/madlad-translation.md) · [FunctionGemma](docs/models/function-gemma.md) · [Qwen3.5 Chat](docs/models/qwen35-chat.md) · [Gemma 4 Chat](docs/models/gemma4-chat.md) · [Qwen3 Dense Chat](docs/models/qwen3-dense-chat.md) · [FireRedVAD](docs/models/fireredvad.md) · [KWS Zipformer](docs/models/kws-zipformer.md) · [Sidon](docs/models/sidon.md) · [Source Separation](docs/models/source-separation.md) · [HTDemucs](docs/models/htdemucs.md) · [MAGNeT](docs/models/magnet-music-gen.md) · [Stable Audio 3](docs/models/stable-audio-3.md) · [FlashSR](docs/models/flashsr.md) +- **โมเดล:** [Qwen3-ASR](docs/models/asr-model.md) · [WhisperASR](docs/models/whisper-asr.md) · [Qwen3-TTS](docs/models/tts-model.md) · [CosyVoice](docs/models/cosyvoice-tts.md) · [Kokoro](docs/models/kokoro-tts.md) · [VoxCPM2](docs/models/voxcpm2-tts.md) · [VibeVoice](docs/models/vibevoice.md) · [Supertonic](docs/models/supertonic-tts.md) · [Chatterbox](docs/models/chatterbox-tts.md) · [Indic-Mio](docs/models/indic-mio-tts.md) · [Fish Audio S2 Pro](docs/models/fish-audio-s2-pro.md) · [Magpie TTS](docs/models/magpie-tts.md) · [Parakeet TDT](docs/models/parakeet-asr.md) · [Parakeet Streaming](docs/models/parakeet-streaming-asr.md) · [Nemotron Streaming](docs/models/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/models/omnilingual-asr.md) · [PersonaPlex](docs/models/personaplex.md) · [Hibiki](docs/models/hibiki.md) · [MADLAD-400](docs/models/madlad-translation.md) · [FunctionGemma](docs/models/function-gemma.md) · [Qwen3.5 Chat](docs/models/qwen35-chat.md) · [Gemma 4 Chat](docs/models/gemma4-chat.md) · [Qwen3 Dense Chat](docs/models/qwen3-dense-chat.md) · [FireRedVAD](docs/models/fireredvad.md) · [KWS Zipformer](docs/models/kws-zipformer.md) · [Sidon](docs/models/sidon.md) · [Source Separation](docs/models/source-separation.md) · [HTDemucs](docs/models/htdemucs.md) · [MAGNeT](docs/models/magnet-music-gen.md) · [Stable Audio 3](docs/models/stable-audio-3.md) · [FlashSR](docs/models/flashsr.md) · [Audio2Face-3D](docs/models/audio2face3d.md) - **การอนุมาน:** [Qwen3-ASR](docs/inference/qwen3-asr-inference.md) · [WhisperASR](docs/inference/whisper-asr-inference.md) · [Parakeet TDT](docs/inference/parakeet-asr-inference.md) · [Parakeet Streaming](docs/inference/parakeet-streaming-asr-inference.md) · [Nemotron Streaming](docs/inference/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/inference/omnilingual-asr-inference.md) · [TTS](docs/inference/qwen3-tts-inference.md) · [VoxCPM2](docs/inference/voxcpm2-inference.md) · [VibeVoice](docs/inference/vibevoice-inference.md) · [Fish Audio S2 Pro](docs/inference/fish-audio-s2-pro.md) · [Magpie TTS](docs/inference/magpie-tts.md) · [Hibiki](docs/inference/hibiki-inference.md) · [MADLAD-400](docs/inference/madlad-translation.md) · [MAGNeT](docs/inference/magnet-music-gen.md) · [Stable Audio 3](docs/inference/stable-audio-3.md) · [FlashSR](docs/inference/flashsr.md) · [Forced Aligner](docs/inference/forced-aligner.md) · [Silero VAD](docs/inference/silero-vad.md) · [FireRedVAD](docs/inference/fireredvad.md) · [Wake-word](docs/inference/wake-word.md) · [Speaker Diarization](docs/inference/speaker-diarization.md) · [Speech Enhancement](docs/inference/speech-enhancement.md) · [Sidon](docs/inference/sidon.md) · [Cache/offline](docs/inference/cache-and-offline.md) - **อ้างอิง:** [Shared Protocols](docs/shared-protocols.md) diff --git a/README_tr.md b/README_tr.md index 49942861..85d7cd5f 100644 --- a/README_tr.md +++ b/README_tr.md @@ -60,6 +60,7 @@ Mac ve iOS için cihaz üzerinde konuşma tanıma, sentezleme ve anlama. Apple S - **[Hibiki Zero-3B](https://soniqo.audio/guides/audio-translate)** — Akışlı konuşmadan konuşmaya çeviri (FR/ES/PT/DE → EN, MLX INT4 + INT8, Kyutai Moshi/Mimi yığını, CC-BY-4.0) - **[PersonaPlex](https://soniqo.audio/guides/respond)** — Tam çift yönlü (full-duplex) konuşmadan konuşmaya (7B, ses girişi → ses çıkışı, 18 ses ön ayarı) +- **[Audio2Face-3D](docs/models/audio2face3d.md)** — Konuşmayla sürülen avatar yüz animasyonu (NVIDIA Audio2Face-3D v2.3 Mark, 301 yüz katsayısı, MLX) **İyileştirme, ayırma ve ses üretimi** @@ -144,7 +145,7 @@ struct DictateView: View { `SpeechUI` yalnızca `TranscriptionView` (kesin sonuçlar + kısmi sonuçlar) ve `TranscriptionStore` (akış ASR adaptörü) sunar. Ses görselleştirme ve oynatma için AVFoundation kullanın. -Mevcut SPM ürünleri: `Qwen3ASR`, `WhisperASR`, `Qwen3TTS`, `Qwen3TTSCoreML`, `ParakeetASR`, `ParakeetStreamingASR`, `NemotronStreamingASR`, `OmnilingualASR`, `KokoroTTS`, `SupertonicTTS`, `VibeVoiceTTS`, `CosyVoiceTTS`, `VoxCPM2TTS`, `ChatterboxTTS`, `OmniVoiceTTS`, `IndicMioTTS`, `FishAudioTTS`, `MagpieTTS`, `MagpieTTSCoreML`, `MAGNeTMusicGen`, `StableAudio3MusicGen`, `FlashSR`, `PersonaPlex`, `HibikiTranslate`, `MADLADTranslation`, `SpeechVAD`, `SpeechWakeWord`, `SpeechEnhancement`, `SpeechRestoration`, `SourceSeparation`, `Qwen3Chat`, `FunctionGemma`, `SpeechCore`, `SpeechUI`, `AudioCommon`. +Mevcut SPM ürünleri: `Qwen3ASR`, `WhisperASR`, `Qwen3TTS`, `Qwen3TTSCoreML`, `ParakeetASR`, `ParakeetStreamingASR`, `NemotronStreamingASR`, `OmnilingualASR`, `KokoroTTS`, `SupertonicTTS`, `VibeVoiceTTS`, `CosyVoiceTTS`, `VoxCPM2TTS`, `ChatterboxTTS`, `OmniVoiceTTS`, `IndicMioTTS`, `FishAudioTTS`, `MagpieTTS`, `MagpieTTSCoreML`, `MAGNeTMusicGen`, `StableAudio3MusicGen`, `FlashSR`, `PersonaPlex`, `Audio2Face3D`, `HibikiTranslate`, `MADLADTranslation`, `SpeechVAD`, `SpeechWakeWord`, `SpeechEnhancement`, `SpeechRestoration`, `SourceSeparation`, `Qwen3Chat`, `FunctionGemma`, `SpeechCore`, `SpeechUI`, `AudioCommon`. ## Modeller @@ -179,6 +180,7 @@ Aşağıda kompakt bir görünüm. **[Boyutlar, kuantizasyonlar, indirme URL'ler | [MADLAD-400](https://soniqo.audio/guides/translate) | Metin → Metin (Çeviri) | MLX | 3B | **400+** | | [Hibiki Zero-3B](https://soniqo.audio/guides/audio-translate) | Konuşma → Konuşma (Çeviri) | MLX | 3B | FR/ES/PT/DE → EN | | [PersonaPlex](https://soniqo.audio/guides/respond) | Konuşma → Konuşma | MLX | 7B | EN | +| [Audio2Face-3D](docs/models/audio2face3d.md) | Konuşma → Yüz animasyonu | MLX | v2.3 Mark | Bağımsız | | [Silero VAD](https://soniqo.audio/guides/vad) | Ses Etkinlik Algılama | MLX, CoreML | 309K | Bağımsız | | [KWS Zipformer](docs/models/kws-zipformer.md) | Audio → Wake word | CoreML (ANE) | 3M | EN/custom keywords | | [Pyannote](https://soniqo.audio/guides/diarize) | VAD + Konuşmacı Ayrımı | MLX | 1.5M | Bağımsız | @@ -453,7 +455,7 @@ speech-swift, kullanıcıların yalnızca içe aktardıkları şey için bedel **[Backend'ler, bellek tabloları ve modül haritasıyla tam mimari diyagramı → soniqo.audio/architecture](https://soniqo.audio/architecture)** · **[API referansı → soniqo.audio/api](https://soniqo.audio/api)** · **[Benchmark'lar → soniqo.audio/benchmarks](https://soniqo.audio/benchmarks)** Yerel dokümantasyon (depo): -- **Modeller:** [Qwen3-ASR](docs/models/asr-model.md) · [WhisperASR](docs/models/whisper-asr.md) · [Qwen3-TTS](docs/models/tts-model.md) · [CosyVoice](docs/models/cosyvoice-tts.md) · [Kokoro](docs/models/kokoro-tts.md) · [VoxCPM2](docs/models/voxcpm2-tts.md) · [VibeVoice](docs/models/vibevoice.md) · [Supertonic](docs/models/supertonic-tts.md) · [Chatterbox](docs/models/chatterbox-tts.md) · [Indic-Mio](docs/models/indic-mio-tts.md) · [Fish Audio S2 Pro](docs/models/fish-audio-s2-pro.md) · [Magpie TTS](docs/models/magpie-tts.md) · [Parakeet TDT](docs/models/parakeet-asr.md) · [Parakeet Streaming](docs/models/parakeet-streaming-asr.md) · [Nemotron Streaming](docs/models/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/models/omnilingual-asr.md) · [PersonaPlex](docs/models/personaplex.md) · [Hibiki](docs/models/hibiki.md) · [MADLAD-400](docs/models/madlad-translation.md) · [FunctionGemma](docs/models/function-gemma.md) · [Qwen3.5 Chat](docs/models/qwen35-chat.md) · [Gemma 4 Chat](docs/models/gemma4-chat.md) · [Qwen3 Dense Chat](docs/models/qwen3-dense-chat.md) · [FireRedVAD](docs/models/fireredvad.md) · [KWS Zipformer](docs/models/kws-zipformer.md) · [Sidon](docs/models/sidon.md) · [Source Separation](docs/models/source-separation.md) · [HTDemucs](docs/models/htdemucs.md) · [MAGNeT](docs/models/magnet-music-gen.md) · [Stable Audio 3](docs/models/stable-audio-3.md) · [FlashSR](docs/models/flashsr.md) +- **Modeller:** [Qwen3-ASR](docs/models/asr-model.md) · [WhisperASR](docs/models/whisper-asr.md) · [Qwen3-TTS](docs/models/tts-model.md) · [CosyVoice](docs/models/cosyvoice-tts.md) · [Kokoro](docs/models/kokoro-tts.md) · [VoxCPM2](docs/models/voxcpm2-tts.md) · [VibeVoice](docs/models/vibevoice.md) · [Supertonic](docs/models/supertonic-tts.md) · [Chatterbox](docs/models/chatterbox-tts.md) · [Indic-Mio](docs/models/indic-mio-tts.md) · [Fish Audio S2 Pro](docs/models/fish-audio-s2-pro.md) · [Magpie TTS](docs/models/magpie-tts.md) · [Parakeet TDT](docs/models/parakeet-asr.md) · [Parakeet Streaming](docs/models/parakeet-streaming-asr.md) · [Nemotron Streaming](docs/models/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/models/omnilingual-asr.md) · [PersonaPlex](docs/models/personaplex.md) · [Hibiki](docs/models/hibiki.md) · [MADLAD-400](docs/models/madlad-translation.md) · [FunctionGemma](docs/models/function-gemma.md) · [Qwen3.5 Chat](docs/models/qwen35-chat.md) · [Gemma 4 Chat](docs/models/gemma4-chat.md) · [Qwen3 Dense Chat](docs/models/qwen3-dense-chat.md) · [FireRedVAD](docs/models/fireredvad.md) · [KWS Zipformer](docs/models/kws-zipformer.md) · [Sidon](docs/models/sidon.md) · [Source Separation](docs/models/source-separation.md) · [HTDemucs](docs/models/htdemucs.md) · [MAGNeT](docs/models/magnet-music-gen.md) · [Stable Audio 3](docs/models/stable-audio-3.md) · [FlashSR](docs/models/flashsr.md) · [Audio2Face-3D](docs/models/audio2face3d.md) - **Çıkarım:** [Qwen3-ASR](docs/inference/qwen3-asr-inference.md) · [WhisperASR](docs/inference/whisper-asr-inference.md) · [Parakeet TDT](docs/inference/parakeet-asr-inference.md) · [Parakeet Streaming](docs/inference/parakeet-streaming-asr-inference.md) · [Nemotron Streaming](docs/inference/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/inference/omnilingual-asr-inference.md) · [TTS](docs/inference/qwen3-tts-inference.md) · [VoxCPM2](docs/inference/voxcpm2-inference.md) · [VibeVoice](docs/inference/vibevoice-inference.md) · [Fish Audio S2 Pro](docs/inference/fish-audio-s2-pro.md) · [Magpie TTS](docs/inference/magpie-tts.md) · [Hibiki](docs/inference/hibiki-inference.md) · [MADLAD-400](docs/inference/madlad-translation.md) · [MAGNeT](docs/inference/magnet-music-gen.md) · [Stable Audio 3](docs/inference/stable-audio-3.md) · [FlashSR](docs/inference/flashsr.md) · [Forced Aligner](docs/inference/forced-aligner.md) · [Silero VAD](docs/inference/silero-vad.md) · [FireRedVAD](docs/inference/fireredvad.md) · [Wake-word](docs/inference/wake-word.md) · [Speaker Diarization](docs/inference/speaker-diarization.md) · [Speech Enhancement](docs/inference/speech-enhancement.md) · [Sidon](docs/inference/sidon.md) · [Cache/offline](docs/inference/cache-and-offline.md) - **Referans:** [Paylaşılan Protokoller](docs/shared-protocols.md) diff --git a/README_vi.md b/README_vi.md index 8e2555c6..b012c4c3 100644 --- a/README_vi.md +++ b/README_vi.md @@ -60,6 +60,7 @@ Nhận dạng, tổng hợp và hiểu giọng nói trên thiết bị cho Mac v - **[Hibiki Zero-3B](https://soniqo.audio/guides/audio-translate)** — Dịch giọng nói sang giọng nói streaming (FR/ES/PT/DE → EN, MLX INT4 + INT8, stack Kyutai Moshi/Mimi, CC-BY-4.0) - **[PersonaPlex](https://soniqo.audio/guides/respond)** — Giọng nói sang giọng nói full-duplex (7B, audio vào → audio ra, 18 preset giọng nói) +- **[Audio2Face-3D](docs/models/audio2face3d.md)** — Hoạt ảnh khuôn mặt avatar điều khiển bằng giọng nói (NVIDIA Audio2Face-3D v2.3 Mark, 301 hệ số khuôn mặt, MLX) **Nâng cấp, tách nguồn và tạo âm thanh** @@ -144,7 +145,7 @@ struct DictateView: View { `SpeechUI` chỉ cung cấp `TranscriptionView` (kết quả cuối + tạm thời) và `TranscriptionStore` (adapter ASR streaming). Hãy dùng AVFoundation để hiển thị trực quan và phát lại âm thanh. -Các sản phẩm SPM có sẵn: `Qwen3ASR`, `WhisperASR`, `Qwen3TTS`, `Qwen3TTSCoreML`, `ParakeetASR`, `ParakeetStreamingASR`, `NemotronStreamingASR`, `OmnilingualASR`, `KokoroTTS`, `SupertonicTTS`, `VibeVoiceTTS`, `CosyVoiceTTS`, `VoxCPM2TTS`, `ChatterboxTTS`, `OmniVoiceTTS`, `IndicMioTTS`, `FishAudioTTS`, `MagpieTTS`, `MagpieTTSCoreML`, `MAGNeTMusicGen`, `StableAudio3MusicGen`, `FlashSR`, `PersonaPlex`, `HibikiTranslate`, `MADLADTranslation`, `SpeechVAD`, `SpeechWakeWord`, `SpeechEnhancement`, `SpeechRestoration`, `SourceSeparation`, `Qwen3Chat`, `FunctionGemma`, `SpeechCore`, `SpeechUI`, `AudioCommon`. +Các sản phẩm SPM có sẵn: `Qwen3ASR`, `WhisperASR`, `Qwen3TTS`, `Qwen3TTSCoreML`, `ParakeetASR`, `ParakeetStreamingASR`, `NemotronStreamingASR`, `OmnilingualASR`, `KokoroTTS`, `SupertonicTTS`, `VibeVoiceTTS`, `CosyVoiceTTS`, `VoxCPM2TTS`, `ChatterboxTTS`, `OmniVoiceTTS`, `IndicMioTTS`, `FishAudioTTS`, `MagpieTTS`, `MagpieTTSCoreML`, `MAGNeTMusicGen`, `StableAudio3MusicGen`, `FlashSR`, `PersonaPlex`, `Audio2Face3D`, `HibikiTranslate`, `MADLADTranslation`, `SpeechVAD`, `SpeechWakeWord`, `SpeechEnhancement`, `SpeechRestoration`, `SourceSeparation`, `Qwen3Chat`, `FunctionGemma`, `SpeechCore`, `SpeechUI`, `AudioCommon`. ## Mô hình @@ -179,6 +180,7 @@ Xem tổng quan gọn bên dưới. **[Danh mục mô hình đầy đủ với k | [MADLAD-400](https://soniqo.audio/guides/translate) | Văn bản → Văn bản (Dịch) | MLX | 3B | **400+** | | [Hibiki Zero-3B](https://soniqo.audio/guides/audio-translate) | Giọng nói → Giọng nói (Dịch) | MLX | 3B | FR/ES/PT/DE → EN | | [PersonaPlex](https://soniqo.audio/guides/respond) | Giọng nói → Giọng nói | MLX | 7B | EN | +| [Audio2Face-3D](docs/models/audio2face3d.md) | Giọng nói → Hoạt ảnh khuôn mặt | MLX | v2.3 Mark | Không phụ thuộc ngôn ngữ | | [Silero VAD](https://soniqo.audio/guides/vad) | Phát hiện hoạt động giọng nói | MLX, CoreML | 309K | Không phụ thuộc ngôn ngữ | | [KWS Zipformer](docs/models/kws-zipformer.md) | Audio → Wake word | CoreML (ANE) | 3M | EN/custom keywords | | [Pyannote](https://soniqo.audio/guides/diarize) | VAD + Phân tách người nói | MLX | 1.5M | Không phụ thuộc ngôn ngữ | @@ -453,7 +455,7 @@ speech-swift được chia thành một target SPM cho mỗi mô hình để ng **[Sơ đồ kiến trúc đầy đủ với backend, bảng bộ nhớ và bản đồ module → soniqo.audio/architecture](https://soniqo.audio/architecture)** · **[Tài liệu API → soniqo.audio/api](https://soniqo.audio/api)** · **[Benchmark → soniqo.audio/benchmarks](https://soniqo.audio/benchmarks)** Tài liệu cục bộ (kho repo): -- **Mô hình:** [Qwen3-ASR](docs/models/asr-model.md) · [WhisperASR](docs/models/whisper-asr.md) · [Qwen3-TTS](docs/models/tts-model.md) · [CosyVoice](docs/models/cosyvoice-tts.md) · [Kokoro](docs/models/kokoro-tts.md) · [VoxCPM2](docs/models/voxcpm2-tts.md) · [VibeVoice](docs/models/vibevoice.md) · [Supertonic](docs/models/supertonic-tts.md) · [Chatterbox](docs/models/chatterbox-tts.md) · [Indic-Mio](docs/models/indic-mio-tts.md) · [Fish Audio S2 Pro](docs/models/fish-audio-s2-pro.md) · [Magpie TTS](docs/models/magpie-tts.md) · [Parakeet TDT](docs/models/parakeet-asr.md) · [Parakeet Streaming](docs/models/parakeet-streaming-asr.md) · [Nemotron Streaming](docs/models/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/models/omnilingual-asr.md) · [PersonaPlex](docs/models/personaplex.md) · [Hibiki](docs/models/hibiki.md) · [MADLAD-400](docs/models/madlad-translation.md) · [FunctionGemma](docs/models/function-gemma.md) · [Qwen3.5 Chat](docs/models/qwen35-chat.md) · [Gemma 4 Chat](docs/models/gemma4-chat.md) · [Qwen3 Dense Chat](docs/models/qwen3-dense-chat.md) · [FireRedVAD](docs/models/fireredvad.md) · [KWS Zipformer](docs/models/kws-zipformer.md) · [Sidon](docs/models/sidon.md) · [Source Separation](docs/models/source-separation.md) · [HTDemucs](docs/models/htdemucs.md) · [MAGNeT](docs/models/magnet-music-gen.md) · [Stable Audio 3](docs/models/stable-audio-3.md) · [FlashSR](docs/models/flashsr.md) +- **Mô hình:** [Qwen3-ASR](docs/models/asr-model.md) · [WhisperASR](docs/models/whisper-asr.md) · [Qwen3-TTS](docs/models/tts-model.md) · [CosyVoice](docs/models/cosyvoice-tts.md) · [Kokoro](docs/models/kokoro-tts.md) · [VoxCPM2](docs/models/voxcpm2-tts.md) · [VibeVoice](docs/models/vibevoice.md) · [Supertonic](docs/models/supertonic-tts.md) · [Chatterbox](docs/models/chatterbox-tts.md) · [Indic-Mio](docs/models/indic-mio-tts.md) · [Fish Audio S2 Pro](docs/models/fish-audio-s2-pro.md) · [Magpie TTS](docs/models/magpie-tts.md) · [Parakeet TDT](docs/models/parakeet-asr.md) · [Parakeet Streaming](docs/models/parakeet-streaming-asr.md) · [Nemotron Streaming](docs/models/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/models/omnilingual-asr.md) · [PersonaPlex](docs/models/personaplex.md) · [Hibiki](docs/models/hibiki.md) · [MADLAD-400](docs/models/madlad-translation.md) · [FunctionGemma](docs/models/function-gemma.md) · [Qwen3.5 Chat](docs/models/qwen35-chat.md) · [Gemma 4 Chat](docs/models/gemma4-chat.md) · [Qwen3 Dense Chat](docs/models/qwen3-dense-chat.md) · [FireRedVAD](docs/models/fireredvad.md) · [KWS Zipformer](docs/models/kws-zipformer.md) · [Sidon](docs/models/sidon.md) · [Source Separation](docs/models/source-separation.md) · [HTDemucs](docs/models/htdemucs.md) · [MAGNeT](docs/models/magnet-music-gen.md) · [Stable Audio 3](docs/models/stable-audio-3.md) · [FlashSR](docs/models/flashsr.md) · [Audio2Face-3D](docs/models/audio2face3d.md) - **Suy luận:** [Qwen3-ASR](docs/inference/qwen3-asr-inference.md) · [WhisperASR](docs/inference/whisper-asr-inference.md) · [Parakeet TDT](docs/inference/parakeet-asr-inference.md) · [Parakeet Streaming](docs/inference/parakeet-streaming-asr-inference.md) · [Nemotron Streaming](docs/inference/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/inference/omnilingual-asr-inference.md) · [TTS](docs/inference/qwen3-tts-inference.md) · [VoxCPM2](docs/inference/voxcpm2-inference.md) · [VibeVoice](docs/inference/vibevoice-inference.md) · [Fish Audio S2 Pro](docs/inference/fish-audio-s2-pro.md) · [Magpie TTS](docs/inference/magpie-tts.md) · [Hibiki](docs/inference/hibiki-inference.md) · [MADLAD-400](docs/inference/madlad-translation.md) · [MAGNeT](docs/inference/magnet-music-gen.md) · [Stable Audio 3](docs/inference/stable-audio-3.md) · [FlashSR](docs/inference/flashsr.md) · [Forced Aligner](docs/inference/forced-aligner.md) · [Silero VAD](docs/inference/silero-vad.md) · [FireRedVAD](docs/inference/fireredvad.md) · [Wake-word](docs/inference/wake-word.md) · [Speaker Diarization](docs/inference/speaker-diarization.md) · [Speech Enhancement](docs/inference/speech-enhancement.md) · [Sidon](docs/inference/sidon.md) · [Cache/offline](docs/inference/cache-and-offline.md) - **Tài liệu tham khảo:** [Giao thức dùng chung](docs/shared-protocols.md) diff --git a/README_zh.md b/README_zh.md index 3f2ef3a8..307da558 100644 --- a/README_zh.md +++ b/README_zh.md @@ -60,6 +60,7 @@ - **[Hibiki Zero-3B](https://soniqo.audio/guides/audio-translate)** — 流式语音到语音翻译（FR/ES/PT/DE → EN，MLX INT4 + INT8，Kyutai Moshi/Mimi 技术栈，CC-BY-4.0） - **[PersonaPlex](https://soniqo.audio/zh/guides/respond)** — 全双工语音到语音（7B，音频输入 → 音频输出，18 种预设音色） +- **[Audio2Face-3D](docs/models/audio2face3d.md)** — 语音驱动的虚拟形象面部动画（NVIDIA Audio2Face-3D v2.3 Mark，301 个面部系数，MLX） **增强、分离与音频生成** @@ -144,7 +145,7 @@ struct DictateView: View { `SpeechUI` 只提供 `TranscriptionView`（最终结果 + 部分结果）与 `TranscriptionStore`（流式 ASR 适配器）。音频可视化和播放请使用 AVFoundation。 -可用的 SPM products：`Qwen3ASR`, `WhisperASR`, `Qwen3TTS`, `Qwen3TTSCoreML`, `ParakeetASR`, `ParakeetStreamingASR`, `NemotronStreamingASR`, `OmnilingualASR`, `KokoroTTS`, `SupertonicTTS`, `VibeVoiceTTS`, `CosyVoiceTTS`, `VoxCPM2TTS`, `ChatterboxTTS`, `OmniVoiceTTS`, `IndicMioTTS`, `FishAudioTTS`, `MagpieTTS`, `MagpieTTSCoreML`, `MAGNeTMusicGen`, `StableAudio3MusicGen`, `FlashSR`, `PersonaPlex`, `HibikiTranslate`, `MADLADTranslation`, `SpeechVAD`, `SpeechWakeWord`, `SpeechEnhancement`, `SpeechRestoration`, `SourceSeparation`, `Qwen3Chat`, `FunctionGemma`, `SpeechCore`, `SpeechUI`, `AudioCommon`。 +可用的 SPM products：`Qwen3ASR`, `WhisperASR`, `Qwen3TTS`, `Qwen3TTSCoreML`, `ParakeetASR`, `ParakeetStreamingASR`, `NemotronStreamingASR`, `OmnilingualASR`, `KokoroTTS`, `SupertonicTTS`, `VibeVoiceTTS`, `CosyVoiceTTS`, `VoxCPM2TTS`, `ChatterboxTTS`, `OmniVoiceTTS`, `IndicMioTTS`, `FishAudioTTS`, `MagpieTTS`, `MagpieTTSCoreML`, `MAGNeTMusicGen`, `StableAudio3MusicGen`, `FlashSR`, `PersonaPlex`, `Audio2Face3D`, `HibikiTranslate`, `MADLADTranslation`, `SpeechVAD`, `SpeechWakeWord`, `SpeechEnhancement`, `SpeechRestoration`, `SourceSeparation`, `Qwen3Chat`, `FunctionGemma`, `SpeechCore`, `SpeechUI`, `AudioCommon`。 ## 模型 @@ -179,6 +180,7 @@ struct DictateView: View { | [MADLAD-400](https://soniqo.audio/zh/guides/translate) | 文本 → 文本（翻译） | MLX | 3B | **400+** | | [Hibiki Zero-3B](https://soniqo.audio/guides/audio-translate) | 语音 → 语音（翻译） | MLX | 3B | FR/ES/PT/DE → EN | | [PersonaPlex](https://soniqo.audio/zh/guides/respond) | 语音 → 语音 | MLX | 7B | EN | +| [Audio2Face-3D](docs/models/audio2face3d.md) | 语音 → 面部动画 | MLX | v2.3 Mark | 语言无关 | | [Silero VAD](https://soniqo.audio/zh/guides/vad) | 语音活动检测 | MLX、CoreML | 309K | 语言无关 | | [KWS Zipformer](docs/models/kws-zipformer.md) | Audio → Wake word | CoreML (ANE) | 3M | EN/custom keywords | | [Pyannote](https://soniqo.audio/zh/guides/diarize) | VAD + 说话人分离 | MLX | 1.5M | 语言无关 | @@ -455,7 +457,7 @@ speech-swift 把每个模型拆成独立的 SPM target，因此使用者只为 i **[完整架构图（含后端、内存表、模块映射）→ soniqo.audio/architecture](https://soniqo.audio/zh/architecture)** · **[API 参考 → soniqo.audio/api](https://soniqo.audio/zh/api)** · **[基准测试 → soniqo.audio/benchmarks](https://soniqo.audio/zh/benchmarks)** 本地文档（仓库内）： -- **模型：** [Qwen3-ASR](docs/models/asr-model.md) · [WhisperASR](docs/models/whisper-asr.md) · [Qwen3-TTS](docs/models/tts-model.md) · [CosyVoice](docs/models/cosyvoice-tts.md) · [Kokoro](docs/models/kokoro-tts.md) · [VoxCPM2](docs/models/voxcpm2-tts.md) · [VibeVoice](docs/models/vibevoice.md) · [Supertonic](docs/models/supertonic-tts.md) · [Chatterbox](docs/models/chatterbox-tts.md) · [Indic-Mio](docs/models/indic-mio-tts.md) · [Fish Audio S2 Pro](docs/models/fish-audio-s2-pro.md) · [Magpie TTS](docs/models/magpie-tts.md) · [Parakeet TDT](docs/models/parakeet-asr.md) · [Parakeet Streaming](docs/models/parakeet-streaming-asr.md) · [Nemotron Streaming](docs/models/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/models/omnilingual-asr.md) · [PersonaPlex](docs/models/personaplex.md) · [Hibiki](docs/models/hibiki.md) · [MADLAD-400](docs/models/madlad-translation.md) · [FunctionGemma](docs/models/function-gemma.md) · [Qwen3.5 Chat](docs/models/qwen35-chat.md) · [Gemma 4 Chat](docs/models/gemma4-chat.md) · [Qwen3 Dense Chat](docs/models/qwen3-dense-chat.md) · [FireRedVAD](docs/models/fireredvad.md) · [KWS Zipformer](docs/models/kws-zipformer.md) · [Sidon](docs/models/sidon.md) · [Source Separation](docs/models/source-separation.md) · [HTDemucs](docs/models/htdemucs.md) · [MAGNeT](docs/models/magnet-music-gen.md) · [Stable Audio 3](docs/models/stable-audio-3.md) · [FlashSR](docs/models/flashsr.md) +- **模型：** [Qwen3-ASR](docs/models/asr-model.md) · [WhisperASR](docs/models/whisper-asr.md) · [Qwen3-TTS](docs/models/tts-model.md) · [CosyVoice](docs/models/cosyvoice-tts.md) · [Kokoro](docs/models/kokoro-tts.md) · [VoxCPM2](docs/models/voxcpm2-tts.md) · [VibeVoice](docs/models/vibevoice.md) · [Supertonic](docs/models/supertonic-tts.md) · [Chatterbox](docs/models/chatterbox-tts.md) · [Indic-Mio](docs/models/indic-mio-tts.md) · [Fish Audio S2 Pro](docs/models/fish-audio-s2-pro.md) · [Magpie TTS](docs/models/magpie-tts.md) · [Parakeet TDT](docs/models/parakeet-asr.md) · [Parakeet Streaming](docs/models/parakeet-streaming-asr.md) · [Nemotron Streaming](docs/models/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/models/omnilingual-asr.md) · [PersonaPlex](docs/models/personaplex.md) · [Hibiki](docs/models/hibiki.md) · [MADLAD-400](docs/models/madlad-translation.md) · [FunctionGemma](docs/models/function-gemma.md) · [Qwen3.5 Chat](docs/models/qwen35-chat.md) · [Gemma 4 Chat](docs/models/gemma4-chat.md) · [Qwen3 Dense Chat](docs/models/qwen3-dense-chat.md) · [FireRedVAD](docs/models/fireredvad.md) · [KWS Zipformer](docs/models/kws-zipformer.md) · [Sidon](docs/models/sidon.md) · [Source Separation](docs/models/source-separation.md) · [HTDemucs](docs/models/htdemucs.md) · [MAGNeT](docs/models/magnet-music-gen.md) · [Stable Audio 3](docs/models/stable-audio-3.md) · [FlashSR](docs/models/flashsr.md) · [Audio2Face-3D](docs/models/audio2face3d.md) - **推理：** [Qwen3-ASR](docs/inference/qwen3-asr-inference.md) · [WhisperASR](docs/inference/whisper-asr-inference.md) · [Parakeet TDT](docs/inference/parakeet-asr-inference.md) · [Parakeet Streaming](docs/inference/parakeet-streaming-asr-inference.md) · [Nemotron Streaming](docs/inference/nemotron-asr-streaming.md) · [Omnilingual ASR](docs/inference/omnilingual-asr-inference.md) · [TTS](docs/inference/qwen3-tts-inference.md) · [VoxCPM2](docs/inference/voxcpm2-inference.md) · [VibeVoice](docs/inference/vibevoice-inference.md) · [Fish Audio S2 Pro](docs/inference/fish-audio-s2-pro.md) · [Magpie TTS](docs/inference/magpie-tts.md) · [Hibiki](docs/inference/hibiki-inference.md) · [MADLAD-400](docs/inference/madlad-translation.md) · [MAGNeT](docs/inference/magnet-music-gen.md) · [Stable Audio 3](docs/inference/stable-audio-3.md) · [FlashSR](docs/inference/flashsr.md) · [Forced Aligner](docs/inference/forced-aligner.md) · [Silero VAD](docs/inference/silero-vad.md) · [FireRedVAD](docs/inference/fireredvad.md) · [Wake-word](docs/inference/wake-word.md) · [Speaker Diarization](docs/inference/speaker-diarization.md) · [Speech Enhancement](docs/inference/speech-enhancement.md) · [Sidon](docs/inference/sidon.md) · [Cache/offline](docs/inference/cache-and-offline.md) - **参考：** [共享协议](docs/shared-protocols.md) diff --git a/Sources/Audio2Face3D/Audio2Face3DDownloader.swift b/Sources/Audio2Face3D/Audio2Face3DDownloader.swift new file mode 100644 index 00000000..61371679 --- /dev/null +++ b/Sources/Audio2Face3D/Audio2Face3DDownloader.swift @@ -0,0 +1,135 @@ +import Foundation +import AudioCommon + +public enum Audio2Face3DDownloader { + public static let requiredFiles: [String] = [ + "audio2face3d.safetensors", + "network_info.json", + "model_config.json", + "default_emotion.f32", + ] + + /// Exact byte sizes of the published bundles, keyed by model id. Passing + /// sizes skips the per-file HEAD resolution, which fails on small non-LFS + /// files when the CDN omits Content-Length. Unknown model ids fall back + /// to HEAD resolution. + public static let expectedSizesByModelId: [String: [String: Int64]] = [ + Audio2Face3DConfiguration.markModelId: [ + "audio2face3d.safetensors": 73_099_520, + "network_info.json": 1_015, + "model_config.json": 984, + "default_emotion.f32": 104, + ], + Audio2Face3DConfiguration.claireModelId: [ + "audio2face3d.safetensors": 157_778_224, + "network_info.json": 1_017, + "model_config.json": 980, + "default_emotion.f32": 104, + ], + Audio2Face3DConfiguration.jamesModelId: [ + "audio2face3d.safetensors": 157_778_224, + "network_info.json": 1_016, + "model_config.json": 979, + "default_emotion.f32": 104, + ], + ] + + public static func ensureDownloaded( + modelId: String = Audio2Face3DConfiguration.defaultModelId, + to directory: URL, + offlineMode: Bool = false, + progressHandler: ((Double, Int64, Int64, String) -> Void)? = nil + ) async throws { + try await HuggingFaceDownloader.downloadFilesByteWeighted( + modelId: modelId, + to: directory, + files: requiredFiles, + expectedSizes: expectedSizesByModelId[modelId], + offlineMode: offlineMode, + progressHandler: progressHandler) + } + + public static func configuration( + from directory: URL, + fallback: Audio2Face3DConfiguration = Audio2Face3DConfiguration() + ) throws -> Audio2Face3DConfiguration { + let url = directory.appendingPathComponent("network_info.json") + guard FileManager.default.fileExists(atPath: url.path) else { return fallback } + let data = try Data(contentsOf: url) + let info = try JSONDecoder().decode(NetworkInfo.self, from: data) + let modelConfig = try? ModelConfig.from(directory: directory) + return Audio2Face3DConfiguration( + modelId: fallback.modelId, + inputSampleRate: info.audioParams.samplerate, + bufferLength: info.audioParams.bufferLen, + hopLength: info.audioParams.bufferOfs, + framesPerSecond: fallback.framesPerSecond, + implicitEmotionCount: info.params.implicitEmotionLen, + explicitEmotionCount: info.params.explicitEmotions.count, + coefficientLayout: Audio2Face3DCoefficientLayout( + skinCount: info.params.numShapesSkin, + tongueCount: info.params.numShapesTongue, + jawCount: info.params.resultJawSize, + eyeCount: info.params.resultEyesSize), + inputStrength: modelConfig?.config.inputStrength ?? fallback.inputStrength + ) + } +} + +private struct ModelConfig: Decodable { + let config: Config + + struct Config: Decodable { + let inputStrength: Float + + enum CodingKeys: String, CodingKey { + case inputStrength = "input_strength" + } + } + + static func from(directory: URL) throws -> ModelConfig { + let url = directory.appendingPathComponent("model_config.json") + let data = try Data(contentsOf: url) + return try JSONDecoder().decode(ModelConfig.self, from: data) + } +} + +private struct NetworkInfo: Decodable { + let params: Params + let audioParams: AudioParams + + enum CodingKeys: String, CodingKey { + case params + case audioParams = "audio_params" + } + + struct Params: Decodable { + let implicitEmotionLen: Int + let explicitEmotions: [String] + let numShapesSkin: Int + let numShapesTongue: Int + let resultJawSize: Int + let resultEyesSize: Int + + enum CodingKeys: String, CodingKey { + case implicitEmotionLen = "implicit_emotion_len" + case explicitEmotions = "explicit_emotions" + case numShapesSkin = "num_shapes_skin" + case numShapesTongue = "num_shapes_tongue" + case resultJawSize = "result_jaw_size" + case resultEyesSize = "result_eyes_size" + } + } + + struct AudioParams: Decodable { + let bufferLen: Int + let bufferOfs: Int + let samplerate: Int + + enum CodingKeys: String, CodingKey { + case bufferLen = "buffer_len" + case bufferOfs = "buffer_ofs" + case samplerate + } + } +} diff --git a/Sources/Audio2Face3D/Audio2Face3DFrame.swift b/Sources/Audio2Face3D/Audio2Face3DFrame.swift new file mode 100644 index 00000000..5cb485b7 --- /dev/null +++ b/Sources/Audio2Face3D/Audio2Face3DFrame.swift @@ -0,0 +1,50 @@ +import Foundation + +/// One timestamped NVIDIA Audio2Face-3D output frame. +/// +/// The v2.3 Mark model emits 301 coefficients per audio window: +/// skin blendshapes, tongue blendshapes, jaw controls, and eye controls. +public struct Audio2Face3DFrame: Codable, Equatable, Sendable { + public let timeSeconds: Double + public let coefficients: [Float] + public let layout: Audio2Face3DCoefficientLayout + + public init( + timeSeconds: Double, + coefficients: [Float], + layout: Audio2Face3DCoefficientLayout = .nvidiaV23Mark + ) { + self.timeSeconds = timeSeconds + self.coefficients = coefficients + self.layout = layout + } +} + +public struct Audio2Face3DCoefficientLayout: Codable, Equatable, Sendable { + public let skinCount: Int + public let tongueCount: Int + public let jawCount: Int + public let eyeCount: Int + + public static let nvidiaV23Mark = Audio2Face3DCoefficientLayout( + skinCount: 272, + tongueCount: 10, + jawCount: 15, + eyeCount: 4) + + public init(skinCount: Int, tongueCount: Int, jawCount: Int, eyeCount: Int) { + self.skinCount = skinCount + self.tongueCount = tongueCount + self.jawCount = jawCount + self.eyeCount = eyeCount + } + + public var coefficientCount: Int { + skinCount + tongueCount + jawCount + eyeCount + } + + public var skinRange: Range { 0 ..< skinCount } + public var tongueRange: Range { skinRange.upperBound ..< skinRange.upperBound + tongueCount } + public var jawRange: Range { tongueRange.upperBound ..< tongueRange.upperBound + jawCount } + public var eyeRange: Range { jawRange.upperBound ..< jawRange.upperBound + eyeCount } +} diff --git a/Sources/Audio2Face3D/Audio2Face3DMLXRuntime.swift b/Sources/Audio2Face3D/Audio2Face3DMLXRuntime.swift new file mode 100644 index 00000000..bf288070 --- /dev/null +++ b/Sources/Audio2Face3D/Audio2Face3DMLXRuntime.swift @@ -0,0 +1,696 @@ +import Foundation +import MLX + +final class Audio2Face3DMLXRuntime { + private let configuration: Audio2Face3DConfiguration + private let weights: [String: MLXArray] + private let defaultEmotion: [Float] + private let graphTensors: GraphTensors + private let transformerCombineWeights: [Float] + + private static let sqrt2Scalar = Float(1.4142135623730951) + private static let halfScalar = Float(0.5) + private static let oneScalar = Float(1.0) + private static let epsScalar = Float(1e-5) + private static let autocorrScaleScalar = Float(1.0 / 640.0) + + private static let autocorrWindow: [Float] = { + let count = 640 + return (0.. [Audio2Face3DFrame] { + guard sampleRate == configuration.inputSampleRate else { + throw Audio2Face3DError.invalidSampleRate(sampleRate) + } + + let hop = hopLength ?? configuration.frameSampleCount + guard hop > 0 else { + throw Audio2Face3DError.invalidAudioWindow(expected: 1, got: hop) + } + var frames: [Audio2Face3DFrame] = [] + var target = 0 + repeat { + let start = target - configuration.hopLength + var window = [Float](repeating: 0, count: configuration.bufferLength) + for index in 0..= 0 && sampleIndex < audio.count { + window[index] = audio[sampleIndex] + } + } + let coefficients = try coefficients(forWindow: window, emotion: emotion) + frames.append(Audio2Face3DFrame( + timeSeconds: Double(target) / Double(configuration.inputSampleRate), + coefficients: coefficients, + layout: configuration.coefficientLayout)) + target += hop + } while target < max(audio.count, 1) + + return frames + } + + func emotionVector(explicit: [Float]) throws -> [Float] { + guard explicit.count == configuration.explicitEmotionCount else { + throw Audio2Face3DError.invalidEmotionVector( + expected: configuration.explicitEmotionCount, + got: explicit.count) + } + var vector = defaultEmotion + let start = configuration.implicitEmotionCount + for index in 0.. [Float] { + guard audioWindow.count == configuration.bufferLength else { + throw Audio2Face3DError.invalidAudioWindow( + expected: configuration.bufferLength, + got: audioWindow.count) + } + + let emotionVector = emotion ?? defaultEmotion + guard emotionVector.count == configuration.emotionVectorLength else { + throw Audio2Face3DError.invalidEmotionVector( + expected: configuration.emotionVectorLength, + got: emotionVector.count) + } + + let scaledWindow = audioWindow.map { $0 * configuration.inputStrength } + let input = MLXArray(scaledWindow, [1, 1, configuration.bufferLength]) + let emotionInput = MLXArray(emotionVector, [1, 1, configuration.emotionVectorLength]) + let output = forward(input: input, emotion: emotionInput) + let flat = output.asType(.float32).reshaped([output.size]).asArray(Float.self) + guard flat.count == configuration.outputCoefficientCount else { + throw Audio2Face3DError.invalidModelOutput( + expected: configuration.outputCoefficientCount, + got: flat.count) + } + return flat + } + + private func forward(input: MLXArray, emotion: MLXArray) -> MLXArray { + let implicitEmotion = emotion[0..., 0..., 0.. MLXArray { + var x = input.reshaped([input.dim(0) * input.dim(1), input.dim(2)]) + .expandedDimensions(axis: 1) + + x = conv1dNCL( + x, + weight: tensor("wave2vec_model.feature_extractor.conv_layers.0.conv.weight"), + stride: 5) + x = instanceNorm1d( + x, + weight: tensor(graphTensors.featureExtractorScale), + bias: tensor(graphTensors.featureExtractorBias)) + x = gelu(x) + + for layer in 1...6 { + let stride = 2 + x = conv1dNCL( + x, + weight: tensor("wave2vec_model.feature_extractor.conv_layers.\(layer).conv.weight"), + stride: stride) + x = gelu(x) + } + + x = x.transposed(0, 2, 1) + x = layerNorm( + x, + weight: tensor("wave2vec_model.encoder.feature_projection.layer_norm.weight"), + bias: tensor("wave2vec_model.encoder.feature_projection.layer_norm.bias")) + x = linearMatMul( + x, + weight: tensor(graphTensors.featureProjection), + bias: tensor("wave2vec_model.encoder.feature_projection.projection.bias")) + + var pos = conv1dNCL( + x.transposed(0, 2, 1), + weight: tensor(graphTensors.positionConv), + bias: tensor("wave2vec_model.encoder.transformer.pos_conv_embed.conv.bias"), + padding: 64, + groups: 16) + pos = pos[0..., 0..., 0.. MLXArray { + var x = input + let batch = x.dim(0) + let time = x.dim(1) + let hidden = x.dim(2) + let heads = 12 + let headDim = hidden / heads + + let q = linearMatMul( + x, + weight: tensor(tensors.queryProjection), + bias: tensor("wave2vec_model.encoder.transformer.layers.\(tensors.index).attention.q_proj.bias")) + .reshaped([batch, time, heads, headDim]) + .transposed(0, 2, 1, 3) + let k = linearMatMul( + x, + weight: tensor(tensors.keyProjection), + bias: tensor("wave2vec_model.encoder.transformer.layers.\(tensors.index).attention.k_proj.bias")) + .reshaped([batch, time, heads, headDim]) + .transposed(0, 2, 3, 1) + let v = linearMatMul( + x, + weight: tensor(tensors.valueProjection), + bias: tensor("wave2vec_model.encoder.transformer.layers.\(tensors.index).attention.v_proj.bias")) + .reshaped([batch, time, heads, headDim]) + .transposed(0, 2, 1, 3) + + var scores = matmul(q * MLXArray(Float(1.0 / Foundation.sqrt(Double(headDim)))), k) + scores = scores - MLX.max(scores, axis: -1, keepDims: true) + let attention = softmax(scores, axis: -1) + let context = matmul(attention, v) + .transposed(0, 2, 1, 3) + .reshaped([batch, time, hidden]) + let attentionOut = linearMatMul( + context, + weight: tensor(tensors.outputProjection), + bias: tensor("wave2vec_model.encoder.transformer.layers.\(tensors.index).attention.out_proj.bias")) + + x = x + attentionOut + x = layerNorm( + x, + weight: tensor("wave2vec_model.encoder.transformer.layers.\(tensors.index).layer_norm.weight"), + bias: tensor("wave2vec_model.encoder.transformer.layers.\(tensors.index).layer_norm.bias")) + + let ff = gelu(linearMatMul( + x, + weight: tensor(tensors.feedForwardIntermediate), + bias: tensor("wave2vec_model.encoder.transformer.layers.\(tensors.index).feed_forward.intermediate_dense.bias"))) + let ffOut = linearMatMul( + ff, + weight: tensor(tensors.feedForwardOutput), + bias: tensor("wave2vec_model.encoder.transformer.layers.\(tensors.index).feed_forward.output_dense.bias")) + x = x + ffOut + + return layerNorm( + x, + weight: tensor("wave2vec_model.encoder.transformer.layers.\(tensors.index).final_layer_norm.weight"), + bias: tensor("wave2vec_model.encoder.transformer.layers.\(tensors.index).final_layer_norm.bias")) + } + + private func combineTransformerOutputs(_ outputs: [MLXArray]) -> MLXArray { + precondition(!outputs.isEmpty, "Audio2Face3D graph must contain at least one transformer layer") + guard outputs.count > 1 else { return outputs[0] } + precondition(transformerCombineWeights.count == outputs.count, "Audio2Face3D combine_weights does not match transformer layer count") + let total = transformerCombineWeights.reduce(Float(0), +) + precondition(total != 0, "Audio2Face3D combine_weights sum must be non-zero") + var combined = outputs[0] * scalar(transformerCombineWeights[0] / total) + for index in 1.. MLXArray { + let scaled = input * tensor("in_scale_autocorr") + let hann = MLXArray(Self.autocorrWindow, [1, 1, 1, 640]) + + var windows: [MLXArray] = [] + windows.reserveCapacity(25) + for index in 0..<25 { + let start = index * 320 + let window = scaled[0..., 0..., start..<(start + 640)] + .expandedDimensions(axis: 2) + windows.append(window) + } + let framed = concatenated(windows, axis: 2) + let centered = (framed - MLX.mean(framed, axis: -1, keepDims: true)) * hann + + var correlations: [MLXArray] = [] + correlations.reserveCapacity(32) + for lag in 0..<32 { + let lhs: MLXArray + let rhs: MLXArray + if lag == 0 { + lhs = centered + rhs = centered + } else { + lhs = centered[0..., 0..., 0..., 0..<(640 - lag)] + rhs = centered[0..., 0..., 0..., lag..<640] + } + correlations.append(MLX.sum(lhs * rhs, axis: -1).expandedDimensions(axis: -1)) + } + + var x = concatenated(correlations, axis: -1) * scalar(Self.autocorrScaleScalar) + x = relu(conv2dNCHW(x, weight: tensor("freq1.weight"), bias: tensor("freq1.bias"), stride: (1, 2), padding: (0, 1))) * scalar(Self.sqrt2Scalar) + x = relu(conv2dNCHW(x, weight: tensor("freq2.weight"), bias: tensor("freq2.bias"), stride: (1, 2), padding: (0, 1))) * scalar(Self.sqrt2Scalar) + x = relu(conv2dNCHW(x, weight: tensor("freq3.weight"), bias: tensor("freq3.bias"), stride: (1, 2), padding: (0, 1))) * scalar(Self.sqrt2Scalar) + x = relu(conv2dNCHW(x, weight: tensor("freq4.weight"), bias: tensor("freq4.bias"), stride: (1, 2), padding: (0, 1))) * scalar(Self.sqrt2Scalar) + x = relu(conv2dNCHW(x, weight: tensor("freq5.weight"), bias: tensor("freq5.bias"), stride: (1, 1), padding: (0, 0))) * scalar(Self.sqrt2Scalar) + return x + } + + private func appendEmotion( + to x: MLXArray, + implicitEmotion: MLXArray, + projectedEmotion: MLXArray + ) -> MLXArray { + let batch = x.dim(0) + let height = x.dim(2) + let width = x.dim(3) + let implicit = MLX.broadcast( + implicitEmotion.reshaped([batch, configuration.implicitEmotionCount, 1, 1]), + to: [batch, configuration.implicitEmotionCount, height, width]) + let projected = MLX.broadcast( + projectedEmotion.reshaped([batch, 8, 1, 1]), + to: [batch, 8, height, width]) + return concatenated([x, implicit, projected], axis: 1) + } + + private func conv1dNCL( + _ x: MLXArray, + weight: MLXArray, + bias: MLXArray? = nil, + stride: Int = 1, + padding: Int = 0, + groups: Int = 1 + ) -> MLXArray { + let weightNLC = weight.transposed(0, 2, 1) + var y = MLX.conv1d( + x.transposed(0, 2, 1), + weightNLC, + stride: stride, + padding: padding, + groups: groups) + .transposed(0, 2, 1) + if let bias { + y = y + bias.reshaped([1, bias.dim(0), 1]) + } + return y + } + + private func conv2dNCHW( + _ x: MLXArray, + weight: MLXArray, + bias: MLXArray, + stride: (Int, Int), + padding: (Int, Int) + ) -> MLXArray { + let weightNHWC = weight.transposed(0, 2, 3, 1) + var y = MLX.conv2d( + x.transposed(0, 2, 3, 1), + weightNHWC, + stride: .init(stride), + padding: .init(padding)) + .transposed(0, 3, 1, 2) + y = y + bias.reshaped([1, bias.dim(0), 1, 1]) + return y + } + + private func instanceNorm1d(_ x: MLXArray, weight: MLXArray, bias: MLXArray) -> MLXArray { + let mean = MLX.mean(x, axis: -1, keepDims: true) + let centered = x - mean + let variance = MLX.mean(centered * centered, axis: -1, keepDims: true) + return centered / MLX.sqrt(variance + scalar(Self.epsScalar)) * weight.reshaped([1, weight.dim(0), 1]) + bias.reshaped([1, bias.dim(0), 1]) + } + + private func layerNorm(_ x: MLXArray, weight: MLXArray, bias: MLXArray) -> MLXArray { + let mean = MLX.mean(x, axis: -1, keepDims: true) + let centered = x - mean + let variance = MLX.mean(centered * centered, axis: -1, keepDims: true) + return centered / MLX.sqrt(variance + scalar(Self.epsScalar)) * weight + bias + } + + private func gelu(_ x: MLXArray) -> MLXArray { + x * (scalar(Self.oneScalar) + MLX.erf(x / scalar(Self.sqrt2Scalar))) * scalar(Self.halfScalar) + } + + private func relu(_ x: MLXArray) -> MLXArray { + MLX.maximum(x, MLXArray(Float(0))) + } + + private func scalar(_ value: Float) -> MLXArray { + MLXArray(value) + } + + private func linearMatMul(_ x: MLXArray, weight: MLXArray, bias: MLXArray) -> MLXArray { + matmul(x, weight) + bias + } + + private func gemm(_ x: MLXArray, weight: MLXArray, bias: MLXArray) -> MLXArray { + matmul(x, weight.transposed(1, 0)) + bias + } + + private func tensor(_ key: String) -> MLXArray { + guard let value = weights[key] else { + preconditionFailure("Missing validated Audio2Face3D tensor \(key)") + } + return value + } + + private static func readDefaultEmotion(from directory: URL, expectedCount: Int) -> [Float]? { + let url = directory.appendingPathComponent("default_emotion.f32") + guard + expectedCount > 0, + let data = try? Data(contentsOf: url), + data.count == expectedCount * MemoryLayout.size + else { + return nil + } + + return data.withUnsafeBytes { rawBuffer in + let bytes = rawBuffer.bindMemory(to: UInt8.self) + return (0...size + let bits = + UInt32(bytes[offset]) | + (UInt32(bytes[offset + 1]) << 8) | + (UInt32(bytes[offset + 2]) << 16) | + (UInt32(bytes[offset + 3]) << 24) + return Float(bitPattern: bits) + } + } + } + + private func validateRequiredTensors() throws { + var required = [ + "audio_feature_map.bias", + "combine_weights", + "emo_linear1.bias", + "freq1.bias", + "freq1.weight", + "freq2.bias", + "freq2.weight", + "freq3.bias", + "freq3.weight", + "freq4.bias", + "freq4.weight", + "freq5.bias", + "freq5.weight", + "in_scale_autocorr", + "out_mapping.bias", + "out_mapping.weight", + "out_mapping1.bias", + "out_mapping1.weight", + "out_mapping_tongue.bias", + "out_mapping_tongue.weight", + "out_mapping1_tongue.bias", + "out_mapping1_tongue.weight", + "time1.bias", + "time1.weight", + "time2.bias", + "time2.weight", + "time3.bias", + "time3.weight", + "time4.bias", + "time4.weight", + "time5.bias", + "time5.weight", + "wave2vec_model.encoder.feature_projection.layer_norm.bias", + "wave2vec_model.encoder.feature_projection.layer_norm.weight", + "wave2vec_model.encoder.feature_projection.projection.bias", + "wave2vec_model.encoder.transformer.layer_norm.bias", + "wave2vec_model.encoder.transformer.layer_norm.weight", + "wave2vec_model.encoder.transformer.pos_conv_embed.conv.bias", + "wave2vec_model.feature_extractor.conv_layers.0.conv.weight", + "wave2vec_model.feature_extractor.conv_layers.1.conv.weight", + "wave2vec_model.feature_extractor.conv_layers.2.conv.weight", + "wave2vec_model.feature_extractor.conv_layers.3.conv.weight", + "wave2vec_model.feature_extractor.conv_layers.4.conv.weight", + "wave2vec_model.feature_extractor.conv_layers.5.conv.weight", + "wave2vec_model.feature_extractor.conv_layers.6.conv.weight" + ] + required.append(contentsOf: [ + graphTensors.audioFeatureMap, + graphTensors.emotionProjection, + graphTensors.featureExtractorBias, + graphTensors.featureExtractorScale, + graphTensors.featureProjection, + graphTensors.positionConv + ]) + for layer in graphTensors.transformerLayers { + required.append(contentsOf: [ + layer.queryProjection, + layer.keyProjection, + layer.valueProjection, + layer.outputProjection, + layer.feedForwardIntermediate, + layer.feedForwardOutput, + "wave2vec_model.encoder.transformer.layers.\(layer.index).attention.k_proj.bias", + "wave2vec_model.encoder.transformer.layers.\(layer.index).attention.out_proj.bias", + "wave2vec_model.encoder.transformer.layers.\(layer.index).attention.q_proj.bias", + "wave2vec_model.encoder.transformer.layers.\(layer.index).attention.v_proj.bias", + "wave2vec_model.encoder.transformer.layers.\(layer.index).feed_forward.intermediate_dense.bias", + "wave2vec_model.encoder.transformer.layers.\(layer.index).feed_forward.output_dense.bias", + "wave2vec_model.encoder.transformer.layers.\(layer.index).final_layer_norm.bias", + "wave2vec_model.encoder.transformer.layers.\(layer.index).final_layer_norm.weight", + "wave2vec_model.encoder.transformer.layers.\(layer.index).layer_norm.bias", + "wave2vec_model.encoder.transformer.layers.\(layer.index).layer_norm.weight" + ]) + } + + for key in required where weights[key] == nil { + throw Audio2Face3DError.missingTensor(key) + } + if transformerCombineWeights.count != graphTensors.transformerLayers.count { + throw Audio2Face3DError.missingTensor("combine_weights[\(graphTensors.transformerLayers.count)]") + } + } + + private static func resolveGraphTensors(weights: [String: MLXArray]) throws -> GraphTensors { + if weights["onnx::MatMul_1738"] != nil { + return GraphTensors( + emotionProjection: "onnx::MatMul_1738", + featureExtractorScale: "onnx::Mul_1742", + featureExtractorBias: "onnx::Add_1743", + featureProjection: "onnx::MatMul_1744", + positionConv: "onnx::Conv_1747", + audioFeatureMap: "onnx::MatMul_1760", + transformerLayers: [ + TransformerLayerTensors( + index: 0, + queryProjection: "onnx::MatMul_1748", + keyProjection: "onnx::MatMul_1755", + valueProjection: "onnx::MatMul_1756", + outputProjection: "onnx::MatMul_1757", + feedForwardIntermediate: "onnx::MatMul_1758", + feedForwardOutput: "onnx::MatMul_1759") + ]) + } + + if weights["2230"] != nil { + return GraphTensors( + emotionProjection: "2230", + featureExtractorScale: "2235", + featureExtractorBias: "2236", + featureProjection: "2237", + positionConv: "2240", + audioFeatureMap: "2289", + transformerLayers: [ + TransformerLayerTensors( + index: 0, + queryProjection: "2241", + keyProjection: "2244", + valueProjection: "2247", + outputProjection: "2250", + feedForwardIntermediate: "2251", + feedForwardOutput: "2252"), + TransformerLayerTensors( + index: 1, + queryProjection: "2253", + keyProjection: "2256", + valueProjection: "2259", + outputProjection: "2262", + feedForwardIntermediate: "2263", + feedForwardOutput: "2264"), + TransformerLayerTensors( + index: 2, + queryProjection: "2265", + keyProjection: "2268", + valueProjection: "2271", + outputProjection: "2274", + feedForwardIntermediate: "2275", + feedForwardOutput: "2276"), + TransformerLayerTensors( + index: 3, + queryProjection: "2277", + keyProjection: "2280", + valueProjection: "2283", + outputProjection: "2286", + feedForwardIntermediate: "2287", + feedForwardOutput: "2288") + ]) + } + + if weights["2232"] != nil { + return GraphTensors( + emotionProjection: "2232", + featureExtractorScale: "2237", + featureExtractorBias: "2238", + featureProjection: "2239", + positionConv: "2242", + audioFeatureMap: "2291", + transformerLayers: [ + TransformerLayerTensors( + index: 0, + queryProjection: "2243", + keyProjection: "2246", + valueProjection: "2249", + outputProjection: "2252", + feedForwardIntermediate: "2253", + feedForwardOutput: "2254"), + TransformerLayerTensors( + index: 1, + queryProjection: "2255", + keyProjection: "2258", + valueProjection: "2261", + outputProjection: "2264", + feedForwardIntermediate: "2265", + feedForwardOutput: "2266"), + TransformerLayerTensors( + index: 2, + queryProjection: "2267", + keyProjection: "2270", + valueProjection: "2273", + outputProjection: "2276", + feedForwardIntermediate: "2277", + feedForwardOutput: "2278"), + TransformerLayerTensors( + index: 3, + queryProjection: "2279", + keyProjection: "2282", + valueProjection: "2285", + outputProjection: "2288", + feedForwardIntermediate: "2289", + feedForwardOutput: "2290") + ]) + } + + throw Audio2Face3DError.missingTensor("known Audio2Face3D graph tensor set") + } +} + +private struct GraphTensors { + let emotionProjection: String + let featureExtractorScale: String + let featureExtractorBias: String + let featureProjection: String + let positionConv: String + let audioFeatureMap: String + let transformerLayers: [TransformerLayerTensors] +} + +private struct TransformerLayerTensors { + let index: Int + let queryProjection: String + let keyProjection: String + let valueProjection: String + let outputProjection: String + let feedForwardIntermediate: String + let feedForwardOutput: String +} diff --git a/Sources/Audio2Face3D/Audio2Face3DModel.swift b/Sources/Audio2Face3D/Audio2Face3DModel.swift new file mode 100644 index 00000000..3326fcc2 --- /dev/null +++ b/Sources/Audio2Face3D/Audio2Face3DModel.swift @@ -0,0 +1,101 @@ +import Foundation +import AudioCommon + +public final class Audio2Face3DModel: @unchecked Sendable { + public let backend: Audio2Face3DBackend + public let configuration: Audio2Face3DConfiguration + + private let mlxRuntime: Audio2Face3DMLXRuntime? + + private init( + backend: Audio2Face3DBackend, + configuration: Audio2Face3DConfiguration, + mlxRuntime: Audio2Face3DMLXRuntime? + ) { + self.backend = backend + self.configuration = configuration + self.mlxRuntime = mlxRuntime + } + + /// Load a model for avatar motion generation. + /// + /// This loads the exported NVIDIA Audio2Face-3D tensors and runs the + /// hand-written MLX graph. No heuristic/fallback avatar motion is exposed + /// from this API. + public static func fromLocal( + directory: URL, + backend: Audio2Face3DBackend = .mlx, + configuration: Audio2Face3DConfiguration = Audio2Face3DConfiguration() + ) throws -> Audio2Face3DModel { + let resolvedConfiguration = try Audio2Face3DDownloader.configuration( + from: directory, + fallback: configuration) + switch backend { + case .mlx: + let weights = directory.appendingPathComponent("audio2face3d.safetensors") + guard FileManager.default.fileExists(atPath: weights.path) else { + throw Audio2Face3DError.missingExportedWeights(weights.path) + } + return Audio2Face3DModel( + backend: backend, + configuration: resolvedConfiguration, + mlxRuntime: try Audio2Face3DMLXRuntime( + directory: directory, + configuration: resolvedConfiguration)) + } + } + + /// Download the exported MLX bundle from Hugging Face into the normal + /// speech cache and load it. + public static func fromPretrained( + modelId: String = Audio2Face3DConfiguration.defaultModelId, + backend: Audio2Face3DBackend = .mlx, + cacheDir: URL? = nil, + offlineMode: Bool = false, + progressHandler: ((Double, String) -> Void)? = nil + ) async throws -> Audio2Face3DModel { + var configuration = Audio2Face3DConfiguration(modelId: modelId) + let dir = try cacheDir ?? HuggingFaceDownloader.getCacheDirectory(for: modelId) + progressHandler?(0.0, "Downloading Audio2Face-3D...") + try await Audio2Face3DDownloader.ensureDownloaded( + modelId: modelId, + to: dir, + offlineMode: offlineMode + ) { progress, _, _, file in + progressHandler?(progress * 0.90, file.isEmpty ? "Downloading Audio2Face-3D..." : "Downloading \(file)...") + } + configuration = try Audio2Face3DDownloader.configuration(from: dir, fallback: configuration) + progressHandler?(0.95, "Loading Audio2Face-3D...") + return try fromLocal(directory: dir, backend: backend, configuration: configuration) + } + + public func frames( + for audio: [Float], + sampleRate: Int, + hopLength: Int? = nil, + emotion: [Float]? = nil + ) throws -> [Audio2Face3DFrame] { + guard sampleRate > 0 else { throw Audio2Face3DError.invalidSampleRate(sampleRate) } + switch backend { + case .mlx: + guard let mlxRuntime else { + throw Audio2Face3DError.unsupportedBackend(backend) + } + return try mlxRuntime.frames( + for: audio, + sampleRate: sampleRate, + hopLength: hopLength, + emotion: emotion) + } + } + + public func emotionVector(explicit: [Float]) throws -> [Float] { + switch backend { + case .mlx: + guard let mlxRuntime else { + throw Audio2Face3DError.unsupportedBackend(backend) + } + return try mlxRuntime.emotionVector(explicit: explicit) + } + } +} diff --git a/Sources/Audio2Face3D/Configuration.swift b/Sources/Audio2Face3D/Configuration.swift new file mode 100644 index 00000000..06d58f17 --- /dev/null +++ b/Sources/Audio2Face3D/Configuration.swift @@ -0,0 +1,91 @@ +import Foundation + +public enum Audio2Face3DBackend: String, Codable, Sendable { + /// Full NVIDIA Audio2Face-3D v2.3 graph exported into MLX tensors. The + /// export script lands the weights; graph execution is a hand-written MLX + /// port and must match the ONNX parity fixtures. + case mlx +} + +public struct Audio2Face3DConfiguration: Codable, Equatable, Sendable { + /// Published MLX bundles, one per NVIDIA identity. Coefficients are + /// identity-specific (Mark: 272 skin, Claire/James: 140 skin), so + /// renderers need the matching rig or retarget projection. + public static let markModelId = "aufklarer/Audio2Face-3D-v2.3-Mark-MLX" + public static let claireModelId = "aufklarer/Audio2Face-3D-v2.3.1-Claire-MLX" + public static let jamesModelId = "aufklarer/Audio2Face-3D-v2.3.1-James-MLX" + public static let defaultModelId = jamesModelId + + public var modelId: String + public var inputSampleRate: Int + public var bufferLength: Int + public var hopLength: Int + public var framesPerSecond: Double + public var implicitEmotionCount: Int + public var explicitEmotionCount: Int + public var coefficientLayout: Audio2Face3DCoefficientLayout + public var inputStrength: Float + + public init( + modelId: String = Self.defaultModelId, + inputSampleRate: Int = 16_000, + bufferLength: Int = 8_320, + hopLength: Int = 4_160, + framesPerSecond: Double = 30, + implicitEmotionCount: Int = 16, + explicitEmotionCount: Int = 10, + coefficientLayout: Audio2Face3DCoefficientLayout = .nvidiaV23Mark, + inputStrength: Float = 1.3 + ) { + self.modelId = modelId + self.inputSampleRate = inputSampleRate + self.bufferLength = bufferLength + self.hopLength = hopLength + self.framesPerSecond = framesPerSecond + self.implicitEmotionCount = implicitEmotionCount + self.explicitEmotionCount = explicitEmotionCount + self.coefficientLayout = coefficientLayout + self.inputStrength = inputStrength + } + + public var emotionVectorLength: Int { + explicitEmotionCount + implicitEmotionCount + } + + public var frameSampleCount: Int { + max(1, Int((Double(inputSampleRate) / framesPerSecond).rounded())) + } + + public var outputCoefficientCount: Int { + coefficientLayout.coefficientCount + } +} + +public enum Audio2Face3DError: Error, LocalizedError, Equatable { + case invalidSampleRate(Int) + case invalidAudioWindow(expected: Int, got: Int) + case invalidEmotionVector(expected: Int, got: Int) + case invalidModelOutput(expected: Int, got: Int) + case missingTensor(String) + case unsupportedBackend(Audio2Face3DBackend) + case missingExportedWeights(String) + + public var errorDescription: String? { + switch self { + case .invalidSampleRate(let sampleRate): + return "Invalid sample rate: \(sampleRate)" + case .invalidAudioWindow(let expected, let got): + return "Invalid Audio2Face3D audio window: expected \(expected) samples, got \(got)" + case .invalidEmotionVector(let expected, let got): + return "Invalid Audio2Face3D emotion vector: expected \(expected) values, got \(got)" + case .invalidModelOutput(let expected, let got): + return "Invalid Audio2Face3D model output: expected \(expected) coefficients, got \(got)" + case .missingTensor(let key): + return "Missing Audio2Face3D tensor '\(key)' in exported MLX weights" + case .unsupportedBackend(let backend): + return "Unsupported Audio2Face3D backend '\(backend.rawValue)'" + case .missingExportedWeights(let path): + return "Missing exported Audio2Face3D MLX weights at \(path)" + } + } +} diff --git a/Sources/AudioCLILib/AudioCLI.swift b/Sources/AudioCLILib/AudioCLI.swift index 13bf71fd..096c8aed 100644 --- a/Sources/AudioCLILib/AudioCLI.swift +++ b/Sources/AudioCLILib/AudioCLI.swift @@ -27,6 +27,7 @@ public struct AudioCLI: ParsableCommand { VibeVoiceEncodeCommand.self, TranslateCommand.self, WakeCommand.self, + AvatarMotionCommand.self, ] ) diff --git a/Sources/AudioCLILib/AvatarMotionCommand.swift b/Sources/AudioCLILib/AvatarMotionCommand.swift new file mode 100644 index 00000000..92a34b99 --- /dev/null +++ b/Sources/AudioCLILib/AvatarMotionCommand.swift @@ -0,0 +1,76 @@ +import Foundation +import ArgumentParser +import AudioCommon +import Audio2Face3D + +public struct AvatarMotionCommand: ParsableCommand { + public static let configuration = CommandConfiguration( + commandName: "avatar-motion", + abstract: "Generate NVIDIA Audio2Face-3D coefficient frames from speech audio" + ) + + @Argument(help: "Input audio file (WAV, m4a, etc.; resampled to 16 kHz)") + public var input: String + + @Option(name: .shortAndLong, help: "Output JSONL file path") + public var output: String = "avatar-motion.jsonl" + + @Option(name: .long, help: "Backend: mlx") + public var backend: Audio2Face3DBackend = .mlx + + @Option(name: .long, help: "Hugging Face model ID for the exported MLX bundle") + public var model: String = Audio2Face3DConfiguration.defaultModelId + + @Option(name: .long, help: "Local exported model directory for --backend mlx") + public var modelDir: String? + + @Flag(name: .long, help: "Print timing and frame count") + public var verbose: Bool = false + + public init() {} + + public func run() throws { + try runAsync { + let inputURL = URL(fileURLWithPath: input) + let audio = try AudioFileLoader.load( + url: inputURL, + targetSampleRate: 16_000, + quality: .standard) + + let motionModel: Audio2Face3DModel + if let modelDir { + motionModel = try Audio2Face3DModel.fromLocal( + directory: URL(fileURLWithPath: modelDir, isDirectory: true), + backend: backend) + } else { + motionModel = try await Audio2Face3DModel.fromPretrained( + modelId: model, + backend: backend, + progressHandler: reportProgress) + } + + let start = CFAbsoluteTimeGetCurrent() + let frames = try motionModel.frames(for: audio, sampleRate: 16_000) + let elapsed = CFAbsoluteTimeGetCurrent() - start + + let encoder = JSONEncoder() + encoder.outputFormatting = [.sortedKeys] + let lines = try frames.map { frame -> String in + let data = try encoder.encode(frame) + return String(decoding: data, as: UTF8.self) + }.joined(separator: "\n") + + let outputURL = URL(fileURLWithPath: output) + try (lines + "\n").write(to: outputURL, atomically: true, encoding: .utf8) + + if verbose { + let duration = Double(audio.count) / 16_000.0 + print(String(format: "Generated %d coefficient frames for %.2fs audio in %.3fs (%.1f fps)", + frames.count, duration, elapsed, Double(frames.count) / max(elapsed, 0.001))) + } + print("Saved avatar motion to \(output)") + } + } +} + +extension Audio2Face3DBackend: ExpressibleByArgument {} diff --git a/Tests/Audio2Face3DTests/Audio2Face3DParityTests.swift b/Tests/Audio2Face3DTests/Audio2Face3DParityTests.swift new file mode 100644 index 00000000..081e6292 --- /dev/null +++ b/Tests/Audio2Face3DTests/Audio2Face3DParityTests.swift @@ -0,0 +1,39 @@ +import XCTest +@testable import Audio2Face3D + +final class Audio2Face3DParityTests: XCTestCase { + func testMLXForwardMatchesONNXFixtureWhenAvailable() throws { + guard let fixtureDir = ProcessInfo.processInfo.environment["AUDIO2FACE3D_FIXTURE_DIR"] else { + throw XCTSkip("set AUDIO2FACE3D_FIXTURE_DIR to an exported Audio2Face3D MLX bundle") + } + + let dir = URL(fileURLWithPath: fixtureDir, isDirectory: true) + let input = try readFloat32(dir.appendingPathComponent("parity_input.f32")) + let emotion = try readFloat32(dir.appendingPathComponent("parity_emotion.f32")) + let expected = try readFloat32(dir.appendingPathComponent("parity_result.f32")) + + let configuration = try Audio2Face3DDownloader.configuration(from: dir) + let runtime = try Audio2Face3DMLXRuntime(directory: dir, configuration: configuration) + let actual = try runtime.coefficients(forWindow: input, emotion: emotion) + + XCTAssertEqual(actual.count, expected.count) + var maxAbs: Float = 0 + var sumAbs: Float = 0 + for (lhs, rhs) in zip(actual, expected) { + let diff = abs(lhs - rhs) + maxAbs = max(maxAbs, diff) + sumAbs += diff + } + let meanAbs = sumAbs / Float(expected.count) + XCTAssertLessThan(maxAbs, 0.05) + XCTAssertLessThan(meanAbs, 0.005) + } + + private func readFloat32(_ url: URL) throws -> [Float] { + let data = try Data(contentsOf: url) + precondition(data.count % MemoryLayout.stride == 0) + return data.withUnsafeBytes { rawBuffer in + Array(rawBuffer.bindMemory(to: Float.self)) + } + } +} diff --git a/Tests/Audio2Face3DTests/Audio2Face3DTests.swift b/Tests/Audio2Face3DTests/Audio2Face3DTests.swift new file mode 100644 index 00000000..d4620215 --- /dev/null +++ b/Tests/Audio2Face3DTests/Audio2Face3DTests.swift @@ -0,0 +1,114 @@ +import XCTest +@testable import Audio2Face3D + +final class Audio2Face3DTests: XCTestCase { + func testDefaultConfigurationMatchesNvidiaV23Window() { + let config = Audio2Face3DConfiguration() + XCTAssertEqual(config.inputSampleRate, 16_000) + XCTAssertEqual(config.bufferLength, 8_320) + XCTAssertEqual(config.hopLength, 4_160) + XCTAssertEqual(config.emotionVectorLength, 26) + XCTAssertEqual(config.outputCoefficientCount, 301) + XCTAssertEqual(config.frameSampleCount, 533) + } + + func testDefaultCoefficientLayoutMatchesNvidiaV23Mark() { + let layout = Audio2Face3DCoefficientLayout.nvidiaV23Mark + XCTAssertEqual(layout.skinCount, 272) + XCTAssertEqual(layout.tongueCount, 10) + XCTAssertEqual(layout.jawCount, 15) + XCTAssertEqual(layout.eyeCount, 4) + XCTAssertEqual(layout.coefficientCount, 301) + XCTAssertEqual(layout.skinRange, 0 ..< 272) + XCTAssertEqual(layout.tongueRange, 272 ..< 282) + XCTAssertEqual(layout.jawRange, 282 ..< 297) + XCTAssertEqual(layout.eyeRange, 297 ..< 301) + } + + func testFrameKeepsFullCoefficientVector() { + let coefficients = (0..<301).map { Float($0) / 300 } + let frame = Audio2Face3DFrame(timeSeconds: 0.25, coefficients: coefficients) + + XCTAssertEqual(frame.timeSeconds, 0.25) + XCTAssertEqual(frame.coefficients.count, 301) + XCTAssertEqual(frame.layout, .nvidiaV23Mark) + XCTAssertEqual(frame.coefficients[282], coefficients[282]) + } + + func testConfigurationCanReadJamesCoefficientLayout() throws { + let dir = temporaryDirectory() + try """ + { + "params": { + "implicit_emotion_len": 16, + "explicit_emotions": [ + "amazement", "anger", "cheekiness", "disgust", "fear", + "grief", "joy", "outofbreath", "pain", "sadness" + ], + "num_shapes_skin": 140, + "num_shapes_tongue": 10, + "result_jaw_size": 15, + "result_eyes_size": 4 + }, + "audio_params": { + "buffer_len": 8320, + "buffer_ofs": 4160, + "samplerate": 16000 + } + } + """.write( + to: dir.appendingPathComponent("network_info.json"), + atomically: true, + encoding: .utf8) + try """ + { "config": { "input_strength": 1.0 } } + """.write( + to: dir.appendingPathComponent("model_config.json"), + atomically: true, + encoding: .utf8) + + let config = try Audio2Face3DDownloader.configuration(from: dir) + + XCTAssertEqual(config.coefficientLayout.skinCount, 140) + XCTAssertEqual(config.coefficientLayout.tongueCount, 10) + XCTAssertEqual(config.coefficientLayout.jawCount, 15) + XCTAssertEqual(config.coefficientLayout.eyeCount, 4) + XCTAssertEqual(config.outputCoefficientCount, 169) + } + + func testJamesMLXRuntimeLoadsWhenBundleIsAvailable() throws { + let env = ProcessInfo.processInfo.environment["AUDIO2FACE3D_JAMES_FIXTURE_DIR"] + let dir = env.map { URL(fileURLWithPath: $0, isDirectory: true) } + ?? URL(fileURLWithPath: FileManager.default.currentDirectoryPath, isDirectory: true) + .appendingPathComponent("out/audio2face3d-mlx-james", isDirectory: true) + let weights = dir.appendingPathComponent("audio2face3d.safetensors") + guard FileManager.default.fileExists(atPath: weights.path) else { + throw XCTSkip("set AUDIO2FACE3D_JAMES_FIXTURE_DIR to an exported James Audio2Face3D MLX bundle") + } + + let model = try Audio2Face3DModel.fromLocal(directory: dir) + let audio = [Float](repeating: 0, count: model.configuration.bufferLength) + let frames = try model.frames(for: audio, sampleRate: model.configuration.inputSampleRate) + + XCTAssertEqual(model.configuration.coefficientLayout.skinCount, 140) + XCTAssertEqual(model.configuration.outputCoefficientCount, 169) + XCTAssertEqual(frames.first?.coefficients.count, 169) + } + + func testLocalModelRequiresExportedMLXWeights() { + let dir = temporaryDirectory() + + XCTAssertThrowsError(try Audio2Face3DModel.fromLocal(directory: dir)) { error in + XCTAssertEqual( + error as? Audio2Face3DError, + .missingExportedWeights(dir.appendingPathComponent("audio2face3d.safetensors").path)) + } + } + + private func temporaryDirectory() -> URL { + let url = FileManager.default.temporaryDirectory + .appendingPathComponent("Audio2Face3DTests-\(UUID().uuidString)", isDirectory: true) + try! FileManager.default.createDirectory(at: url, withIntermediateDirectories: true) + return url + } +} diff --git a/Tests/Audio2Face3DTests/E2EAudio2Face3DTests.swift b/Tests/Audio2Face3DTests/E2EAudio2Face3DTests.swift new file mode 100644 index 00000000..072b2f58 --- /dev/null +++ b/Tests/Audio2Face3DTests/E2EAudio2Face3DTests.swift @@ -0,0 +1,60 @@ +import XCTest +import Foundation +@testable import Audio2Face3D + +/// End-to-end tests that download the published MLX bundles from Hugging Face +/// and run real inference. Skipped by CI via `--skip E2E` filter. +final class E2EAudio2Face3DTests: XCTestCase { + func testFromPretrainedMarkGeneratesFrames() async throws { + try await assertPublishedBundleGeneratesFrames( + modelId: Audio2Face3DConfiguration.markModelId, + expectedCoefficientCount: 301) + } + + func testFromPretrainedClaireGeneratesFrames() async throws { + try await assertPublishedBundleGeneratesFrames( + modelId: Audio2Face3DConfiguration.claireModelId, + expectedCoefficientCount: 169) + } + + func testFromPretrainedJamesGeneratesFrames() async throws { + try await assertPublishedBundleGeneratesFrames( + modelId: Audio2Face3DConfiguration.jamesModelId, + expectedCoefficientCount: 169) + } + + private func assertPublishedBundleGeneratesFrames( + modelId: String, + expectedCoefficientCount: Int + ) async throws { + let model = try await Audio2Face3DModel.fromPretrained(modelId: modelId) + + let layout = model.configuration.coefficientLayout + XCTAssertEqual( + layout.skinCount + layout.tongueCount + layout.jawCount + layout.eyeCount, + expectedCoefficientCount, + "\(modelId) layout mismatch") + + // 1 s of speech-band chirp with an envelope, 16 kHz mono. + let sampleRate = 16_000 + let samples = (0.. Float in + let t = Float(index) / Float(sampleRate) + return 0.5 * sin(2 * .pi * (120 + 80 * t) * t) * sin(.pi * t) + } + + let frames = try model.frames(for: samples, sampleRate: sampleRate) + + XCTAssertGreaterThan(frames.count, 1) + for frame in frames { + XCTAssertEqual(frame.coefficients.count, expectedCoefficientCount) + XCTAssertTrue(frame.coefficients.allSatisfy(\.isFinite)) + } + + // The chirp must actually drive the face: coefficients vary across frames. + let first = frames[0].coefficients + let anyMotion = frames.dropFirst().contains { frame in + zip(frame.coefficients, first).contains { abs($0 - $1) > 1e-3 } + } + XCTAssertTrue(anyMotion, "\(modelId) produced static frames") + } +} diff --git a/docs/models/audio2face3d.md b/docs/models/audio2face3d.md new file mode 100644 index 00000000..e305148c --- /dev/null +++ b/docs/models/audio2face3d.md @@ -0,0 +1,67 @@ +# Audio2Face-3D Avatar Motion + +`Audio2Face3D` is the avatar-motion boundary for speech-driven faces. The +public output target is `Audio2Face3DFrame`: a timestamp plus the full NVIDIA +Audio2Face-3D coefficient vector for downstream avatar renderers. This module +does not synthesize viseme or energy-derived controls. + +## Architecture + +Audio2Face-3D is a two-branch audio-to-coefficient regressor. Per 8320-sample +window of 16 kHz mono audio (hop 4160) it emits one frame of geometry +coefficients, conditioned on a 26-value emotion vector (16 implicit + 10 +explicit; the explicit part is projected to 8 dims by a learned linear layer). + +```text +audio window ──┬─ audio branch: Wav2Vec2 feature extractor (7 convs, GELU) + │ -> feature projection -> grouped positional conv + │ -> transformer encoder layers -> learned layer blend + │ -> linear audio head + │ + └─ frequency branch: 25 Hann-windowed frames + x 32 autocorrelation lags -> 5 conv stages + +fusion: concat both branches -> 4 strided conv stages, with the emotion + vector broadcast onto the channels before each stage -> dense (256) +outputs: [features, emotion] -> face head (skin + jaw + eyes, two stacked + linears) and tongue head (two stacked linears) +``` + +- The audio branch is a trimmed Wav2Vec2 encoder; per-layer transformer + outputs are blended with the learned `combine_weights` vector. +- The frequency branch computes windowed autocorrelation (pitch/periodicity + cues) directly in the graph — no FFT dependency. +- Input audio is pre-scaled by `input_strength` from `model_config.json`. + +Coefficient layouts and encoder depth are identity-specific: + +| Identity | Version | Encoder layers | Skin | Tongue | Jaw | Eyes | Total | +|----------|---------|----------------|------|--------|-----|------|-------| +| Mark | v2.3 | 1 | 272 | 10 | 15 | 4 | 301 | +| Claire | v2.3.1 | 4 | 140 | 10 | 15 | 4 | 169 | +| James | v2.3.1 | 4 | 140 | 10 | 15 | 4 | 169 | + +Skin coefficients live in the identity's own geometry basis, so renderers need +the matching rig or a retarget projection for the chosen identity. + +## Current Runtime + +The only target backend is the full NVIDIA Audio2Face-3D model exported to MLX +tensors. The hand-written MLX graph port runs the real model forward pass and +is parity-checked against an ONNX fixture generated from the official NVIDIA +model, which is distributed under the NVIDIA Open Model License. + +Published bundles, downloaded on first use: + +- [`aufklarer/Audio2Face-3D-v2.3.1-James-MLX`](https://huggingface.co/aufklarer/Audio2Face-3D-v2.3.1-James-MLX) (default) +- [`aufklarer/Audio2Face-3D-v2.3.1-Claire-MLX`](https://huggingface.co/aufklarer/Audio2Face-3D-v2.3.1-Claire-MLX) +- [`aufklarer/Audio2Face-3D-v2.3-Mark-MLX`](https://huggingface.co/aufklarer/Audio2Face-3D-v2.3-Mark-MLX) + +Generate coefficient frames from a WAV file: + +```bash +swift run speech avatar-motion input.wav --output motion.jsonl --verbose +``` + +Pass `--model` to select another identity bundle, or `--model-dir` to run a +locally exported bundle.