Add Audio2Face-3D avatar motion runtime#376
Merged
Merged
Conversation
ivan-digital
force-pushed
the
feat/audio2face-avatar-runtime
branch
3 times, most recently
from
f88a460 to
9697bf7
Compare
MLX Swift port of NVIDIA Audio2Face-3D: a hand-written graph runs the official model forward pass and emits timestamped coefficient frames for speech-driven avatars. - Audio2Face3D library target; fromPretrained downloads the published MLX bundles (Mark v2.3 default, Claire and James v2.3.1) from Hugging Face through the shared Hub path. - speech avatar-motion CLI command: WAV in, JSONL coefficient frames out. - Deterministic unit tests, ONNX parity fixtures, and E2E tests that download each published bundle and run real inference. - Model doc covering the network architecture and identity layouts; catalogue entries in README.md and all 13 translations.
ivan-digital
force-pushed
the
feat/audio2face-avatar-runtime
branch
from
9697bf7 to
da1657d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MLX Swift port of NVIDIA Audio2Face-3D for speech-driven avatar faces.
Audio2Face3Dlibrary product. It loads the exported safetensors bundle and runs the official model forward pass as a hand-written MLX graph — no ONNX runtime dependency.Audio2Face3DFrame: a timestamp plus the full facial coefficient vector, with the skin/tongue/jaw/eyes layout exposed for renderers.speech avatar-motionCLI command: WAV in, JSONL coefficient frames out.fromPretraineddownloads the published bundles:Audio2Face-3D-v2.3.1-James-MLX(default) andAudio2Face-3D-v2.3.1-Claire-MLX(169 coefficients each), plusAudio2Face-3D-v2.3-Mark-MLX(301 coefficients).Docs
docs/models/audio2face3d.md: network architecture (two-branch Wav2Vec2 + autocorrelation regressor with emotion conditioning), per-identity coefficient layouts, published bundles, CLI usage.README.mdand all 13 translations.Testing
swift test --filter E2EAudio2Face3DTests— downloads each published bundle from Hugging Face and runs real inference: Mark, Claire, and James all pass; frames are well-shaped (301/169/169 coefficients), finite, and vary with the audio. CI skips via the--skip E2Econvention.swift test --filter Audio2Face3DTests— deterministic unit tests pass.AUDIO2FACE3D_FIXTURE_DIR(skips without fixtures).