Local Voice Agent

This project's goal is to enable anyone to easily build a powerful, private, local voice AI agent.

A real-time voice AI assistant — STT, LLM, TTS — running in one container, supervised by a single Python parent process. Powered by LiveKit Agents.

To keep up with what I'm building or request new features send me a DM on X

Overview

Everything runs as managed children of one Python supervisor (python -m local_voice_ai serve):

• LiveKit server (Go binary subprocess) for WebRTC signaling — skipped if LIVEKIT_URL points at LiveKit Cloud.
• llama.cpp (llama-server binary subprocess) for the LLM — default model is Gemma 4 E2B (quantization-aware-trained 4-bit, ~2.6 GB); swap it with LLAMA_HF_REPO=org/repo:quant. Skipped if LLAMA_BASE_URL points elsewhere.
• Nemotron STT or Whisper (faster-whisper) — Python uvicorn child, OpenAI-compatible.
• Kokoro TTS — Python uvicorn child, OpenAI-compatible.
• LiveKit Agents worker — the orchestrator child.
• FastAPI in the supervisor itself, serving POST /api/connection-details (token minting) and the statically-exported Next.js frontend.

Children speak HTTP only over 127.0.0.1. The image exposes four ports: 8080 (web), 7880, 7881, 7882/udp (LiveKit WebRTC, only if running locally).

Getting started

Run the prebuilt image (amd64 + arm64):

docker run --rm -it \
  -p 8080:8080 -p 7880:7880 -p 7881:7881 -p 7882:7882/udp \
  -v local-voice-ai-models:/models \
  ghcr.io/shaynep/local-voice-ai:latest

Or build from source (also the path for GPU builds):

docker compose up --build

Open http://localhost:8080. The first boot downloads the Nemotron + LLM weights — the page shows per-service progress with download sizes, and the terminal logs a compact status heartbeat plus an unmissable “ready” banner when everything is up. Weights are cached in the models volume, so later boots are fast and work offline.

GPU (NVIDIA)

docker compose -f docker-compose.yml -f docker-compose.gpu.yml up --build

The overlay swaps in the CUDA llama.cpp binary + CUDA torch wheels, grants the GPU to the container, and offloads the whole LLM (LLAMA_N_GPU_LAYERS=999, override to partially offload). Requires the NVIDIA container toolkit — verify with docker run --gpus all ubuntu nvidia-smi.

Apple Silicon

The prebuilt image runs natively (arm64), but CPU-only — Docker on macOS is a VM with no Metal access. For GPU (Metal) inference, run bare-metal via Local development below, where llama-server picks up Metal automatically.

Swapping in cloud providers

Each service has a single "manage" decision driven by its base URL — point it at a remote endpoint and the local subprocess is skipped:

Goal	Set
Use LiveKit Cloud	LIVEKIT_URL=wss://your-project.livekit.cloud (+ LIVEKIT_API_KEY / …_SECRET)
Use OpenAI for the LLM	LLAMA_BASE_URL=https://api.openai.com/v1, LLAMA_MODEL=gpt-4o-mini, LLAMA_API_KEY=sk-…
Use a remote OpenAI-compatible STT	STT_BASE_URL=…, STT_MODEL=…, STT_API_KEY=…
Use a remote OpenAI-compatible TTS	TTS_BASE_URL=…, TTS_API_KEY=…

The supervisor logs which children it manages on startup.

Local development (no Docker)

Requires Python 3.11+, plus the livekit-server and llama-server binaries on your PATH (macOS: brew install livekit llama.cpp).

# Python side
uv pip install -e ".[ml,dev]"
python -m local_voice_ai serve

# Frontend side, in another shell (only needed if you're editing the UI)
cd frontend && pnpm install && pnpm run dev

Architecture

┌──────────────────────── single container ────────────────────────┐
│  python -m local_voice_ai serve                                  │
│  │                                                                │
│  ├── child: livekit-server     (skipped if LIVEKIT_URL external) │
│  ├── child: llama-server       (skipped if LLAMA_BASE_URL ext.)  │
│  ├── child: nemotron | whisper (skipped if STT_BASE_URL ext.)    │
│  ├── child: kokoro             (skipped if TTS_BASE_URL ext.)    │
│  ├── child: livekit-agents worker                                │
│  └── in-process: FastAPI on :8080                                 │
│        ├── POST /api/connection-details  (token minting)         │
│        ├── GET  /api/status              (per-child readiness)   │
│        └── GET  /*                       (static frontend)       │
└───────────────────────────────────────────────────────────────────┘

Project structure

.
├─ local_voice_ai/         # Python package: supervisor + agent + services
│  ├─ __main__.py          # python -m local_voice_ai serve
│  ├─ supervisor.py        # async process supervisor
│  ├─ config.py            # env-driven config + manage-X flags
│  ├─ api.py               # FastAPI: token route, status, static frontend
│  ├─ agent.py             # LiveKit Agents worker
│  ├─ wakeword.py          # optional "hey livekit" gate for the agent
│  └─ services/
│     ├─ nemotron/server.py
│     ├─ whisper/server.py
│     └─ kokoro/server.py
├─ frontend/               # Next.js (configured for static export)
├─ tests/                  # pytest suite
├─ Dockerfile              # multi-stage build
├─ docker-compose.yml      # one service (CPU default)
├─ docker-compose.gpu.yml  # NVIDIA overlay: CUDA build + GPU reservation
├─ .github/workflows/      # CI: tests + multi-arch image publish to GHCR
└─ pyproject.toml          # one Python package, one venv

Environment variables

See .env for the full list. The most important ones:

• LIVEKIT_URL, LIVEKIT_API_KEY, LIVEKIT_API_SECRET — local-default; override for cloud.
• LLAMA_BASE_URL, LLAMA_MODEL, LLAMA_HF_REPO, LLAMA_N_GPU_LAYERS
• LLAMA_OFFLINE — offline LLM startup. Auto by default: once the model is cached, it starts with no internet (skips the Hugging Face lookup); the first run still downloads. Set LLAMA_OFFLINE=1 to force it, or 0 to always re-check. LLAMA_MODEL_PATH=/models/…​.gguf loads a local file directly instead.
• WAKE_WORD=1 — the agent joins deaf and only starts listening after it hears “Hey LiveKit” (on-device detection via livekit-wakeword, model baked into the image). WAKE_WORD_THRESHOLD (default 0.5) tunes sensitivity; scores are logged at DEBUG for calibration.
• STT_PROVIDER (nemotron|whisper), STT_BASE_URL, STT_MODEL; WHISPER_MODEL picks the faster-whisper model for the whisper provider.
• TTS_BASE_URL, TTS_VOICE
• WEB_PORT (default 8080)
• MANAGE_LIVEKIT, MANAGE_LLAMA, MANAGE_STT, MANAGE_TTS — explicit overrides for the auto-detected "is the URL external?" logic.

Credits

• LiveKit: https://livekit.io/
• LiveKit Agents: https://docs.livekit.io/agents/
• NVIDIA Nemotron Speech: https://huggingface.co/nvidia/nemotron-speech-streaming-en-0.6b
• llama.cpp: https://github.com/ggml-org/llama.cpp
• Gemma 4 (default LLM, Unsloth QAT GGUF): https://huggingface.co/unsloth/gemma-4-E2B-it-qat-GGUF
• Kokoro TTS: https://github.com/hexgrad/kokoro
• faster-whisper (Whisper fallback): https://github.com/SYSTRAN/faster-whisper
• livekit-wakeword ("hey livekit" detection): https://github.com/livekit/livekit-wakeword