Skip to content

localai 4.4.0#287347

Merged
BrewTestBot merged 2 commits into
mainfrom
bump-localai-4.4.0
Jun 10, 2026
Merged

localai 4.4.0#287347
BrewTestBot merged 2 commits into
mainfrom
bump-localai-4.4.0

Conversation

@BrewTestBot

@BrewTestBot BrewTestBot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Created by brew bump

Created with brew bump-formula-pr.

Details

release notes 
# 🎉 LocalAI 4.4.0 Release! 🚀


  

  
  

  




LocalAI 4.4.0 is out!


This is a big, multimodal-and-distributed release. Two brand-new audio backends land - parakeet.cpp (NVIDIA NeMo Parakeet ASR) and CrispASR (a multi-architecture ASR and TTS engine) - alongside native object detection + segmentation (rfdetr-cpp), video understanding in llama-cpp, and LTX-2 video generation in stablediffusion-ggml. Distributed mode grows up: prefix-cache-aware routing is on by default, and file transfers become resumable. There's a new intelligent middleware layer for request routing, PII filtering and cloud-model proxying, a security hardening pass that closes a credential-leak class across every outbound HTTP client, an interactive local-ai chat CLI, RAG source citations for agents, and a long run of reasoning / tool-call streaming fixes.




📌 TL;DR






Area
Summary






🎙️ Two new ASR backends
parakeet-cpp (NeMo FastConformer TDT/CTC/RNNT, streaming, word/segment timestamps) and crispasr (many ASR architectures + TTS in one binary).




🧭 Intelligent Middleware
Capability-based model routing, PII detection/redaction, cloud-model proxies + a MITM proxy for subscription-auth Claude Code / Codex.




🛰️ Distributed v4
Prefix-cache-aware routing (on by default), NATS JWT auth + TLS/mTLS, worker registration-token enforcement, resumable HTTP file transfers, boot-time model prefetch, ds4 layer-split inference.




🎥 Video, both ways
Video input (understanding) in llama-cpp via mtmd, and video generation via LTX-2 in stablediffusion-ggml.




👁️ Detection + Segmentation
New native rfdetr-cpp backend (RF-DETR), 32 prebuilt GGUFs, bbox + per-detection PNG masks.




🔐 Outbound HTTP hardening
pkg/httpclient refuses cross-host credential-leaking redirects across every outbound client (GHSA-3mj3-57v2-4636).




🗣️ TTS per-request control
instructions + a generic params map plumbed end to end (Qwen3-TTS VoiceDesign / CustomVoice, Chatterbox).




💻 local-ai chat
Interactive terminal chat against a running server, with /models, /model, /clear.




📚 RAG citations
Agent answers now append a clickable Sources: block from the Knowledge Base.




🧠 Models
Gemma 4 QAT family + QAT-matched MTP speculative-decoding bundles, Ideogram4, LTX-2.3 22B GGUFs.








🚀 New Features & Major Enhancements


🎙️ Audio Gets Serious: Two New ASR Backends


This release doubles down on speech-to-text with two independent, cgo-less Go backends (purego, CGO_ENABLED=0), each shipping a full CI matrix, gallery importer and docs.


parakeet-cpp - NVIDIA NeMo Parakeet (#10084). Wraps parakeet.cpp, a C++/ggml port of NeMo Parakeet (FastConformer TDT/CTC/RNNT/hybrid) that matches the upstream PyTorch models on CPU. Text transcription, OpenAI-compatible word timestamps, and cache-aware streaming (16 kHz PCM chunks, <EOU>/<EOB> utterance boundaries). GGUFs for all 10 Parakeet models × 5 quants ship in mudler/parakeet-cpp-gguf. Follow-ups in this cycle made it production-grade:


    

  • Dynamic batching (#10112) - concurrent transcription requests are batched for throughput.
    • 

  • Real, NeMo-faithful segment timestamps (#10207) - words are grouped into segments exactly like NeMo's get_segment_offsets (sentence-punctuation boundaries by default, opt-in segment_gap_threshold silence splitting in encoder frames). Streaming FinalResult segments now carry start/end when the library exposes the ABI v4 JSON entry points.
    • 

  • nemotron-3.5-asr multilingual streaming (#10199) + per-request language selection.
    • 



crispasr - many architectures + TTS in one backend (#10099). Wraps CrispASR (a whisper.cpp/ggml fork, MIT) through its session C-ABI. One backend serves ASR or TTS depending on the loaded model, with the architecture auto-detected from the GGUF (or forced via backend:). The gallery gains 36 -crispasr entries (32 ASR + 4 TTS):


    

  • ASR (e2e-verified across Whisper / Parakeet / Moonshine): parakeet, canary, cohere, qwen3, voxtral, granite, fastconformer-ctc, wav2vec2, hubert, data2vec, glm-asr, kyutai-stt, firered-asr, moonshine, mimo-asr, and more.
    • 

  • TTS (all four e2e-verified to valid 24 kHz mono WAV): vibevoice, chatterbox, qwen3-tts CustomVoice, orpheus - via backend: / codec: / speaker: / voice: model options.
    • 





🧭 Intelligent Middleware: Routing, PII Filtering & Cloud Proxies


A new middleware layer (#9802) analyzes, routes, filters and transforms chat requests before they hit a model.


    

  • Capability-based routing. Requests are classified (e.g. via an ArchRouter-style model) and scored across the capabilities they may require, then routed to the smallest model that satisfies them - easy requests go to small specialized models, hard or uncertain ones to larger general-purpose models. Classified embeddings are reused via cosine similarity so similar requests skip re-classification.
    • 

  • PII filtering. Private information is detected per-pattern and can be redacted, rerouted, or blocked, with a streaming PII filter that preserves a buffered-emit invariant on /v1/chat/completions, Anthropic /v1/messages, and /v1/completions. A per-model PII pattern editor lives in the model config UI.
    • 

  • Cloud model proxies + MITM. Cloud models and a MITM proxy can take part in routing/filtering - send easy requests to local models and hard ones to the cloud, and use Claude Code / Codex subscriptions (OAuth) through the PII filter via the MITM proxy (subject to provider ToS). Emits proxy_connect + proxy_traffic audit events and restores its listener from runtime_settings.json on restart.
    • 



Usage stats are recorded end to end and surfaced in REST, the UI, and MCP. Outbound clients used by this path were also the trigger for the security pass below.




🛰️ Distributed Mode v4


Distributed mode keeps maturing across routing, security and resilience.


Prefix-cache-aware routing, on by default (#10071). Routing now biases toward the replica that already holds the relevant KV/prefix cache, as a load-guarded hint that never routes worse than today's round-robin. A generic prefix tree (pkg/radixtree) maps cumulative prompt-prefix hashes to nodes; core/services/nodes/prefixcache turns the rendered prompt into a deterministic xxhash chain and makes a filter-then-score decision (narrow to load-eligible replicas, then prefer the longest-prefix match), feeding a preferredNodeID into the existing atomic SELECT ... FOR UPDATE pick. Observations sync across frontends over NATS. Round-robin is the floor; disable with --distributed-prefix-cache=false.


NATS JWT auth + TLS/mTLS (#10159). Previously anyone with access to the NATS port could publish backend-install messages or agent jobs (an SSRF / accidental-exposure risk). This adds JWT authentication and TLS/mTLS options, with workers acquiring and auto-refreshing their NATS credentials. Complemented by worker file-transfer registration-token enforcement (#10183).


Resumable file transfers (#10109). Large model GGUFs over flaky/throttled links no longer restart from byte 0. The worker's PUT /v1/files/<key> honors Content-Range (308/416 resume semantics, X-Content-SHA256 binding, final-hash verification) and the master-side stager HEAD-probes for the last accepted offset and resumes, switching to an outer time budget (LOCALAI_FILE_TRANSFER_BUDGET, default 1h) with exponential backoff.


ds4 layer-split distributed inference (#10098). Manual layer-split inference for the ds4 backend: a coordinator owns layers 0:K and listens; workers dial in and own higher ranges, each loading only its slice of the GGUF (a new dependency-free ds4-worker binary, driven via local-ai worker ds4-distributed). Fully back-compatible when ds4_role is absent.


Operational glue. Boot-time gallery prefetch via LOCALAI_PREFETCH_MODELS (#10108); a gated X-LocalAI-Node response header for attribution (#9976); plus fixes: self-heal stale "model not loaded" routing (#10181), stage directory-based models to remote nodes (#10175), in-flight tracking for non-LLM methods - VAD, diarize, voice (#10238), reconciler survives frontend restarts (#9981), cross-replica OpCache sync (#9983), and the reinstall/upgrade UI no longer sticks on "reinstalling" (#10214).




🎥 Video, Both Directions


Video input / understanding in llama-cpp (#10216). Video-capable multimodal models (e.g. SmolVLM2-Video) can now be sent a video in a chat request, mirroring the existing image and audio paths. Tracks the upstream mtmd video landing (ggml-org/llama.cpp#24269); grpc-server.cpp forwards request->videos() into the mtmd files vector on both the template and non-template paths, and the React chat UI accepts video/*, renders an inline <video controls> player, and emits video_url content parts. allow_video is auto-gated by whether the loaded mmproj supports it. ffmpeg/ffprobe (already in the runtime image) extract frames.


Video generation via LTX-2 (#9980). stablediffusion-ggml wires audio_vae_path and embeddings_connectors_path through to the upstream LTX-2 fields, with a new gallery/ltx-ggml.yaml template (T2V / I2V / FLF2V recipes) and six LTX-2.3 22B GGUF gallery entries (dev + distilled, UD-Q4_K_M / Q4_K_M / Q8_0), each bundling the text encoder + video VAE + audio VAE + embeddings connectors. Follow-up fixes wired the diffusion_model flag and vae_decode_only:false for the i2v/flf2v paths (#9986, #9987) and muxed LTX-2 audio into the output MP4 (#9990).




👁️ Native Object Detection + Segmentation: rfdetr-cpp


A new Go native gRPC backend (#10028) dlopens librfdetr.so (built from mudler/rf-detr.cpp) and exposes the RF-DETR pipeline through LocalAI's Detect RPC. Supports all 5 detection variants (Nano…Large) and 3 segmentation variants (SegNano/SegSmall/SegMedium) at F32/F16/Q8_0/Q4_K, with 32 prebuilt GGUFs on HuggingFace. Detection returns bbox + class_name + confidence; segmentation adds per-detection PNG-encoded masks. Matches PyTorch on CPU (sub-pixel bbox match, mask IoU 0.99+), with an HF gallery importer that auto-routes GGUF repos to the native backend.




🔗 PR: #10028. Also new: Ideogram4 support in stablediffusion-ggml (#10201).






🗣️ TTS: Per-Request Instructions & Params


The OpenAI-compatible /v1/audio/speech instructions field was silently dropped at the HTTP→gRPC boundary, so style/voice could only come from static YAML. PR #10172 plumbs a generic per-request instructions string and an optional backend-specific params map end to end (proto, schema, core/backend/tts.go), unlocking per-line emotion/style (Qwen3-TTS CustomVoice, Chatterbox) and describe-a-voice (Qwen3-TTS VoiceDesign) from a single model config. Fully backward compatible - empty instructions falls back to YAML.


curl http://localhost:8080/v1/audio/speech -H "Content-Type: application/json" -d '{
  "model": "qwen-tts-design",
  "input": "Hello world, this is a test.",
  "instructions": "A calm, low-pitched elderly storyteller with a warm tone."
}'


Also: Qwen3-TTS request-language normalization for flexible matching (#10174), and LocalVQE v1.3 with input/output spectrogram views in the Audio Transform UI (#10113).




🧠 Reasoning & Tool-Call Streaming Hardening


A focused run of correctness fixes for reasoning models and streaming tool calls:


    

  • reasoning_effort honored per request and forwarded to the backend so jinja models can act on it (#10082, #10184).
    • 

  • <think> parsing: stop <think> leaking into content in pure-content mode (#9991), stop a prefilled <think> from swallowing tag-less answers (#10225), and don't auto-enable self-spec MTP for draft-only assistant GGUFs (#10208).
    • 

  • Streaming + tools: stop tool-call double-emission when the autoparser is active (#10055), stop tool-call JSON leaking into content on tokenizer-template models (#10057), validate auto-detected XML tool-call names with a robust glm-4.5/Hermes guard (#10059), and stop healing-marker stubs / prefill-misclassified content from corrupting the stream (#9999, #10000).
    • 





💻 local-ai chat + 📚 RAG Citations + 🛰️ Realtime


    

  • Interactive CLI chat (#10226). A new opt-in local-ai chat command connects to a running server over the OpenAI-compatible API, streams completions, and supports /models, /model <name>, /clear, /exit. Keeps local-ai run focused on the server lifecycle. (Fixes #1535.)
    • 

  • RAG source citations (#10228). When an agent answers from the Knowledge Base, the response now appends a clickable Sources: block listing the original documents - deduplicated per source, with the citation-free version saved to long-term memory. (Closes #9331.)
    • 

  • Configurable WebRTC ICE candidates (#10231). New LOCALAI_WEBRTC_NAT_1TO1_IPS / LOCALAI_WEBRTC_ICE_INTERFACES knobs fix /v1/realtime calls dropping a few seconds in under Docker host networking (unroutable docker0/veth candidates).
    • 

  • "Fits in my GPU" filter (#10017) on the Install Models page, plus a single shared /api/operations poller across UI consumers (#10029) and a React bundle code-split (#10042).
    • 





🧩 Backend Capability Registration & Startup Speed


    

  • Backend capability registration fixes so the right backend is picked for the right job: register 5 backends missing from BackendCapabilities (#10107), and add face/speaker-recognition constants registering insightface + speaker-recognition (#10110).
    • 

  • Faster startup (#10213): skip vocab arrays and mmap GGUF headers during config parsing.
    • 






 
Click for the full changelog below!


What's Changed


Bug fixes :bug:



Exciting New Features 🎉



🧠 Models



📖 Documentation and examples



👒 Dependencies



Other Changes







🙌 New Contributors



Enjoy!




Full Changelog: mudler/LocalAI@v4.3.0...v4.4.0

View the full release notes at https://github.com/mudler/LocalAI/releases/tag/v4.4.0.

@github-actions github-actions Bot added go Go use is a significant feature of the PR or issue nodejs Node or npm use is a significant feature of the PR or issue bump-formula-pr PR was created using `brew bump-formula-pr` labels Jun 10, 2026
@github-actions

github-actions Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

🤖 An automated task has requested bottles to be published to this PR.

Caution

Please do not push to this PR branch before the bottle commits have been pushed, as this results in a state that is difficult to recover from. If you need to resolve a merge conflict, please use a merge commit. Do not force-push to this PR branch.

@github-actions github-actions Bot added the CI-published-bottle-commits The commits for the built bottles have been pushed to the PR branch. label Jun 10, 2026
@BrewTestBot BrewTestBot enabled auto-merge June 10, 2026 21:46
@BrewTestBot BrewTestBot added this pull request to the merge queue Jun 10, 2026
Merged via the queue into main with commit 8875902 Jun 10, 2026
22 checks passed
@BrewTestBot BrewTestBot deleted the bump-localai-4.4.0 branch June 10, 2026 21:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actions github-actions[bot] github-actions[bot] approved these changes

@botantony botantony botantony approved these changes

Assignees

No one assigned

Labels

bump-formula-pr PR was created using `brew bump-formula-pr` CI-published-bottle-commits The commits for the built bottles have been pushed to the PR branch. go Go use is a significant feature of the PR or issue nodejs Node or npm use is a significant feature of the PR or issue

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@BrewTestBot @botantony