Add OpenVINO backend

[wine99]

Overview

This PR introduces an OpenVINO backend for llama.cpp, enabling hardware-accelerated inference on Intel® CPUs, GPUs, and NPUs. The backend leverages OpenVINO to deliver optimized inference with the existing llama.cpp GGUF model ecosystem. Enables performance improvements via OpenVINO’s graph compilation and kernel fusion.

Key Features:

• New backend implementation

  • Added OpenVINO backend in ggml/src/ggml-openvino.
  • Implemented translations for core GGML operations

• Supported precisions

  • FP16 GGUF models supported.
  • Initial quantization support available in quant branch.

• Supported devices

  • Intel CPUs
  • Intel integrated and discrete GPUs
  • Intel NPUs (requires UD32+ driver - Windows available, Linux coming soon).

Tested Models

In this PR, OpenVINO backend supports FP16 GGUF models. The following models have been validated (Additional GGUF FP16 models may also be compatible):

• Llama-3.2-1B-Instruct-GGUF
• microsoft/Phi-3-mini-4k-instruct-gguf
• Qwen/Qwen2.5-1.5B-Instruct-GGUF

Work in Progress

• Performance and memory optimizations
• Broader quantization coverage.
• Support for additional model architectures.

[SearchSavior]

Hello,

in this repo https://github.com/yangsu2022/GGUF-to-OpenVINO and the article https://blog.openvino.ai/blog-posts/openvino-genai-supports-gguf-models only a small set of models are supported.

Will this feature in llama.cpp offer wider gguf coverage via something like the parameter mapping described here,

https://github.com/yangsu2022/GGUF-to-OpenVINO/blob/405a95e300f8307fb4b779a12d46cf86adf5a441/convert_llama3.1_gguf_to_torch.py#L14

A few other questions:

• What parts of OpenVINO feature set are intended to be brought into llama.cpp?

• Is this PR trying to bring in only performance from openvino runtime to support llama.cpp usecase?

• Pipeline parallel is coming in next release (I think), will that be implemented here for heterogeneous execution in llama.cpp?

Thank you for your work!

[ravi9]

Hi @SearchSavior ,

Q: Will this feature in llama.cpp offer wider GGUF coverage via something like parameter mapping?

Instead of converting GGUF models to PyTorch format with parameter mapping, this implementation uses OpenVINO's GGML frontend to directly translate GGML computation graphs to OpenVINO operations at runtime. The translation happens through a comprehensive operation mapping system that covers the core GGML operations. Since it works at the GGML operation level, it should support any model architecture that llama.cpp supports (assuming we map/translate all the GGML operators to OpenVINO.)

Q: What parts of the OpenVINO feature set are intended to be brought into llama.cpp?

The immediate focus is on runtime acceleration: kernel fusion, optimized graph execution,memory optimizations and hardware scheduling on CPU, GPU, and NPU.

Q: Is this PR trying to bring in only performance from openvino runtime to support llama.cpp usecase?

The scope of this PR is primarily performance enablement using OpenVINO runtime to accelerate llama.cpp inference while preserving compatibility with the GGUF ecosystem. It’s not introducing a new model conversion flow, so everything remains driven by GGUF models in llama.cpp.

Q: Will pipeline parallel / heterogeneous execution be supported here?

We are currently reviewing this. llama.cpp already has infrastructure for pipeline parallelism, and the OpenVINO backend exposes async operations and events, so it should be possible. Further evaluation is needed to confirm integration details.

[SearchSavior]

Hey @ravi9 ,

Thanks for the detailed answer. It's nice to see more serious work bringing OpenVINO to the rest of the ecosystem.