Support for older AMD CPUs?

[rwalle]

According to project description, this only supports computers with the latest "Ryzen AI" CPUs.

Is there any blocker to support older CPUs without an NPU? Can we run LLMs with iGPU only?

Very few people have the latest generation of PCs, and I doubt anyone is going to update their perfectly fine computer purchased one year ago just to use LLM on consumer hardware.

I understand that there is a performance benefit of having the NPU, as mentioned in FAQ:

The iGPU can also handle LLMs but will consume more power and might have slower inference times compared to the NPU.

But hey, we all understand that we are not looking for the best performance on consumer hardware (otherwise I would be renting some H100s). If it runs on a machine with 680M or 780M, it is better than not being able to run it at all, despite being slower. This project could provide a meaningful alternative to Ollama, since it does not support Vulkan (ollama/ollama#5059). (I am aware that llama.cpp supports Vulkan by itself, but having a GUI is still nice.)

(Alternatively, if someone could create a fork or point me to where the restrictions are, I'll be happy to take a look.)

[kovtcharov]

Hi @rwalle, thanks for bringing this up, we've heard this request from several folks so we are actively working on enabling/supporting older systems on CPU and iGPU. I will report here once it becomes available, I expect in a few weeks.

[kovtcharov]

You may want to checkout LM Studio in the meanwhile, they have a UI+llama.cpp/Vulkan support.