feat: add support for llama.cpp, ollama, and deepseek completions #22
Conversation
hxueh
commented
Feb 1, 2025
hxueh
commented
- Add integration for llama.cpp to enable new completion workflows.
- Add integration for Ollama to enable new completion workflows.
- Add integration for DeepSeek to enable new completion workflows.
* Add integration for [llama.cpp](https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md#post-infill-for-code-infilling) to enable new completion workflows. * Add integration for [Ollama](https://github.com/ollama/ollama/blob/main/docs/api.md#generate-a-completion) to enable new completion workflows. * Add integration for [DeepSeek](https://api-docs.deepseek.com/api/create-completion) to enable new completion workflows.
feat: add support for llama.cpp, ollama, and deepseek completions
|
Hi @hxueh, Thank you so much for taking the time to create this PR - I really appreciate your effort! 🙌 I was going through the changes, and I realized that llama.cpp, Ollama, and DeepSeek completions follow the OpenAI-compatible API. Since we can simply use the existing OpenAI provider and configure the base URL accordingly, that should be sufficient to support these models, right? Just wanted to share these references: Let me know what you think! Again, really appreciate your contribution |
|
Hi @mohankumarelec , I do miss the Ollama and llama.cpp docs. Thanks for the reference. I did implement the DeepSeek as compatible to OpenAI. However, for Ollama, I do miss that docs since I just read the docs I provided in the PR, and miss the compatibility part. For llama.cpp, there is a dedicated API for FIM, which the author of llama.cpp use in his VSCode extension so I think this will be a better solution. Thanks for creating such a great VSCode extension. I really love it. Hope my contribution makes it better. |
|
Hi @hxueh, Thank you once again for your valuable contribution! The changes look great. If you could remove the Deepseek and Ollama implementations from the PR (since we can utilize the OpenAI provider for these), I’d be happy to merge the PR with the Llama.cpp provider updates. Looking forward to your update! |
Ollama completion could be done with OpenAI completion.
|
Hi @mohankumarelec , I have removed the Ollama implementation. However, the DeepSeek implementation could not removed. According to the get model docs, it use If the base URL doesn't set as |
