Are there plans to use local hosted LLMs

[72Joe72]

Hello, I was able to write a nice story with this script. However, it completely overloaded the Gemini API, which prevents me from writing a second story (a sequel).

Are there any future plans to integrate locale LLMs, for example, via Ollama? This would allow me to use the script independently of the Google service using my own computing power.

Here is the error message I'm currently getting:

Error:

Gemini API Error: got status: 503 . {"error":{"code":503,"message":"The model is overloaded. Please try again later.","status":"UNAVAILABLE"}}

Anyway, Thank you very much for this script. <3

[KazKozDev]

Thank you very much for your feedback! 😊

Ollama and local models were supported in earlier versions of the script. However, I eventually had to move away from them, as they weren’t able to consistently maintain the level of narrative coherence needed for long-form storytelling.

That said, I completely understand the limitations of relying solely on remote APIs like Gemini — especially with the 503 overload issues you encountered. I recommend trying to run the generation again after some time — it will likely start working once the load decreases, although I can’t say for sure, as I haven’t personally encountered this issue myself.

Thanks again for your kind words and for using the NovelGenerator!

[marcotama]

In my opinion, the low quality of local LLMs that you encountered is not a reason to remove support for them.

We don't know that they are not going to get better, and we don't know that larger ones won't work well (have you tried a llama 405B?).