Ability to use local LLM(LM Studio or Ollama)

[eliliam]

This is such an amazing package, but with some larger codebases that I work with the costs would just be too high to run this using cloud models. How difficult would it be to support locally running LLM models through the likes of LM Studio or Ollama? I know both provide OpenAI compatible APIs as well as a suite of other ways to interact with the locally running model. This feature would be killer and would set this out as a tool similar to Claude Code for local codebase analysis.

[zachary62]

Hi @eliliam,
The tool needs a call_llm function that takes a string as input and outputs a string: https://github.com/The-Pocket/Tutorial-Codebase-Knowledge/blob/main/utils/call_llm.py
So you can simply replace the function with an implementation based on LM Studio or Ollama: https://the-pocket.github.io/PocketFlow/utility_function/llm.html
Let me know if this makes sense. Thanks!

[sitestudio]

So I asked an LLM what to put in here and it came back with this (which works beautifully):

def call_llm(prompt, use_cache: bool = True):
    """
    Calls an Ollama model to generate a text response.

    Args:
        prompt (str): The prompt to send to the model.
        use_cache (bool, optional): Whether to use Ollama's caching mechanism. Defaults to True.

    Returns:
        str: The generated text response from the model.
    """
    try:
        response = ollama.chat(
            model='cogito:14b',  # deepcoder:14b  gemma3:12b  phi4:14b Replace with your desired Ollama model name
            messages=[
                {
                    'role': 'user',
                    'content': prompt,
                },
            ],
            stream=False, # important to set stream to false to get the response.
            options = {
                'use_cache': use_cache,
            }

        )
        return response['message']['content']
    except ollama.ResponseError as e:
        print(f"Ollama Error: {e}")
        return None  # Or handle the error as needed.

Also had to add the following to requirements.txt:

ollama >=0.4.7

[zachary62]

@sitestudio
Amazing! One minor change I would suggest is to remove the except: return None
It may generate, e.g., a tutorial with empty chapters. Just let it fail, as pocket flow has native node retry.

[xiongyw]

This is such an amazing package, but with some larger codebases that I work with the costs would just be too high to run this using cloud models. How difficult would it be to support locally running LLM models through the likes of LM Studio or Ollama? I know both provide OpenAI compatible APIs as well as a suite of other ways to interact with the locally running model. This feature would be killer and would set this out as a tool similar to Claude Code for local codebase analysis.

FYI: the following simple update works for me, for both ollama and grok as tested. call_llm() for ollama and grok

[FatCache]

Are there any plans to merge the Ollama into the code base?

[zachary62]

Are there any plans to merge the Ollama into the code base?

Check out the code snippet provided by @xiongyw for ollama support: call_llm() for ollama and grok

[piranna]

Ollama also allows to work with DeepSeek, I don't know if it supports Gemini and OpenAI and other closed ones, but if so, we could replace call_llm() function for a single one using Ollama, delegating the actual model to be used to it :-)

[eliliam]

@zachary62 the code provided by @xiongyw looks good, could we have that made into a PR to get merged into master?

[zachary62]

@zachary62 the code provided by @xiongyw looks good, could we have that made into a PR to get merged into master?

Yes! Could you make this code commented out by default, and say something like "uncomment it for ollama". Thank you!

[Le09]

I cherry-picked the commit an implemented a simple switch in #50, let me know if you prefer some changes to it.

[gethari]

What's the best model to use if using for a React+TS codebase ? I tried the generation with llama3.2 and the output generated is just hot garbage, none of the content generated is relevant to the codebase I pointed at.

[sitestudio]

Try the brand new Qwen3:8b (or bigger if your hardware can handle it) - switched to it yesterday and was getting much better results than with anything else. I was only generating python and have limited hardware atm, but was very impressed with the step up in it's "cognition".

[gethari]

@sitestudio I tried using codellama:13b, the script seems to be failing randomly, I tried copilot to fix this, but not a python expert so, I could'nt use this model.

 yaml_str = response.strip().split("```yaml")[1].split("```")[0].strip()
               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

Looks like the response is not in expected format and differs across models