Skip to content

Latest commit

 

History

History
169 lines (114 loc) · 5.82 KB

ollama.md

File metadata and controls

169 lines (114 loc) · 5.82 KB

Ollama

Setup

You can use your model for free with Ollama and it is available to use both Ollama and remote providers simultaneously.

  1. Install Ollama from https://ollama.com

  2. Start it with your model

ollama run llama3.2 # model you want use. ex) codellama, deepseek-coder
  1. Set the host, model and numCtx. (The default numCtx value in Ollama is 2048. It is recommended to set it to 4096 or higher.)
aicommit2 config set OLLAMA.host=<your host>
aicommit2 config set OLLAMA.model=<your model>
aicommit2 config set OLLAMA.numCtx=4096

If you want to use Ollama, you must set OLLAMA.model.

  1. Run aicommit2 with your staged in git repository
git add <files...>
aicommit2

👉 Tip: Ollama can run LLMs in parallel from v0.1.33. Please see this section.

📌 Important Note

Before configuring, please review:

Example Configuration

Basic Setup

aicommit2 config set OLLAMA.model="llama3.2"

Advanced Setup

aicommit2 config set OLLAMA.model="codellama" \
  OLLAMA.numCtx=4096 \
  OLLAMA.temperature=0.7 \
  OLLAMA.maxTokens=4000 \
  OLLAMA.locale="en" \
  OLLAMA.generate=3 \
  OLLAMA.topP=0.9

Settings

Setting Description Default
model Model(s) to use (comma-separated list) -
host Ollama host URL http://localhost:11434
auth Authentication type Bearer
key Authentication key -
numCtx The maximum number of tokens the model can process at once 2048

Configuration

OLLAMA.model

The Ollama Model. Please see a list of models available

aicommit2 config set OLLAMA.model="llama3.1"
aicommit2 config set OLLAMA.model="llama3,codellama" # for multiple models

aicommit2 config add OLLAMA.model="gemma2" # Only Ollama.model can be added.

OLLAMA.model is string array type to support multiple Ollama. Please see this section.

OLLAMA.host

Default: http://localhost:11434

The Ollama host

aicommit2 config set OLLAMA.host=<host>

OLLAMA.auth

Not required. Use when your Ollama server requires authentication. Please see this issue.

aicommit2 config set OLLAMA.auth=<auth>

OLLAMA.key

Not required. Use when your Ollama server requires authentication. Please see this issue.

aicommit2 config set OLLAMA.key=<key>

Few examples of authentication methods:

Authentication Method OLLAMA.auth OLLAMA.key
Bearer Bearer <API key>
Basic Basic <Base64 Encoded username:password>
JWT Bearer <JWT Token>
OAuth 2.0 Bearer <Access Token>
HMAC-SHA256 HMAC <Base64 Encoded clientId:signature>

OLLAMA.numCtx

The maximum number of tokens the model can process at once, determining its context length and memory usage. It is recommended to set it to 4096 or higher.

aicommit2 config set OLLAMA.numCtx=4096

Unsupported Options

Ollama does not support the following options in General Settings.

  • maxTokens

Loading Multiple Ollama Models

OLLAMA_PARALLEL

You can load and make simultaneous requests to multiple models using Ollama's experimental feature, the OLLAMA_MAX_LOADED_MODELS option.

  • OLLAMA_MAX_LOADED_MODELS: Load multiple models simultaneously

Setup Guide

Follow these steps to set up and utilize multiple models simultaneously:

1. Running Ollama Server

First, launch the Ollama server with the OLLAMA_MAX_LOADED_MODELS environment variable set. This variable specifies the maximum number of models to be loaded simultaneously. For example, to load up to 3 models, use the following command:

OLLAMA_MAX_LOADED_MODELS=3 ollama serve

Refer to configuration for detailed instructions.

2. Configuring aicommit2

Next, set up aicommit2 to specify multiple models. You can assign a list of models, separated by commas(,), to the OLLAMA.model environment variable. Here's how you do it:

aicommit2 config set OLLAMA.model="mistral,dolphin-llama3"

With this command, aicommit2 is instructed to utilize both the "mistral" and "dolphin-llama3" models when making requests to the Ollama server.

3. Run aicommit2
aicommit2

Note that this feature is available starting from Ollama version 0.1.33 and aicommit2 version 1.9.5.