Skip to content

Commit 5399315

Browse files
BinlogoBinlogo
authored
Update the docs for local models with ollama (#4)
1 parent d66e2e9 commit 5399315

File tree

6 files changed

+229
-2
lines changed

6 files changed

+229
-2
lines changed

‎docs/blog/why-aiscript.mdx‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -191,6 +191,12 @@ api_key = "YOUR_API_KEY"
191191

192192
[ai.anthropic]
193193
api_key = "YOUR_API_KEY"
194+
195+
# Local models with Ollama (no API key needed)
196+
[ai.ollama]
197+
api_endpoint = "http://localhost:11434/v1"
198+
model = "llama3.2"
199+
api_key = ""
194200
```
195201
</Tab>
196202
</Tabs>

‎docs/guide/ai/_meta.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
["prompt", "function", "agent"]
1+
["prompt", "function", "agent", "local-models"]

‎docs/guide/ai/local-models.mdx‎

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
import { Badge, Tab, Tabs } from "rspress/theme";
2+
3+
# Local Models with Ollama
4+
5+
AIScript supports running AI models locally through [Ollama](https://ollama.ai/), providing privacy, cost-effectiveness, and offline capabilities.
6+
7+
## Setting Up Ollama
8+
9+
### Installation
10+
11+
1. **Install Ollama** from [ollama.ai](https://ollama.ai/)
12+
2. **Start Ollama** service (runs on port 11434 by default)
13+
3. **Pull models** you want to use:
14+
15+
```bash
16+
# Popular general-purpose models
17+
ollama pull llama3.2
18+
19+
# Coding models
20+
ollama pull codellama
21+
ollama pull deepseek-coder
22+
23+
# Specialized models
24+
ollama pull deepseek-r1
25+
ollama pull gemma2
26+
ollama pull qwen2.5
27+
```
28+
29+
### Verification
30+
31+
Check that Ollama is running and models are available:
32+
33+
```bash
34+
# List installed models
35+
ollama list
36+
37+
# Test a model
38+
ollama run llama3.2 "Hello, how are you?"
39+
```
40+
41+
## Configuration
42+
43+
Configure AIScript to use Ollama models:
44+
45+
<Tabs>
46+
<Tab label="project.toml">
47+
48+
```toml
49+
[ai.ollama]
50+
api_endpoint = "http://localhost:11434/v1"
51+
model = "llama3.2" # Default model
52+
```
53+
54+
</Tab>
55+
<Tab label="Environment Variable">
56+
57+
```bash
58+
export OLLAMA_API_ENDPOINT="http://localhost:11434/v1"
59+
```
60+
61+
</Tab>
62+
</Tabs>
63+
64+
## Basic Usage
65+
66+
### Simple Prompts
67+
68+
```rust
69+
let response = prompt {
70+
input: "Explain how neural networks work",
71+
model: "llama3.2"
72+
};
73+
```
74+
75+
## Troubleshooting
76+
77+
### Common Issues
78+
79+
**Ollama not responding:**
80+
```bash
81+
# Check if Ollama is running
82+
curl http://localhost:11434/api/tags
83+
84+
# Restart Ollama service
85+
ollama serve
86+
```
87+
88+
**Model not found:**
89+
```bash
90+
# List available models
91+
ollama list
92+
93+
# Pull missing model
94+
ollama pull llama3.2
95+
```
96+
97+
**Out of memory:**
98+
- Try a smaller model (e.g., `phi3` instead of `llama3.1`)
99+
- Close other applications
100+
- Use quantized models
101+
102+
### Performance Issues
103+
104+
- **Slow responses**: Try smaller models or increase hardware resources
105+
- **High memory usage**: Monitor with `htop` or Activity Monitor
106+
- **GPU not utilized**: Ensure GPU drivers are properly installed
107+
108+
## Comparison: Local vs Cloud Models
109+
110+
| Aspect | Local Models | Cloud Models |
111+
|--------|-------------|--------------|
112+
| **Privacy** | ✅ Complete | ❌ Data sent externally |
113+
| **Cost** | ✅ One-time setup | ❌ Per-token billing |
114+
| **Internet** | ✅ Works offline | ❌ Requires connection |
115+
| **Latency** | ✅ Low (local) | ❌ Network dependent |
116+
| **Quality** | ⚠️ Model dependent | ✅ Usually higher |
117+
| **Maintenance** | ❌ Self-managed | ✅ Fully managed |
118+
| **Scaling** | ❌ Hardware limited | ✅ Unlimited |
119+
120+
Local models with Ollama provide a powerful way to integrate AI capabilities into your AIScript applications while maintaining full control over your data and costs.

‎docs/guide/ai/prompt.mdx‎

Lines changed: 47 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ let response = prompt f"Explain {topic} at a {detail_level} level";
5151

5252
## API Keys and Configuration
5353

54-
To use the `prompt` feature, you need to configure your API keys. AIScript supports multiple LLM providers.
54+
To use the `prompt` feature, you need to configure your API keys. AIScript supports multiple LLM providers including cloud-based services and local models.
5555

5656
### Environment Variables
5757

@@ -63,6 +63,9 @@ export OPENAI_API_KEY="your_openai_key"
6363

6464
# For Anthropic Claude
6565
export CLAUDE_API_KEY="your_claude_key"
66+
67+
# For Ollama (local models)
68+
export OLLAMA_API_ENDPOINT="http://localhost:11434/v1"
6669
```
6770

6871
### Configuration File
@@ -78,10 +81,47 @@ api_key = "YOUR_OPENAI_API_KEY"
7881

7982
[ai.anthropic]
8083
api_key = "YOUR_CLAUDE_API_KEY"
84+
85+
# For Ollama (local models)
86+
[ai.ollama]
87+
api_endpoint = "http://localhost:11434/v1"
88+
model = "llama3.2" # or any other model installed in your Ollama instance
8189
```
8290
</Tab>
8391
</Tabs>
8492

93+
## Local Models with Ollama
94+
95+
AIScript supports running local models through [Ollama](https://ollama.ai/), which allows you to run AI models on your own hardware without sending data to external services.
96+
97+
### Setting Up Ollama
98+
99+
1. **Install Ollama** from [ollama.ai](https://ollama.ai/)
100+
2. **Pull models** you want to use:
101+
```bash
102+
ollama pull llama3.2
103+
ollama pull deepseek-r1
104+
ollama pull codellama
105+
```
106+
3. **Start Ollama** (it runs on port 11434 by default)
107+
4. **Configure AIScript** to use Ollama
108+
109+
### Example with Local Models
110+
111+
```rust
112+
// Using a local Llama model
113+
let response = prompt {
114+
input: "What is rust?",
115+
model: "llama3.2"
116+
};
117+
118+
// Using a local coding model
119+
let code_explanation = prompt {
120+
input: "Review this Python function for potential issues",
121+
model: "codellama"
122+
};
123+
```
124+
85125
## Error Handling <Badge text="Not supported yet" type="warning" />
86126

87127
Like other AIScript operations, you can use error handling with the `prompt` keyword:
@@ -135,6 +175,12 @@ let claude_response = prompt {
135175
input: f"Summarize this article: {article_text}" ,
136176
model: "claude-3.7"
137177
};
178+
179+
// Using Ollama (local models)
180+
let local_response = prompt {
181+
input: f"Summarize this article: {article_text}",
182+
model: "llama3.2"
183+
};
138184
```
139185

140186
### Chaining Prompts

‎docs/guide/getting-started/introduction.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,21 @@ You can open [http://localhost:8080/redoc](http://localhost:8080/redoc) to see t
7878

7979
![](/guide/open-api.png)
8080

81+
### Local Models Support
82+
83+
AIScript also supports local models through [Ollama](https://ollama.ai/), allowing you to run AI models on your own hardware:
84+
85+
```javascript
86+
$ ollama pull llama3.2
87+
$ export OLLAMA_API_ENDPOINT=http://localhost:11434/v1
88+
$ cat local.ai
89+
let a = prompt {
90+
input: "What is rust?",
91+
model: "llama3.2"
92+
};
93+
print(a);
94+
```
95+
8196
## Use Cases
8297

8398
AIScript excels in these scenarios:

‎docs/reference/configuration/ai.md‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,15 @@
22

33
> For AI observability, please refer to [observability](./observability.md).
44
5+
AIScript supports multiple AI providers:
6+
7+
- **OpenAI**: Cloud-based models including GPT-4 and GPT-3.5
8+
- **DeepSeek**: Efficient and powerful models
9+
- **Anthropic**: Claude models for advanced reasoning
10+
- **Ollama**: 100+ local models from various providers running on your own hardware
11+
12+
## Configuration
13+
514
```toml
615
[ai.openai]
716
api_key = "YOUR_API_KEY"
@@ -20,4 +29,35 @@ api_key = "YOUR_API_KEY"
2029
# optional, default is the official Anthropic API endpoint
2130
api_endpoint = "YOUR_API_ENDPOINT"
2231
model = "claude-3-5-sonnet-latest"
32+
33+
# Ollama - Local models
34+
[ai.ollama]
35+
api_endpoint = "http://localhost:11434/v1" # Default Ollama endpoint
36+
model = "llama3.2" # or any other model installed in your Ollama instance
37+
```
38+
39+
## Using Ollama
40+
41+
[Ollama](https://ollama.ai/) allows you to run local AI models on your own hardware. To use Ollama with AIScript:
42+
43+
1. **Install Ollama** from [ollama.ai](https://ollama.ai/)
44+
2. **Pull your desired models** (e.g., `ollama pull llama3.2`)
45+
3. **Make sure Ollama is running** locally
46+
4. **Configure AIScript** to use Ollama as shown above or by setting the `OLLAMA_API_ENDPOINT` environment variable
47+
48+
### Available Models
49+
50+
Ollama provides access to 100+ models ranging from small 135M parameter models to massive 671B parameter models, including:
51+
52+
- **Llama family**: `llama3.2`, `llama3.1`, `codellama`
53+
- **DeepSeek models**: `deepseek-r1`, `deepseek-v3`, `deepseek-coder`
54+
- **Specialized models**: `phi3`, `gemma2`, `qwen2.5`, `mistral`
55+
- **And many more**: See the [full list of available models](https://ollama.com/search)
56+
57+
### Environment Variables
58+
59+
You can also configure Ollama using environment variables:
60+
61+
```bash
62+
export OLLAMA_API_ENDPOINT="http://localhost:11434/v1"
2363
```

0 commit comments

Comments
 (0)