Skip to content

setup.md

Latest commit

 

History

History
140 lines (108 loc) · 3.29 KB

setup.md

File metadata and controls

140 lines (108 loc) · 3.29 KB
Raw

Installation & Setup

Prerequisites

Requirement Version Check Command
Python 3.12+ python --version
Docker 24.0+ docker --version
Git 2.30+ git --version
NVIDIA Driver 535+ (if GPU) nvidia-smi

Quick Start

# 1. Clone repository
git clone --recursive https://github.com/KatherLab/ToolMaker
cd ToolMaker

# 2. Install dependencies
uv sync

# 3. Install Ollama and pull model
# macOS: brew install ollama
# Linux: curl -fsSL https://ollama.com/install.sh | sh
# Windows: Download from https://ollama.com/download
ollama serve &
ollama pull qwen2.5-coder:7b

# 4. Configure environment
cat > .env << 'EOF'
TOOLMAKER_LLM_BACKEND=ollama
TOOLMAKER_MODEL=qwen2.5-coder:7b
OLLAMA_BASE_URL=http://localhost:11434
EOF

# 5. Verify setup
uv run python verify_setup.py

Docker Setup

# Pull ToolMaker images
docker pull ghcr.io/katherlab/toolmaker:cpu
docker pull ghcr.io/katherlab/toolmaker:cuda  # If GPU available

# Test Docker
docker run --rm ghcr.io/katherlab/toolmaker:cpu python --version

GPU Support (Optional)

# Install NVIDIA Container Toolkit (Ubuntu/Debian)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# Test GPU access
docker run --rm --gpus all ghcr.io/katherlab/toolmaker:cuda nvidia-smi

LLM Backend Options

Ollama (Recommended for Development)

ollama serve
ollama pull qwen2.5-coder:7b   # 8GB VRAM
ollama pull qwen2.5-coder:14b  # 16GB VRAM
ollama pull qwen2.5-coder:32b  # 24GB+ VRAM
TOOLMAKER_LLM_BACKEND=ollama
TOOLMAKER_MODEL=qwen2.5-coder:7b
OLLAMA_BASE_URL=http://localhost:11434

vLLM (Recommended for Production)

pip install vllm
vllm serve Qwen/Qwen2.5-Coder-32B-Instruct --port 8000
TOOLMAKER_LLM_BACKEND=vllm
TOOLMAKER_MODEL=Qwen/Qwen2.5-Coder-32B-Instruct
VLLM_BASE_URL=http://localhost:8000/v1

llama.cpp (CPU/Low Resource)

./llama-server -m qwen2.5-coder-7b-instruct-q4_k_m.gguf -c 32768 --port 8080
TOOLMAKER_LLM_BACKEND=openai_compatible
TOOLMAKER_MODEL=qwen2.5-coder
OPENAI_API_BASE=http://localhost:8080/v1
OPENAI_API_KEY=dummy

Hardware Requirements

Setup RAM VRAM Notes
Minimal (CPU) 32GB - Slow, llama.cpp with Q4
Development 16GB 8GB Ollama + 7B model
Recommended 32GB 24GB Ollama + 32B model
Production 64GB 48GB+ vLLM + multi-GPU

Conda Environment (Alternative)

conda create -n daedalus python=3.12 -y
conda activate daedalus
pip install -e .
pip install pytest pytest-asyncio pytest-mock

Verify Installation

# Run verification script
uv run python verify_setup.py

# Run tests
uv run pytest tests/ -v

# Test CLI
uv run python -m toolmaker --help