Strands MLX Integration

A demonstration of integrating Mac-native MLX models with the Strands agent framework, enabling local LLM inference using Apple Silicon GPU acceleration.

Overview

This project implements a custom MacNativeModel adapter that allows the Strands agent framework to use locally-running MLX models. Instead of relying on cloud-based APIs, this enables private, offline AI inference directly on your Mac's GPU.

Features

• Local GPU-accelerated inference using MLX
• Integration with Strands agent framework
• Support for quantized models (4-bit) for efficient memory usage
• Async/await support for non-blocking inference
• Compatible with Llama 3.2 and other MLX-compatible models

Prerequisites

• macOS with Apple Silicon (M1/M2/M3/M4)
• Python 3.13+
• uv (Python package manager)

Installation

1. Clone the repository:

git clone <repository-url>
cd strands

2. Install dependencies using uv:

uv sync

Usage

Run the example:

uv run python main.py

The script will:

1. Load the Llama-3.2-3B-Instruct-4bit model onto your Mac's GPU
2. Create a Strands agent with the local model
3. Execute a test prompt
4. Display the AI response

How It Works

The MacNativeModel class implements the Strands Model interface with the following key methods:

• __init__: Loads the MLX model and tokenizer
• stream: Handles inference with proper event streaming for Strands
• Message formatting compatible with Strands' content block structure
• Async thread execution to prevent blocking

Dependencies

• strands-agents: Core agent framework
• mlx-lm: Apple MLX for local model inference
• apple-foundation-models: Apple's ML foundation tools

Model Configuration

Default model: mlx-community/Llama-3.2-3B-Instruct-4bit

To use a different model, modify the model_path in main.py:

model_path = "mlx-community/your-model-name"
native_model = MacNativeModel(model_id=model_path)