ComputerUseAgent Agent

An intelligent automation Agent framework with LLM capabilities, supporting task planning, RPA (Robotic Process Automation) execution, and workflow orchestration for macOS and Windows.

Features

Task Planning: Automatic task decomposition and planning using LLM
RPA Execution: Complete set of mouse, keyboard, screen, and system automation tools
Workflow Engine: Flexible node-based workflow orchestration with evaluation capabilities
Multi-LLM Support: Compatible with OpenAI, Qwen, and Gemini models
Visual Intelligence: Screenshot-based action planning and evaluation
Evaluation Mode: Compare before/after screenshots to verify task completion

Quick Start

1. Requirements

Go 1.24+
macOS (required for RPA features)

2. Install Dependencies

go mod download

3. Configuration

The config.toml configuration file is optional. The application includes built-in default configurations. The config file only needs to contain fields you want to override:

[app]
name = "agent"
version = "1.0.0"
mode = "production"
debug = false
task = "Open Safari and search for cat pictures"
max_steps = 100
screen_index = 0
is_plan = false
is_evaluation = false

[llm]
provider = "qwen"  # Supported: openai, qwen, gemini
api_key = "${OPENAI_API_KEY}"  # Supports environment variables
base_url = "http://127.0.0.1:1234/v1"
model = "qwen/qwen3-vl-4b"
timeout = 30

Note: Fields in the config file override default settings. Unconfigured fields use default values.

4. Run

# Using Makefile
make run

# Or run directly
go run .

# Specify task
go run . -task "Open calculator and calculate 1+1"

# Plan only mode (no execution)
go run . -task "Search weather forecast" -plan

# Enable evaluation mode
go run . -task "Open Notes app" -eval=true

# Set maximum execution steps
go run . -task "Automated testing workflow" -max-steps=50

# Specify screen index
go run . -task "Open Safari on second display" -screen-index=1

5. Build

make build
./build/agent -task "Your task description"

Command Line Arguments

Parameter	Description	Default
`-help`	Show help information	-
`-version`	Show version information	-
`-task`	Task description to execute	Read from config file
`-plan`	Plan only mode, no execution	Read from config file (default false)
`-eval`	Enable evaluation mode	Read from config file (default false)
`-max-steps`	Maximum execution steps	Read from config file (default 100)
`-screen-index`	Working screen index	Read from config file (default 0)

Note: All parameters can be set via config file with default values. Command line arguments override config file values.

Examples:

# View all parameters
go run . -h

# Complete example
./build/agent -task "Open Safari and visit apple.com" -plan=false -eval=true -max-steps=20 -screen-index=0

Configuration Details

LLM Configuration

Supports multiple LLM providers. Configure in config.toml (optional, uses defaults if not configured):

[llm]
# Provider: openai, qwen, gemini
provider = "qwen"

# Model name
model = "qwen/qwen3-vl-4b"

# API key (supports environment variables like ${OPENAI_API_KEY})
api_key = ""

# Custom API endpoint (optional, for proxy or local deployment)
base_url = "http://127.0.0.1:1234/v1"

# Request timeout (seconds)
timeout = 30

Supported Provider Examples:

OpenAI: provider = "openai", base_url = "https://api.openai.com/v1"
Qwen: provider = "qwen", base_url = "https://dashscope.aliyuncs.com/api/v1"
Gemini: provider = "gemini"

Using LM Studio for Local Models

To use local LLM models, we recommend LM Studio:

Install LM Studio
- Visit https://lmstudio.ai/ to download and install
- Supports macOS, Windows, and Linux
Start Local Server
- Open LM Studio
- Select and download a model (recommended: Qwen2.5-7B, Llama-3-8B, etc.)
- Click the "Server" tab to start the local API server
- Default port is 1234, API address is http://127.0.0.1:1234/v1

Configure config.toml

[llm]
provider = "openai"  # LM Studio uses OpenAI-compatible API
api_key = ""        # Can be empty for local deployment
base_url = "http://127.0.0.1:1234/v1"
model = "Downloaded model name"

Advantages
- Completely offline, privacy protected
- No API call costs
- Customizable model parameters (temperature, top_p, etc.)
- Support for multiple open-source models

RPA Configuration

The following settings can be modified in config.toml. Unconfigured fields use defaults:

[rpa]
# Enable RPA logging
enable_logging = false

# Log level: debug, info, warn, error
log_level = "info"

# Action timeout (seconds)
action_timeout = 30

# Smooth mouse movement
smooth_movement = true

# Mouse movement speed (pixels/millisecond)
movement_speed = 5

# Click visual indicator pause time (milliseconds)
click_visual_sleep = 500

# Scroll step interval (milliseconds)
scroll_step_sleep = 20

# Wait time after app launch (seconds)
app_launch_sleep = 1

App Configuration

[app]
name = "mac-use-agent"
version = "1.0.0"

# Run mode: development or production
mode = "production"

# Debug mode (enables verbose logging)
debug = false

# Default task
task = "Open Safari and download a cat picture to desktop"

# Maximum execution steps
max_steps = 100

# Working screen index
screen_index = 0

# Enable planning mode
is_plan = false

# Enable evaluation mode
is_evaluation = false

Logging Configuration

[logging]
# Log level: debug, info, warn, error
level = "info"

# Format: json, text
format = "json"

# Output: stdout, stderr, or file path
output = "stdout"

Server Configuration

[server]
host = "localhost"
port = 8080

RPA Tools

The Agent provides the following automation tools:

Mouse Tools

Tool	Description
`left_click`	Left click at specified coordinates
`right_click`	Right click at specified coordinates
`double_click`	Double click at specified coordinates
`move`	Move mouse cursor to coordinates
`drag`	Drag and drop operation
`scroll`	Scroll in specified direction with amount
`mouse_position`	Get current mouse position

Keyboard Tools

Tool	Description
`type_text`	Input text string
`key_press`	Press keyboard key or combination

Screen Tools

Tool	Description
`screenshot`	Capture screen screenshot (base64 encoded)
`screen_size`	Get screen dimensions

System Tools

Tool	Description
`wait`	Wait for specified duration
`open_app`	Open application by name
`get_app_list`	List running applications
`done`	Mark task as completed

macOS Specific Tools

Tool	Description
`apple_script`	Execute AppleScript commands

Workflow Engine

The Agent uses a node-based workflow engine for task orchestration:

Workflow Nodes

PlannerNode: Creates a step-by-step plan from the main goal
GetActionNode: Determines the next actions based on current state
RPAExecutorNode: Executes the planned RPA actions
EvaluatorNode: Evaluates task completion by comparing screenshots
CompletionNode: Finalizes the workflow

Workflow Flow

PlannerNode
    ↓
GetActionNode
    ↓
RPAExecutorNode
    ↓
EvaluatorNode → GetActionNode (continue)
    ↓
CompletionNode (completed)

Evaluation Mode

When evaluation mode is enabled, the agent:

Takes a screenshot before executing actions
Executes the planned actions
Takes another screenshot after execution
Compares the two screenshots using the LLM
Determines if the step was completed successfully
Continues or marks as completed based on evaluation

Usage Examples

Example 1: Automated Web Search

./build/agent -task "Open Safari browser, search today's weather forecast, and save a screenshot"

Example 2: Application Automation

./build/agent -task "Open calculator, input 123 + 456, get result"

Example 3: Multi-step Task

./build/agent -task "Open Finder, create new folder named 'test', open TextEdit and write 'Hello World', save to new folder" -max-steps=30

Example 4: Plan Only Without Execution

./build/agent -task "Analyze system performance and generate report" -plan

Example 5: Evaluation Mode

./build/agent -task "Download a picture of a cat to the desktop" -eval=true

Testing

# Run all tests
make test

# Or
go test ./...

# Run specific tests
go test -v ./internal/core/llm -run TestMockLLMClient

# Generate coverage report
make test-coverage

# Run race detection
make test-race

# Run benchmarks
make bench

Makefile Commands

Command	Description
`make build`	Build application (automatically injects version info)
`make run`	Run application
`make clean`	Clean build files and test reports
`make test`	Run tests
`make test-coverage`	Run tests and generate coverage report
`make test-race`	Run race detection
`make bench`	Run benchmark tests
`make fmt`	Format code
`make vet`	Run go vet
`make lint`	Run code checks (requires golangci-lint)
`make deps`	Download dependencies
`make deps-update`	Update dependencies
`make install`	Install to GOPATH/bin
`make help`	Display all available commands

# View all commands
make help

Development

Code Standards

Run make fmt to format code with gofmt
Run make vet to check code with go vet
Run make lint for code quality checks with golangci-lint

Environment Variables

You can set the LLM API key via environment variable:

export OPENAI_API_KEY="your-api-key"
go run .

Or use it in config.toml:

[llm]
api_key = "${OPENAI_API_KEY}"

Reference Code

Agent Implementation: /internal/core/workflow/agent.go
Workflow Nodes: /internal/core/workflow/node.go
LLM Client: /internal/core/llm/
RPA Server: /internal/core/rpa/rpa.go
Configuration: /internal/config/config.go

FAQ

Q: Is the config.toml file required?

A: No. The application includes complete default configurations. config.toml is only used to override fields you want to modify. If there is no config.toml file, the application will run automatically with default settings.

Q: How to use local LLM models?

A: Modify the base_url in the config file to point to your local service, for example:

base_url = "http://127.0.0.1:1234/v1"

Q: What to do if Agent execution gets stuck?

A: Use the -max-steps parameter to limit maximum execution steps, or use Ctrl+C to interrupt execution.

Q: Does evaluation mode work with screenshots?

A: Yes. When evaluation mode is enabled, the agent takes screenshots before and after each action, then uses the LLM to compare them and determine if the task was completed successfully.

Q: Which operating systems are supported?

A: RPA features currently only support macOS (using CGO and AppleScript). Core LLM functionality can run on any platform.

Q: How to view detailed execution logs?

A: Set in the config file:

[app]
mode = "development"  # Enable debug mode
debug = true

[logging]
level = "debug"

Limitations

RPA tools currently only support macOS (requires CGO and AppleScript)
Cross-platform compilation not supported (CGO dependencies)
Requires display access for screenshot and automation features

Roadmap

Improve OpenAI and Gemini clients
Support Windows and Linux RPA features
Enhance LLM features (streaming output, function calling)
Web UI workflow management interface
Performance monitoring and metrics
Distributed execution support
Advanced screenshot comparison algorithms
Custom tool registration API

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

License

This project is licensed under the MIT License - see the LICENSE file for details.

FilesExpand file tree

README.md

Latest commit

History