Cylestio Gateway

A configurable Python proxy server for LLM API requests with interceptor support, built with FastAPI.

✨ Features

Core Functionality

🔄 LLM Provider Support: Proxy requests to OpenAI, Anthropic, and other LLM providers
📡 Streaming Support: Handle Server-Sent Events (SSE) for real-time responses
📊 Request Tracing: Capture and save request/response data to JSON files
🔍 Session Management: Intelligent session detection using message history hashing
🏷️ External ID Support: Custom session and agent IDs via x-cylestio-* headers
⚙️ Interceptor System: Extensible interceptors for tracing, logging, and analysis
💻 CLI Interface: Simple command-line interface with configuration file support
🐳 Docker Support: Ready-to-use Docker containers
📈 Metrics Endpoint: Monitor proxy performance and session statistics

Live Trace & Security

🖥️ Live Trace Dashboard: Real-time web UI for monitoring agent sessions and events
🔒 Security Analysis: OWASP LLM Top 10 pattern detection and risk scoring
🕵️ PII Detection: Automatic detection of personally identifiable information
🤖 MCP Integration: AI assistant integration via Model Context Protocol

Quick Start

Installation

Install through pip:
```
pip install cylestio-perimeter
```

For Developers: Install directly from source code:

git clone https://github.com/cylestio/cylestio-perimeter.git
cd cylestio-perimeter

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install the package in development mode
pip install -e .

Run the server:

# With CLI arguments
cylestio-perimeter run --base-url https://api.openai.com --type openai --api-key sk-your-key

# With config file
cylestio-perimeter run --config config.yaml

Or run with config file:
```
python -m src.main --config config.yaml
```

Docker Usage

Using docker-compose (recommended):

# Set environment variables
export LLM_BASE_URL=https://api.openai.com
export LLM_TYPE=openai
export LLM_API_KEY=sk-your-key-here

# Start the service
docker-compose up -d

Using Docker directly:

docker build -t llm-proxy .
docker run -p 4000:4000 -e LLM_BASE_URL=https://api.openai.com -e LLM_TYPE=openai -e LLM_API_KEY=sk-your-key llm-proxy

Usage Examples

Basic Proxy Usage

# Start proxy server
cylestio-perimeter run --base-url https://api.openai.com --type openai --api-key sk-your-key

# Make requests to the proxy
curl -X POST http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-3.5-turbo", "messages": [{"role": "user", "content": "Hello!"}]}'

Streaming Requests

curl -X POST http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-3.5-turbo", "messages": [{"role": "user", "content": "Hello!"}], "stream": true}'

With Configuration File

# config.yaml
server:
  port: 4000
  host: "0.0.0.0"

llm:
  base_url: "https://api.openai.com"
  type: "openai"
  api_key: "sk-your-key-here"

interceptors:
  - type: "printer"
    enabled: true
    config:
      log_requests: true
      log_responses: true

External ID Headers

The gateway supports custom session and agent identification via headers:

# Custom session ID
curl -X POST http://localhost:4000/v1/chat/completions \
  -H "x-cylestio-session-id: my-session-123" \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-4", "messages": [{"role": "user", "content": "Hello!"}]}'

# Custom agent ID
curl -X POST http://localhost:4000/v1/chat/completions \
  -H "x-cylestio-agent-id: math-tutor-v2" \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-4", "messages": [{"role": "user", "content": "What is 2+2?"}]}'

# Both custom session and agent ID
curl -X POST http://localhost:4000/v1/chat/completions \
  -H "x-cylestio-session-id: user-session-456" \
  -H "x-cylestio-agent-id: customer-support-bot" \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-4", "messages": [{"role": "user", "content": "Help me reset my password"}]}'

See External Agent ID Documentation for complete details.

CLI Commands

The CLI supports several subcommands for different operations:

Run the Server

cylestio-perimeter run --base-url https://api.openai.com --type openai --api-key sk-your-key
cylestio-perimeter run --config config.yaml

Generate Example Config

cylestio-perimeter generate-config example.yaml

Validate Configuration

cylestio-perimeter validate-config config.yaml

Replay Recorded Traffic

cylestio-perimeter replay recordings/ --config config.yaml --delay 0.5

Get Help

cylestio-perimeter --help
cylestio-perimeter run --help

Development Mode

uvicorn src.main:app --reload --port 4000

Configuration

CLI Options

--base-url: Base URL of target LLM API (required)
--type: LLM provider type (required)
--api-key: API key to inject into requests
--port: Proxy server port (default: 4000)
--host: Server host (default: 0.0.0.0)
--log-level: Logging level (INFO, DEBUG, etc.)
--config: Path to YAML configuration file

Interceptor Configuration

Available interceptors that can be enabled in your config file:

Interceptor	Description
`printer`	Logs requests/responses to console
`message_logger`	Saves LLM conversations to JSONL files
`event_recorder`	Records raw events per session
`http_recorder`	Records HTTP traffic for replay testing
`cylestio_trace`	Sends events to Cylestio platform
`live_trace`	Real-time dashboard with security analysis

Printer Interceptor

interceptors:
  - type: "printer"
    enabled: true
    config:
      log_requests: true
      log_responses: true
      log_body: false

Live Trace Interceptor

interceptors:
  - type: "live_trace"
    enabled: true
    config:
      server_port: 7100
      auto_open_browser: true
      storage_mode: "sqlite"  # "memory" (default) or "sqlite" for persistence
      enable_presidio: true   # PII detection

Testing

# Install dev dependencies
pip install -r requirements-dev.txt

# Run tests
pytest

# Run with coverage
pytest --cov=src

# Run specific tests
pytest tests/test_config.py -v

API Endpoints

GET /health - Health check endpoint
GET /metrics - Metrics endpoint with session statistics
GET /config - Current server configuration and interceptor status
/{path:path} - Catch-all proxy route (all HTTP methods)

Live Trace endpoints (when enabled):

GET /api/dashboard - Dashboard data with agents and sessions
GET /api/agent/{id} - Agent details with risk analysis
GET /api/session/{id} - Session timeline with events
POST /mcp - MCP protocol endpoint (SSE) for AI assistants

Session Management

The proxy includes intelligent session detection that tracks conversations across multiple requests:

Hash-based Tracking: Uses message history hashing to identify unique conversations
LRU Cache: Maintains up to 10,000 sessions with automatic eviction
Session TTL: Sessions expire after 1 hour of inactivity
Fuzzy Matching: Detects continued conversations even with slight variations
Multiple Heuristics: Identifies new sessions based on message count and reset phrases

Session Configuration

session:
  enabled: true
  max_sessions: 10000
  session_ttl_seconds: 3600

Monitoring Sessions

Access session metrics via the /metrics endpoint:

curl http://localhost:4000/metrics

Response includes:

Active sessions count
Cache hit/miss rates
Session creation rate
Fuzzy match statistics

Live Trace Dashboard

When the live_trace interceptor is enabled, a real-time web dashboard is available for monitoring agent activity.

Features

Real-time Monitoring: View active sessions and events as they happen
Security Analysis: Automatic OWASP LLM Top 10 pattern detection
PII Detection: Identify personally identifiable information in requests/responses
Risk Scoring: Aggregate security scores for agents and sessions
MCP Integration: AI assistants can query data via the /mcp endpoint

Quick Start

interceptors:
  - type: "live_trace"
    enabled: true
    config:
      server_port: 7100        # Dashboard port
      auto_open_browser: true  # Open browser on startup
      storage_mode: "sqlite"   # Persistent storage

Access the dashboard at http://localhost:7100 after starting the server.

Environment Variables

LLM_BASE_URL - Base URL for LLM provider
LLM_TYPE - LLM provider type
LLM_API_KEY - API key for authentication
LOG_LEVEL - Logging level (INFO, DEBUG, etc.)

Security

Cylestio Gateway implements comprehensive security measures to ensure safe deployment in enterprise environments.

Security Measures:

Automated Vulnerability Scanning: Every release is scanned for known security issues
Dependency Security: All third-party packages are continuously monitored for vulnerabilities
Static Code Analysis: Source code is analyzed for security vulnerabilities using industry-standard tools
Secret Detection: Pre-commit hooks prevent accidental credential exposure
Supply Chain Security: Complete Software Bill of Materials (SBOM) provides full transparency
License Compliance: Automated scanning ensures only approved licenses are used

Documentation: Security Policy

Development

Setting Up Development Environment

# Install development dependencies
pip install -r requirements-dev.txt

# Set up pre-commit hooks (includes security scanning)
pre-commit install

# Run tests with coverage
pytest --cov=src

# Run security checks locally
pre-commit run --all-files

Security Development Practices

Never commit secrets - Use environment variables for all credentials
Run pre-commit hooks - Automated security checks before each commit
Review security reports - Check CI security scan results
Follow secure coding - Follow standard Python security best practices

See CLAUDE.md for detailed development guidance and architecture information.

License

This project is developed according to the specifications in INSTRUCTIONS.md.

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

Cylestio Gateway

✨ Features

Core Functionality

Live Trace & Security

Quick Start

Installation

Docker Usage

Usage Examples

Basic Proxy Usage

Streaming Requests

With Configuration File

External ID Headers

CLI Commands

Run the Server

Generate Example Config

Validate Configuration

Replay Recorded Traffic

Get Help

Development Mode

Configuration

CLI Options

Interceptor Configuration

Printer Interceptor

Live Trace Interceptor

Testing

API Endpoints

Session Management

Session Configuration

Monitoring Sessions

Live Trace Dashboard

Features

Quick Start

Environment Variables

Security

Development

Setting Up Development Environment

Security Development Practices

License