Open-source, high-performance LLM gateway written in Rust. Connect to any LLM provider with a single API. Observability Included.

Get started »

Slack | Docs

Hub is a next generation smart proxy for LLM applications. It centralizes control and tracing of all LLM calls and traces. It's built in Rust so it's fast and efficient. It's completely open-source and free to use.

Built and maintained by Traceloop under the Apache 2.0 license.

🚀 Getting Started

Make sure to copy a config.yaml file from config-example.yaml and set the correct values, following the configuration instructions.

You can then run the hub using the docker image:

docker run --rm -p 3000:3000 -v $(pwd)/config.yaml:/etc/hub/config.yaml:ro -e CONFIG_FILE_PATH='/etc/hub/config.yaml'  -t traceloop/hub

You can also run it locally. Make sure you have rust v1.82 and above installed and then run:

cargo run

Connect to the hub by using the OpenAI SDK on any language, and setting the base URL to:

http://localhost:3000/api/v1

For example, in Python:

client = OpenAI(
    base_url="http://localhost:3000/api/v1",
    api_key=os.getenv("OPENAI_API_KEY"),
    # default_headers={"x-traceloop-pipeline": "azure-only"},
)
completion = client.chat.completions.create(
    model="claude-3-5-sonnet-20241022",
    messages=[{"role": "user", "content": "Tell me a joke about opentelemetry"}],
    max_tokens=1000,
)

🌱 Contributing

Whether big or small, we love contributions ❤️ Check out our guide to see how to get started.

Not sure where to get started? You can:

• Book a free pairing session with one of our teammates!
• Join our Slack, and ask us any questions there.

💚 Community & Support

• Slack (For live discussion with the community and the Traceloop team)
• GitHub Discussions (For help with building and deeper conversations about features)
• GitHub Issues (For any bugs and errors you encounter using OpenLLMetry)
• Twitter (Get news fast)

Hub

A unified API interface for routing LLM requests to various providers.

Supported Providers

• OpenAI
• Anthropic
• Azure OpenAI
• Google VertexAI (Gemini)

Configuration

See config-example.yaml for a complete configuration example.

Provider Configuration

OpenAI

providers:
  - key: openai
    type: openai
    api_key: "<your-openai-api-key>"

Azure OpenAI

providers:
  - key: azure-openai
    type: azure
    api_key: "<your-azure-api-key>"
    resource_name: "<your-resource-name>"
    api_version: "<your-api-version>"

Google VertexAI (Gemini)

providers:
  - key: vertexai
    type: vertexai
    api_key: "<your-gcp-api-key>"
    project_id: "<your-gcp-project-id>"
    location: "<your-gcp-region>"
    credentials_path: "/path/to/service-account.json"

Authentication Methods:

1. API Key Authentication:
   • Set the api_key field with your GCP API key
   • Leave credentials_path empty
2. Service Account Authentication:
   • Set credentials_path to your service account JSON file path
   • Can also use GOOGLE_APPLICATION_CREDENTIALS environment variable
   • Leave api_key empty when using service account auth

Supported Features:

• Chat Completions (with Gemini models)
• Text Completions
• Embeddings
• Streaming Support
• Function/Tool Calling
• Multi-modal Inputs (images + text)

Example Model Configuration:

models:
  # Chat and Completion model
  - key: gemini-1.5-flash
    type: gemini-1.5-flash
    provider: vertexai
  
  # Embeddings model
  - key: textembedding-gecko
    type: textembedding-gecko
    provider: vertexai

Example Usage with OpenAI SDK:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:3000/api/v1",
    api_key="not-needed-for-vertexai"
)

# Chat completion
response = client.chat.completions.create(
    model="gemini-1.5-flash",
    messages=[{"role": "user", "content": "Tell me a joke"}]
)

# Embeddings
response = client.embeddings.create(
    model="textembedding-gecko",
    input="Sample text for embedding"
)

Pipeline Configuration

pipelines:
  - name: default
    type: chat
    plugins:
      - model-router:
          models:
            - gemini-pro

Development

Running Tests

The test suite uses recorded HTTP interactions (cassettes) to make tests reproducible without requiring actual API credentials.

To run tests:

cargo test

To record new test cassettes:

1. Set up your API credentials:
   • For service account auth: Set VERTEXAI_CREDENTIALS_PATH to your service account key file path
   • For API key auth: Use the test with API key (currently marked as ignored)
2. Delete the existing cassette files in tests/cassettes/vertexai/
3. Run the tests with recording enabled:

RECORD_MODE=1 cargo test

Additional test configurations:

• RETRY_DELAY: Set the delay in seconds between retries when hitting quota limits (default: 60)
• Tests automatically retry up to 3 times when hitting quota limits

Note: Some tests may be marked as #[ignore] if they require specific credentials or are not ready for general use.

License

See LICENSE file.