diff --git a/docs/.env.md b/docs/.env.md new file mode 100644 index 0000000..9777485 --- /dev/null +++ b/docs/.env.md @@ -0,0 +1,113 @@ +# Documentation for xAI Backend Integration + +This documentation covers a recent code change that introduces new environment variables for configuring an xAI (Grok) backend, enhances LLM (Large Language Model) backend auto-detection, and updates related configurations for better flexibility and cost-efficiency. The changes aim to allow users to seamlessly switch between different LLM providers (e.g., Anthropic, xAI, or Ollama) while prioritizing cost-effective options like xAI. + +## Overview + +### What This Does and Why It Matters +This change adds support for the xAI backend (powered by Grok) as an alternative to existing options like Anthropic (Claude) and Ollama. It introduces new environment variables such as `TOME_WEBHOOK_SECRET`, `XAI_API_KEY`, and `TOME_XAI_MODEL` in the `.env.example` file. These variables enable: + +- **Integration with LLM backend auto-detection**: The application automatically selects an LLM backend based on which API keys are provided, falling back to a default order (Anthropic > xAI > Ollama). This reduces manual configuration and ensures the app uses the most appropriate backend. +- **Cost-efficiency**: xAI is recommended for users seeking affordable AI inference, as it offers competitive pricing compared to other providers. By setting up xAI, users can leverage its models for tasks like prompt generation without incurring high costs. +- **Enhanced security and flexibility**: Variables like `TOME_WEBHOOK_SECRET` improve webhook security, while the new setup allows for easier switching between backends based on availability or performance needs. + +This update matters because it makes the application more versatile, cost-effective, and user-friendly, especially for developers working with AI-driven features in a production environment. + +### Key Changes +- **.env.example**: A new file with examples for all environment variables, including the new ones for xAI. +- **config.py**: Updated the `LLM_BACKEND` logic to include "xai" in auto-detection. +- **app.py and engine.py**: Added checks and functions to handle the xAI backend, ensuring it integrates with the existing LLM workflow. + +## New Environment Variables + +The following variables have been added or updated in `.env.example`. They control various aspects of the application, with a focus on LLM backends. + +| Variable | Description | Default Value | Required? | Notes | +|-----------------------|-------------|---------------|-----------|--------| +| **TOME_WEBHOOK_SECRET** | A secret key for verifying GitHub webhook signatures. This enhances security by ensuring incoming webhooks are authentic. | (empty) | Optional | Set this to a strong, random string if you're using webhooks. Example: `TOME_WEBHOOK_SECRET=my_secure_secret_123`. | +| **XAI_API_KEY** | Your xAI API key, required for authenticating requests to the xAI backend (e.g., Grok models). | (empty) | Required for xAI backend | Obtain this from the xAI dashboard. It's used for cost-efficient AI inference. | +| **TOME_XAI_MODEL** | Specifies the xAI model to use (e.g., "grok-3-mini-fast"). This integrates with the LLM backend for tasks like text generation. | "grok-3-mini-fast" | Optional | Choose a model based on your needs; lighter models are faster and cheaper. | + +Other variables in `.env.example` (e.g., for Anthropic or Ollama) remain unchanged but are now part of the auto-detection system. + +### Parameters and Options +- **LLM Backend Options**: The `TOME_LLM_BACKEND` variable can be set to "anthropic", "xai", or "ollama". If not explicitly set, the app auto-detects based on available API keys: + - Priority order: Anthropic (if `ANTHROPIC_API_KEY` is set) > xAI (if `XAI_API_KEY` is set) > Ollama (default, no key needed). + - Example: If both `ANTHROPIC_API_KEY` and `XAI_API_KEY` are set, Anthropic will be chosen unless `TOME_LLM_BACKEND` is manually set to "xai". +- **Model Selection**: Variables like `TOME_XAI_MODEL` allow you to specify exact models. For xAI, options include "grok-3-mini-fast" (recommended for speed) or others available in the xAI API. + +## How to Use It + +### Setting Up xAI as the LLM Backend +To use xAI for cost-efficiency: +1. Obtain an xAI API key from the xAI developer portal. +2. Edit your `.env` file (based on `.env.example`) and set the relevant variables: + - Set `XAI_API_KEY` to your actual key. + - Optionally, set `TOME_XAI_MODEL` to your preferred model (e.g., "grok-3-mini-fast"). + - To force xAI as the backend, set `TOME_LLM_BACKEND=xai`. Otherwise, let auto-detection handle it. + +**Example .env Configuration for xAI:** +``` +TOME_GITHUB_TOKEN=ghp_your_token_here +TOME_WEBHOOK_SECRET=my_secure_secret_123 # For security +TOME_LLM_BACKEND=xai # Explicitly set for xAI +XAI_API_KEY=xai-your_key_here # Replace with your actual key +TOME_XAI_MODEL=grok-3-mini-fast # Recommended for cost-efficiency +``` + +### Code Examples +Here's how these changes are used in the code: + +1. **Auto-Detection in config.py**: + ```python + import os + + class Config: + # LLM backend auto-detection + LLM_BACKEND = os.getenv("TOME_LLM_BACKEND", + "anthropic" if os.getenv("ANTHROPIC_API_KEY") + else "xai" if os.getenv("XAI_API_KEY") + else "ollama" + ) + XAI_API_KEY = os.getenv("XAI_API_KEY", "") + XAI_MODEL = os.getenv("TOME_XAI_MODEL", "grok-3-mini-fast") + ``` + - This code checks for API keys in order and sets the backend automatically. + +2. **Usage in app.py (Health Check)**: + ```python + async def health(): + if Config.LLM_BACKEND == "xai": + llm_status = "configured" if Config.XAI_API_KEY else "missing_key" + model = Config.XAI_MODEL + ``` + - The app checks if xAI is configured and reports the status. + +3. **LLM Generation in engine.py**: + ```python + async def llm_generate(prompt: str, json_mode: bool = False) -> str: + if Config.LLM_BACKEND == "xai": + return await _xai_generate(prompt, json_mode) # Internal function for xAI calls + ``` + - This function routes prompts to the xAI backend if selected. + +To integrate this into your workflow: +- Run the app with the updated `.env` file: `python app.py`. +- Test the LLM backend by sending a prompt; monitor logs for backend selection. + +## Common Patterns and Gotchas + +- **Best Practices**: + - **Prioritize Security**: Always set `TOME_WEBHOOK_SECRET` for production environments to prevent unauthorized access. + - **Cost Management**: Use xAI for high-volume tasks due to its efficiency. Monitor your API usage via the xAI dashboard to avoid unexpected costs. + - **Auto-Detection Order**: Remember the fallback order (Anthropic > xAI > Ollama). If you want xAI as the default, set `TOME_LLM_BACKEND=xai` explicitly. + - **Environment File Handling**: Copy `.env.example` to `.env` and fill in your values. Use a tool like `python-dotenv` to load these variables securely. + +- **Potential Issues and How to Avoid Them**: + - **Missing API Key**: If `XAI_API_KEY` is not set, the app will skip xAI and fall back to another backend. Check logs for errors like "missing_key" and ensure your key is valid. + - **Rate Limiting**: xAI may impose limits on requests. Handle this by implementing retry logic or switching backends dynamically. + - **Model Compatibility**: Not all xAI models support every feature (e.g., JSON mode). Test your chosen model with sample prompts before deploying. + - **Cross-Platform Setup**: If running Ollama locally, ensure it's accessible via the specified URL (e.g., `http://localhost:11434`). For xAI, verify network access to their API endpoints. + - **Debugging Tip**: Add logging in `config.py` to output the detected `LLM_BACKEND` for easier troubleshooting. + +This documentation ensures you can implement and benefit from the xAI integration effectively. If you encounter issues, refer to the code changes or the project's README for additional context. \ No newline at end of file diff --git a/docs/config.md b/docs/config.md new file mode 100644 index 0000000..1702fe5 --- /dev/null +++ b/docs/config.md @@ -0,0 +1,96 @@ +# Documentation for LLM Backend Expansion to Include xAI + +This documentation covers the recent changes to the application's configuration, which expand the supported LLM (Large Language Model) backends to include xAI (Grok). This update enhances flexibility, allowing users to choose from multiple backends based on their needs, with a focus on cost-efficiency and auto-detection. + +## What This Does and Why It Matters + +This change updates the `LLM_BACKEND` configuration in `config.py` to support "xai" as an additional option alongside "anthropic" (for Anthropic's Claude models) and "ollama" (for local inference with Ollama). The system now auto-detects the backend based on the presence of API keys in the environment variables, prioritizing "anthropic" if its key is set, then "xai", and defaulting to "ollama". + +Why it matters: +- **Increased Options**: Users can now integrate xAI's Grok models, which are designed for efficiency and may reduce costs compared to other providers. +- **Auto-Detection**: This simplifies setup by automatically selecting the backend without manual configuration, reducing errors and improving user experience. +- **Seamless Integration**: New variables (`XAI_API_KEY` and `XAI_MODEL`) enable easy configuration for xAI, ensuring the application can switch backends without major code changes. +- **Broader Use Cases**: This supports diverse environments, such as cloud-based AI services (Anthropic/xAI) or local setups (Ollama), making the application more versatile for developers and organizations. + +## How to Use It + +To use the updated LLM backend configuration, you'll primarily work with environment variables in a `.env` file (e.g., based on the provided `.env.example`). The system auto-detects the backend, but you can override it explicitly. + +### Steps to Set Up xAI as the Backend + +1. **Create or Update Your `.env` File**: + - Copy the provided `.env.example` to `.env` in your project root. + - Set the required variables for xAI. + +2. **Example Configuration**: + Here's an example of a `.env` file configured for xAI: + + ``` + # LLM Backend — auto-detects based on which key is set + # Options: "anthropic", "xai", "ollama" + TOME_LLM_BACKEND=xai # Optional override; if omitted, it auto-detects based on keys + + # xAI (Grok) Configuration + XAI_API_KEY=xai-your_api_key_here # Required for xAI; obtain from xAI dashboard + XAI_MODEL=grok-3-mini-fast # Default model; can be changed for other xAI models + ``` + + - If you don't specify `TOME_LLM_BACKEND`, the system checks for `ANTHROPIC_API_KEY` first, then `XAI_API_KEY`, and defaults to "ollama". + - Restart your application after updating the `.env` file to apply changes. + +3. **Code Example in Your Application**: + Once configured, you can use the LLM backend in your code as before. For instance, in `engine.py`, the `llm_generate` function now supports xAI: + + ```python + import config as Config + + async def main(): + # Example: Generate text using the configured LLM + prompt = "Explain the theory of relativity in simple terms." + response = await Config.llm_generate(prompt) # This will use xAI if configured + print(response) + ``` + + - The `llm_generate` function in `engine.py` automatically routes to the correct backend based on `Config.LLM_BACKEND`. + - In `app.py`, the health check now includes xAI status: + + ```python + async def health(): + if Config.LLM_BACKEND == "xai": + llm_status = "configured" if Config.XAI_API_KEY else "missing_key" + model = Config.XAI_MODEL + # ... other backends ... + ``` + + This allows you to verify the backend configuration via an API endpoint. + +## Parameters and Options + +The following configuration options have been updated or added in `config.py`. These are accessed via environment variables and have default values for ease of use. + +| Variable | Description | Default Value | Notes | +|-----------------------|--------------------------------------|--------------------------------|-------| +| **TOME_LLM_BACKEND** | Specifies the LLM backend to use. | Auto-detected: "anthropic" if `ANTHROPIC_API_KEY` is set, otherwise "xai" if `XAI_API_KEY` is set, otherwise "ollama". | Options: "anthropic", "xai", "ollama". Set explicitly to override auto-detection. | +| **XAI_API_KEY** | API key for xAI (Grok) services. | Empty string ("") | Required for xAI backend; obtain from the xAI platform. If not set, xAI won't be selected via auto-detection. | +| **XAI_MODEL** | The default model to use with xAI. | "grok-3-mini-fast" | Specifies the xAI model for inference. Change this to other available xAI models as needed. | + +These variables are loaded in `config.py` using `os.getenv`, ensuring they are environment-safe and easy to manage. + +## Common Patterns and Gotchas + +- **Auto-Detection Order**: The backend is detected in this priority: Anthropic > xAI > Ollama. If multiple keys are set, Anthropic takes precedence. Always check your `.env` file to avoid unexpected behavior. + +- **Overriding Defaults**: While defaults exist (e.g., `XAI_MODEL`), explicitly set them in `.env` for production to ensure consistency across environments. + +- **Error Handling**: If an API key is missing for the selected backend (e.g., `XAI_API_KEY` for "xai"), the application may fail gracefully (e.g., in health checks) or raise errors during runtime. Monitor logs for messages like "missing_key". + +- **Security Best Practices**: Treat API keys as sensitive; use environment variables or secure vaults instead of hardcoding. Never commit actual keys to version control—use the `.env.example` as a template. + +- **Testing Patterns**: After changing backends, test with a simple prompt (e.g., via `llm_generate`) to verify integration. For xAI, ensure your API key has the necessary permissions for the selected model. + +- **Gotchas**: + - xAI requires an internet connection, unlike Ollama which can run locally. + - If switching backends frequently, restart the application to reload configurations. + - Rate limits: xAI and Anthropic have API usage limits; handle retries or fallbacks in your code if needed. + +This update makes the application more robust and adaptable—refer to the code diffs in the change log for more details. If you encounter issues, check the environment variables and application logs first. \ No newline at end of file diff --git a/docs/engine.md b/docs/engine.md new file mode 100644 index 0000000..a7bcd95 --- /dev/null +++ b/docs/engine.md @@ -0,0 +1,81 @@ +# Documentation for New xAI Backend Integration in engine.py + +## Overview +This documentation covers a recent code change that introduces support for the xAI Grok API as a new backend for large language model (LLM) generation in the application. Specifically, a new function `_xai_generate` has been added to `engine.py`. This enhancement allows users to leverage xAI's models for more cost-efficient and flexible LLM operations, expanding beyond existing backends like Anthropic and Ollama. + +### What This Does and Why It Matters +- **What it does**: The `_xai_generate` function is a new asynchronous function that interfaces with the xAI Grok API to generate responses based on a given prompt. It integrates into the existing `llm_generate` function, which acts as a dispatcher for different LLM backends. When the application is configured to use the "xai" backend, `llm_generate` will call `_xai_generate` to handle the API request. +- **Why it matters**: xAI's models, such as Grok-3, offer a cost-effective alternative for AI-driven tasks like text generation. This addition increases the application's flexibility, allowing developers to switch backends based on performance, cost, or availability. It also demonstrates how the system can be extended for custom LLM integrations, promoting modularity and ease of maintenance. + +## How It Fits into the Existing Code +The `_xai_generate` function is called from `llm_generate`, which is the central entry point for LLM generation. Here's a simplified overview: +- `llm_generate` checks the configured backend (via `Config.LLM_BACKEND`) and routes the request accordingly. +- If `Config.LLM_BACKEND` is set to "xai", it invokes `_xai_generate(prompt, json_mode)`. +- This design allows seamless switching between backends without altering high-level code, as long as the environment is properly configured. + +Configuration for the xAI backend is handled in `config.py`, which auto-detects the backend based on environment variables (e.g., `XAI_API_KEY`). Updates in `app.py` ensure that the health check reflects the xAI status. + +## Parameters and Options +The `_xai_generate` function has the following parameters: +- **prompt (str)**: The input text or query to send to the xAI Grok API. This is the main content that the model will process to generate a response. +- **json_mode (bool, optional)**: Defaults to `False`. If set to `True`, the function requests the API to format the response as structured JSON, which is useful for applications needing parseable output. + +To use the xAI backend, you must set the following environment variables in your `.env` file (as shown in the updated `.env.example`): +- **XAI_API_KEY**: Your xAI API key (e.g., `XAI_API_KEY=xai-your_key_here`). This is required for authentication. +- **TOME_XAI_MODEL**: The specific xAI model to use (e.g., `TOME_XAI_MODEL=grok-3-mini-fast`). This defaults to "grok-3-mini-fast" if not specified. + +The backend is auto-detected in `config.py`, so setting `XAI_API_KEY` will prioritize "xai" over other options like "anthropic" or "ollama". + +## How to Use It +To integrate and use the xAI backend, follow these steps: + +1. **Set up your environment**: + Add the necessary variables to your `.env` file: + ``` + TOME_LLM_BACKEND=xai # Explicitly set if needed, but auto-detection will use this if XAI_API_KEY is present + XAI_API_KEY=your_xai_api_key_here + TOME_XAI_MODEL=grok-3-mini-fast # Optional, but recommended for specificity + ``` + +2. **Call the LLM generation function**: + In your code, use `llm_generate` as you normally would. The system will automatically route to `_xai_generate` if configured. + + **Example**: + ```python + import asyncio + from engine import llm_generate + + async def main(): + prompt = "Explain the theory of relativity in simple terms." + json_mode = True # Request JSON-formatted response + + try: + response = await llm_generate(prompt, json_mode) + print(response) # Output: A JSON string or plain text based on the model + except Exception as e: + print(f"Error: {e}") + + if __name__ == "__main__": + asyncio.run(main()) + ``` + + In this example: + - If `TOME_LLM_BACKEND` is "xai", `_xai_generate` will handle the request. + - The response will be a string from the xAI API, potentially in JSON format if `json_mode` is True. + +3. **Integrating Custom LLM Backends**: + For developers looking to add their own backends: + - Add a new condition in `llm_generate` similar to the existing ones (e.g., `if Config.LLM_BACKEND == "custom": return await _custom_generate(...)`). + - Create a new function like `_xai_generate` in `engine.py`. + - Update `config.py` to include any new environment variables and auto-detection logic. + - Ensure error handling for API keys and network issues. + +## Common Patterns and Gotchas +- **Auto-Detection**: The system prioritizes backends based on the order in `config.py` (e.g., Anthropic first, then xAI, then Ollama). If multiple keys are set, xAI will only be used if explicitly set or if Anthropic's key is absent. +- **Error Handling**: Always wrap calls to `llm_generate` in try-except blocks, as API failures (e.g., invalid keys or rate limits) can raise exceptions. For xAI specifically, ensure your API key is valid and not expired. +- **Performance Considerations**: xAI models may have different response times and token limits compared to Anthropic or Ollama. Test with your prompts to avoid truncation or incomplete responses. +- **JSON Mode**: If using `json_mode=True`, parse the response with a JSON library (e.g., `json.loads(response)`) to handle it properly, as the output might not always be perfectly formatted. +- **Security**: Never hardcode API keys in your code. Always use environment variables or secure vaults. +- **Testing**: Before deploying, verify the backend in a health check or debug mode, as shown in the updated `app.py`. If `XAI_API_KEY` is missing, the system will fall back to other backends or report an error. + +This integration enhances the application's versatility for LLM tasks—refer to the code changes in `engine.py`, `config.py`, and `.env.example` for full details. If you're extending this for other backends, follow the modular pattern to keep the codebase maintainable. \ No newline at end of file