From a4c90321120eb0d21b00e617e42ad2d027634ee4 Mon Sep 17 00:00:00 2001 From: siddharthsambharia-portkey Date: Fri, 31 Jan 2025 17:09:08 +0530 Subject: [PATCH 1/2] portkey cookbook --- third_party/Portkey/Readme.md | 84 ++++++ third_party/Portkey/cookbook.ipynb | 406 +++++++++++++++++++++++++++++ 2 files changed, 490 insertions(+) create mode 100644 third_party/Portkey/Readme.md create mode 100644 third_party/Portkey/cookbook.ipynb diff --git a/third_party/Portkey/Readme.md b/third_party/Portkey/Readme.md new file mode 100644 index 00000000..fd53a90e --- /dev/null +++ b/third_party/Portkey/Readme.md @@ -0,0 +1,84 @@ +# Portkey + Mistral Integration + +This cookbook demonstrates how to effectively use Mistral models through Portkey's AI Gateway. Learn how to integrate Mistral AI with enterprise-grade reliability, observability, and performance features. + +portkey + +[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/16Z1AlAsW_d_mB5UK1EbailGt4laUU7E9?usp=sharing) + + + +## Features + +- 🚀 Unified API Gateway for Mistral models +- 📊 Full-stack observability with 40+ metrics +- 💾 Semantic & simple caching strategies +- 🔄 Automated retries & model fallbacks +- 🛡️ AI Guardrails & content safety controls + +## Getting Started + +1. Install the required packages: +```bash +pip install portkey-ai +``` + +2. Set up your API keys in environment variables: +```python +export PORTKEY_API_KEY="your_portkey_key" +export MISTRAL_VIRTUAL_KEY="your_mistral_key" # Get from Portkey dashboard +``` + +## Examples + +The cookook in the same folder contains several examples demonstrating different features: + +- Simple chat completion setup +- Load balancing and Fallback deployments +- Metadata tracking and request tracing +- Semantic caching implementation + + +## Usage + +Basic chat completion with Mistral: +```python +from portkey_ai import Portkey + +portkey = Portkey( + api_key="your_portkey_key", + virtual_key="your_mistral_key" +) + +completion = portkey.chat.completions.create( + messages=[{"role": "user", "content": "Hello!"}], + model="mistral-large-latest" +) +print(completion.choices[0].message.content) +``` + +## Features Overview + +### Production-Grade Controls +- **Load Balancing**: Distribute traffic between Mistral models/versions +- **Semantic Retries**: Auto-rewrite failed prompts for better results +- **Guardrails**: Prevent PII leaks, block prompt injections +- **Virtual Keys**: Secure API key management with budget controls + +### Enterprise Observability +- Real-time cost tracking +- Custom metadata tagging +- Request tracing with Trace IDs +- Audit logs with full history + +### Team Collaboration +- Version-controlled prompt library +- Shared routing configurations +- RBAC permissions +- Central monitoring dashboard + +## Support + +- [Portkey AI Mistral Documentation](https://docs.portkey.ai/integrations/llms/mistral-ai) +- [Portkey Wesbite](https://portkey.ai/) +- [Join Portkey Discord](https://discord.gg/SqX9epQKNR) diff --git a/third_party/Portkey/cookbook.ipynb b/third_party/Portkey/cookbook.ipynb new file mode 100644 index 00000000..7dadb966 --- /dev/null +++ b/third_party/Portkey/cookbook.ipynb @@ -0,0 +1,406 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "name": "python3", + "display_name": "Python 3" + }, + "language_info": { + "name": "python" + } + }, + "cells": [ + { + "cell_type": "markdown", + "source": [ + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/16Z1AlAsW_d_mB5UK1EbailGt4laUU7E9?usp=sharing)" + ], + "metadata": { + "id": "KR-QyseIBUuD" + } + }, + { + "cell_type": "markdown", + "source": [ + "# Portkey + Mistral AI Integration\n", + "\n", + "[Portkey](https://portkey.ai) + Mistral AI Integration\n", + "\n", + "Portkey provides a robust gateway to integrate Mistral AI models into your applications with features like:\n", + "\n", + "- Fast AI gateway access\n", + "- Full Stack observability with cost and 40+ metrics\n", + "- Semantic caching\n", + "- Automated retries & fallbacks\n", + "- Track Custom metadata & logging dashboard\n", + "\n", + "This cookbook demonstrates how to effectively use Mistral models through Portkey's AI Gateway. [Link to Portkey Docs](https://portkey.ai/docs/integrations/llms/mistral-ai)" + ], + "metadata": { + "id": "LzyRJjWS8x1a" + } + }, + { + "cell_type": "markdown", + "source": [ + "# Setup & Installation\n", + "\n", + "Install Portkey Python SDK and intialise the LLM client.\n", + "\n", + "Virtual Keys are Portkey's secure way to manage your Mistral API keys. They provide essential controls like:\n", + "- Budget limits for API usage\n", + "- Rate limiting capabilities\n", + "- Secure API key storage\n", + "\n", + "Craeate a [Virtual Key](https://app.portkey.ai/virtual-keys) in your Portkey dashboard and save it for future use.\n", + "\n", + "For detailed information on budget limits, [refer to this documentation](/product/ai-gateway/virtual-keys/budget-limits)" + ], + "metadata": { + "id": "IsWxvvu588t3" + } + }, + { + "cell_type": "code", + "source": [ + "!pip install -qU portkey-ai" + ], + "metadata": { + "id": "_RD6HM0z8xdM" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "3v7FsUPd0fqv" + }, + "outputs": [], + "source": [ + "from portkey_ai import Portkey\n", + "\n", + "# Initialize Portkey with your API keys\n", + "portkey = Portkey(\n", + " api_key=\"PORTKEY_API_KEY\", # Your Portkey API key\n", + " virtual_key=\"MISTRAL_VIRTUAL_KEY\" # Your Mistral Virtual Key from Portkey\n", + ")" + ] + }, + { + "cell_type": "markdown", + "source": [ + "# 1. Basic Chat Completion\n", + "Let's start with a simple chat completion using Mistral model:" + ], + "metadata": { + "id": "3mT9sHx59H_X" + } + }, + { + "cell_type": "code", + "source": [ + "from portkey_ai import Portkey\n", + "\n", + "# Initialize Portkey with your API keys\n", + "portkey = Portkey(\n", + " api_key=\"PORTKEY_API_KEY\", # Your Portkey API key\n", + " virtual_key=\"MISTRAL_VIRTUAL_KEY\" # Your Mistral Virtual Key from Portkey\n", + ")\n", + "\n", + "completion = portkey.chat.completions.create(\n", + " messages=[{\n", + " \"role\": \"user\",\n", + " \"content\": \"What's the purpose of Generative AI?\"\n", + " }],\n", + " model=\"mistral-large-latest\"\n", + ")\n", + "\n", + "print(completion.choices[0].message.content)" + ], + "metadata": { + "id": "zmuhbLeN9KV4" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "# Caching\n", + "Portkey offers built in caching for your Mistral AI requests. Portey has 2 types of caching built-in\n", + "\n", + "\n", + "1. Simple\n", + "2. Semantic\n", + "\n", + "Here's how you can implement cacing in your Mistral AI requests\n", + "\n" + ], + "metadata": { + "id": "I4NvLrGeC0hT" + } + }, + { + "cell_type": "code", + "source": [ + "portkey_caching_config = {\n", + " \"cache\": { \"mode\": \"simple\" }\n", + "}\n", + "\n", + "portkey_client = Portkey(\n", + " api_key=\"PORTKEY_API_KEY\", # Enter Yout Portkey API Key\n", + " config=portkey_caching_config\n", + " virtual_key=\"MISTRAL_VIRTUAL_KEY\" # Your Mistral Virtual Key from Portkey\n", + ")\n", + "\n", + "# Test load-balanced request\n", + "response = portkey_client.chat.completions.create(\n", + " messages=[{\"role\": \"user\", \"content\": \"Hello!\"}]\n", + " model=\"mistral-large-latest\"\n", + ")\n", + "print(response.choices[0].message.content)" + ], + "metadata": { + "id": "EQBgReT_Cz5M" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "# Advanced Routing Features\n" + ], + "metadata": { + "id": "zJPJO0-T9KmE" + } + }, + { + "cell_type": "markdown", + "source": [ + "## 2.1 Load Balancing\n", + "\n", + "Portkey allows you to distribute traffic across multiple models or API keys:" + ], + "metadata": { + "id": "bu99LzUB9Uhd" + } + }, + { + "cell_type": "code", + "source": [ + "load_balance_config = {\n", + " \"strategy\": {\n", + " \"mode\": \"loadbalance\"\n", + " },\n", + " \"targets\": [\n", + " {\n", + " \"virtual_key\": \"mistral-prod\", # Your Mistral Virtual Key API key\n", + " \"override_params\": {\n", + " \"model\": \"mistral-large-latest\"\n", + " },\n", + " \"weight\": 0.7\n", + " },\n", + " {\n", + " \"virtual_key\": \"misteal-backup\", # Your Mistral Virtual Key key\n", + " \"override_params\": {\n", + " \"model\": \"mistral-3b-latest\"\n", + " },\n", + " \"weight\": 0.3\n", + " }\n", + " ]\n", + "}\n", + "\n", + "portkey_lb = Portkey(\n", + " api_key=\"PORTKEY_API_KEY\", # Enter Yout Portkey API Key\n", + " config=load_balance_config\n", + ")\n", + "\n", + "# Test load-balanced request\n", + "response = portkey_lb.chat.completions.create(\n", + " messages=[{\"role\": \"user\", \"content\": \"Hello!\"}]\n", + ")\n", + "print(response.choices[0].message.content)" + ], + "metadata": { + "id": "hw1olemh9LA4" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "# 2.2 Fallbacks & Retries\n", + "Configure automatic retries and fallbacks for improved reliability:\n" + ], + "metadata": { + "id": "-bWprnGt9LSE" + } + }, + { + "cell_type": "code", + "source": [ + "fallback_config = {\n", + " \"strategy\": {\n", + " \"mode\": \"fallback\"\n", + " },\n", + " \"targets\": [\n", + " {\n", + " \"virtual_key\": \"Mistral-prod\",\n", + " \"override_params\": {\"model\": \"mistral-large-latest\"}\n", + " },\n", + " {\n", + " \"virtual_key\": \"Mistral-backup\",\n", + " \"override_params\": {\"model\": \"ministral-3b-latest\"}\n", + " }\n", + " ]\n", + "}\n", + "\n", + "portkey_fb = Portkey(\n", + " api_key=\"PORTKEY_API_KEY\", # Enter Yout Portkey API Key\n", + " config=fallback_config\n", + ")" + ], + "metadata": { + "id": "qLdKmHoU9LqU" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "Portkey has powerful routing strategies beyond what we've covered. You can use conditional routing to direct requests based on custom rules, implement advanced retry mechanisms, conduct A/B testing between models, and much more. Check out [Portkey docs](docs.portkey.ai) to explore these advanced routing capabilities in detail." + ], + "metadata": { + "id": "aeskmD31AO3A" + } + }, + { + "cell_type": "markdown", + "source": [ + "# Enhanced Observability\n", + "## Track User Metadata\n", + "Add metadata and trace IDs to track requests:\n", + "\n", + "\n" + ], + "metadata": { + "id": "80NYnIxE9mdH" + } + }, + { + "cell_type": "code", + "source": [ + "TRACE_ID = 'mistral-test'\n", + "METADATA = {\n", + " \"_environment\": \"production\",\n", + " \"_user\": \"user_123\",\n", + " \"_prompt_type\": \"general_query\"\n", + "}\n", + "\n", + "completion = portkey.chat.completions.create(\n", + " messages=[{\n", + " \"role\": \"user\",\n", + " \"content\": \"What are the main features of Portkey?\"\n", + " }],\n", + " model=\"mistral-large-latest\",\n", + " metadata=METADATA,\n", + " trace_id=TRACE_ID\n", + ")\n", + "\n", + "print(completion.choices[0].message.content)" + ], + "metadata": { + "id": "c8HHmDto9mLs" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "![dasbhboard](https://github.com/siddharthsambharia-portkey/Portkey-Product-Images/blob/main/Portkey-Dashboard.png?raw=true)\n", + "\n", + "![logs](https://github.com/siddharthsambharia-portkey/Portkey-Product-Images/blob/main/Portkey-Logs.png?raw=true)\n" + ], + "metadata": { + "id": "lf4BRZhx_oJq" + } + }, + { + "cell_type": "markdown", + "source": [ + "# 6.3 Prompt Library\n", + "\n", + "The Portkey Prompt Library is a powerful tool for managing your prompts efficiently:\n", + "\n", + "- Create, edit, and organize prompts in a central location\n", + "- Version control your prompts without needing to redeploy your application\n", + "- Collaborate with team members on prompt development\n", + "- Test prompts directly in the interface\n", + "- Organize prompts into collections for better management\n", + "\n", + "\n", + "\n", + "\n", + "![prompt library](https://github.com/siddharthsambharia-portkey/Portkey-Product-Images/blob/main/Portkey-Prompt-Library.png?raw=tru)" + ], + "metadata": { + "id": "yqSQnGI-_L3J" + } + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "## Portkey is the Production Stack for your Building your Gen AI Application with Mistral AI\n", + "\n", + "**⚙️ Advanced Controls**\n", + "- **Guardrails**: Prevent PII leaks, enforce output formats, block prompt injections\n", + "- **Conditional Routing**: Route requests based on content, cost, or performance\n", + "- **Multimodal Support**: Combine Mistral with image/audio models in single pipelines\n", + "\n", + "**🤝 Team Collaboration**\n", + "- **Prompt Library**: Version-controlled prompts with team collaboration\n", + "- **Central Monitoring**: Unified dashboard for all Mistral endpoints\n", + "- **RBAC Controls**: Granular permissions for models/features\n", + "- **Shared Configs**: Reusable routing templates across projects\n", + "\n", + "**🏢 Enterprise Foundation**\n", + "- SOC 2/GDPR/HIPAA compliance\n", + "- Private cloud deployments\n", + "- Custom retention policies\n", + "- SSO/SAML integration\n", + "- Dedicated support SLAs\n", + "\n", + "[Explore Portkey's Full Capabilities →](https://portkey.ai)" + ], + "metadata": { + "id": "equLhYXeEYfs" + } + }, + { + "cell_type": "markdown", + "source": [ + "# Next Steps\n", + "\n", + "- Visit [Portkey Documentation](https://docs.portkey.ai) for detailed API reference\n", + "- Join the [Portkey Discord](https://discord.gg/SqX9epQKNR) for community support\n", + "\n", + "\n", + "This cookbook demonstrates the key features of using Mistral models through Portkey's AI Gateway" + ], + "metadata": { + "id": "1fhzvWVP9xOA" + } + } + ] +} From b9327fff64e1b640f9b1531b68450ff42a1b97cc Mon Sep 17 00:00:00 2001 From: siddharthsambharia-portkey Date: Fri, 31 Jan 2025 17:17:36 +0530 Subject: [PATCH 2/2] portkey cookbook v2 --- README.md | 1 + third_party/Portkey/Readme.md | 2 +- .../Portkey/{cookbook.ipynb => portkey_with_mistral.ipynb} | 0 3 files changed, 2 insertions(+), 1 deletion(-) rename third_party/Portkey/{cookbook.ipynb => portkey_with_mistral.ipynb} (100%) diff --git a/README.md b/README.md index dbe5864f..89b2bf71 100644 --- a/README.md +++ b/README.md @@ -84,6 +84,7 @@ Disclaimer: Examples contributed by the community and partners do not represent | [Panel Integration - Chat with PDF](third_party/panel/README.md) | UI chat, demo, RAG | Panel | | [phospho integration](third_party/phospho/cookbook_phospho_mistral_integration.ipynb) | Evaluation, Analytics | phospho | | [pinecone_rag.ipynb](third_party/Pinecone/pinecone_rag.ipynb) | RAG | Pinecone | +| [portkey_with_mistral.ipynb](third_party/Portkey/portkey_with_mistral.ipynb) | LLM Observability, Prompt Management & Governance | Portkey | | [RAG.ipynb](third_party/LlamaIndex/RAG.ipynb) | RAG | LLamaIndex | | [RouterQueryEngine.ipynb](third_party/LlamaIndex/RouterQueryEngine.ipynb) | agent | LLamaIndex | | [self_rag_mistral.ipynb](third_party/langchain/self_rag_mistral.ipynb) | RAG | Langchain | diff --git a/third_party/Portkey/Readme.md b/third_party/Portkey/Readme.md index fd53a90e..2fea3f7a 100644 --- a/third_party/Portkey/Readme.md +++ b/third_party/Portkey/Readme.md @@ -79,6 +79,6 @@ print(completion.choices[0].message.content) ## Support -- [Portkey AI Mistral Documentation](https://docs.portkey.ai/integrations/llms/mistral-ai) +- [Portkey AI Mistral Documentation](https://portkey.sh/mistral) - [Portkey Wesbite](https://portkey.ai/) - [Join Portkey Discord](https://discord.gg/SqX9epQKNR) diff --git a/third_party/Portkey/cookbook.ipynb b/third_party/Portkey/portkey_with_mistral.ipynb similarity index 100% rename from third_party/Portkey/cookbook.ipynb rename to third_party/Portkey/portkey_with_mistral.ipynb