profiler_agent

AIQ Profiler Agent

The profiler agent is a tool that allows you to analyze the performance of AIQ toolkit workflows. It uses the Phoenix server to store and retrieve traces of workflow runs.

Installation and Setup

If you have not already done so, follow the instructions in the Install Guide to create the development environment and install AIQ toolkit.

Install this Workflow:

From the root directory of the AIQ toolkit library, run the following commands:

uv pip install -e examples/advanced/profiler_agent

Set Up API Keys

If you have not already done so, follow the Obtaining API Keys instructions to obtain an NVIDIA API key. You need to set your NVIDIA API key as an environment variable to access NVIDIA AI services:

export NVIDIA_API_KEY=<YOUR_API_KEY>

Usage

1. Start the Phoenix server if not already running. If you are using a remote Phoenix server, you can skip this step and modify the config/config.yml to point to the URL.

   docker run -p 6006:6006 -p 4317:4317 -i -t arizephoenix/phoenix:latest

2. Ensure that there are traces in the Phoenix server. You can use the simple calculator example to generate traces.

   Note: This requires installing both the optional telemetry dependencies along with the simple calculator. You can do this by running the following commands:

   uv pip install -e examples/basic/functions/simple_calculator

   Then, run the simple calculator example to generate traces:

   aiq run --config_file examples/basic/functions/simple_calculator/configs/config-tracing.yml --input "Is the product of 2 * 4 greater than the current hour of the day?"
   aiq run --config_file examples/basic/functions/simple_calculator/configs/config-tracing.yml --input "Is the product of 33 * 4 greater than the current hour of the day?"
   aiq run --config_file examples/basic/functions/simple_calculator/configs/config-tracing.yml --input "Is the sum of 44 and 55 greater than the current hour of the day?"
   aiq run --config_file examples/basic/functions/simple_calculator/configs/config-tracing.yml --input "Is the difference between 7 and 5 less than the current hour of the day?"

3. Run the profiler agent:

   aiq serve --config_file=examples/advanced/profiler_agent/configs/config.yml
   

4. Launch the AIQ toolkit User Interface by using the instructions in the Launching the User Interface guide.

5. Query the agent with natural language via the UI:

   Show me the token usage of last run
   

   

   More examples:

   Show me flowchart of last 3 runs
   

   Analyze the last 2 runs
   

Features

• Query Phoenix traces with natural language
• Analyze LLM application performance metrics
• Generate trace visualizations
• Extract user queries across trace spans