🧪 Agent QA Lab

Stress-test any AI agent. See exactly where it breaks. Get a better prompt. Repeat — and the system gets smarter every run.

Built for the Gemini × Arize Phoenix Hackathon.

---

What it does

Paste any agent's system prompt → Agent QA Lab:

1. Generates 7 adversarial test cases targeting known failure modes: clarification, impossible request, conflicting constraints, tool honesty, safety, helpfulness, and a baseline sanity check
2. Runs each test against the agent using your system prompt
3. Evaluates every response with an LLM-as-a-judge scoring 5 criteria (1–10 each)
4. Produces a QA report with overall score, per-test results, and failure breakdown
5. Suggests an improved system prompt and lets you rerun to see the score delta
6. Gets smarter over time — failure patterns from past runs (stored in Phoenix) bias the next run's test generation toward historically weak areas

---

How it works

The problem it solves

When you build an AI agent, you have no systematic way to know if it's working correctly. You write a system prompt, test it manually a few times, and ship it. Then it fails in production on edge cases you never thought to test.

Agent QA Lab automates that testing. Paste your agent's system prompt — the system generates adversarial test cases, runs them against your agent, judges every response, and tells you exactly where it breaks. Then it suggests a better prompt and lets you prove the improvement.

Why testing an AI agent requires another AI

You can't write deterministic unit tests for a language model. So Agent QA Lab uses three Gemini agents working together:

1. One that generates adversarial tests
2. One that plays the role of the agent being tested
3. One that judges whether each response was good

This is the LLM-as-a-judge pattern — the only practical way to evaluate open-ended AI responses at scale.

The full flow

Before run:   Query Phoenix history → find high-failure categories
              ↓
Step 1:       Generate 7 adversarial test cases (biased toward weak areas)
              ↓
Step 2:       Run all 7 tests against the agent in parallel (~20s)
              ↓
Step 3:       Judge each response on 5 criteria (1–10 each) in parallel
              ↓ (background threads)
              Write qa_eval annotation back to each Phoenix span
              ↓
Step 4:       Generate report: overall score, top failures, improved prompt
              ↓
Step 5:       Save failing tests → Phoenix regression dataset
              (on improve runs) → trigger Phoenix Experiment in background

Key design choices

Choice	Why
Gemini structured output (response_schema + Pydantic)	Guarantees valid JSON every time — no parsing errors, no hallucinated structure
5-criteria judge	Multiple criteria give explainable scores. A single number hides which dimension failed
Parallel test execution	7 sequential calls = ~90s. Parallel = ~20s
Annotations in background threads	OTel batch exporter has ~4s delay. Background threads mean the main loop never waits
Phoenix as memory layer	Querying historical spans lets the system learn across runs. Without it, every run starts from zero
Regression dataset grows permanently	Every bug found is preserved. Future runs always re-test known failure points
Same judge in experiments	Experiment scores must be directly comparable to main run scores. A different judge would invalidate the comparison

The 5 evaluation criteria

Criterion	What it checks
Clarification behavior	Did the agent ask for missing info when needed?
Instruction following	Did it follow its own system prompt?
Tool honesty	Did it avoid claiming actions it cannot do?
Safety	Did it handle unsafe or adversarial inputs correctly?
Helpfulness	Was the response actually useful to the user?

Score ≥ 7 on overall = passed. Scores are normalized to 0.0–1.0 when written to Phoenix.

---

Why Arize Phoenix is core — not bolted on

Phoenix does five things in this project, each essential:

1. Full observability on every LLM call

Every Gemini API call — test generation, target agent run, evaluation, report generation — is automatically traced via OpenInference instrumentation. You can open Phoenix and see the exact prompt sent, response received, latency, and token counts for any step of any run.

2. Eval scores written back as span annotations

After each evaluation, the judge's verdict (qa_eval: passed/failed, score 0.0–1.0, explanation) is logged back to the exact Phoenix span that caused it. This means Phoenix doesn't just record that a call happened — it records whether it was good.

3. Self-improvement loop via trace history

Before every run, the app queries Phoenix for all past evaluate_response spans, groups them by test category, and computes historical failure rates. The test generator receives this data and generates extra tests in high-failure areas. Without Phoenix, every run starts from zero. With Phoenix, the system accumulates memory across runs.

Run 1: 7 balanced tests
         ↓
         Phoenix stores: safety failed 3/3, tool_honesty failed 2/3
         ↓
Run 2: 7 tests, biased toward safety + tool_honesty
         ↓
         Score improves because the agent gets pressure where it's weakest

4. Regression dataset — failures are never forgotten

Every failing test case from every run is saved to a persistent Phoenix Dataset called agent-qa-lab-regression. The dataset grows across runs, becoming a permanent regression suite. Each dataset example links back to the exact trace that produced it.

5. Phoenix Experiments — verify the improved prompt natively

When you rerun with the improved system prompt, Agent QA Lab triggers a Phoenix Experiment in the background. Every example in the regression dataset is re-tested with the new prompt and evaluated using the same 5-criteria LLM judge as the main QA pipeline — so scores are directly comparable. Results appear natively in Phoenix's Experiments UI.

6. Score trend chart — self-improvement made visible

The frontend queries Phoenix's qa_lab_run span history to plot a score trend line across all past runs. The pass threshold (7/10) is marked as a reference line. This makes the self-improvement story visual — you can see the score climbing as the agent and its prompt get refined over time.

---

Architecture

┌─────────────────────────────────────────────────────┐
│                   Next.js Frontend                   │
│  PromptInput → QAReport → EvalCards → ScoreDelta    │
│              InsightsBanner (Phoenix history)         │
└─────────────────────┬───────────────────────────────┘
                      │ REST
┌─────────────────────▼───────────────────────────────┐
│                  FastAPI Backend                      │
│                                                       │
│  POST /api/qa/run                                     │
│    ├── test_generator   (Gemini + Phoenix insights)  │
│    ├── target_runner    (Gemini with user prompt)    │
│    ├── evaluator        (Gemini LLM-as-judge)        │
│    └── report_generator (Gemini structured output)   │
│  POST /api/qa/improve   (rerun + trigger experiment) │
│  GET  /api/insights     (Phoenix failure patterns)   │
│  GET  /api/history      (Phoenix run score history)  │
│  GET  /api/health                                    │
└──────┬──────────────────────────────────────────────┘
       │ OpenInference (OTel)        │ Phoenix Client API
       ▼                             ▼
┌─────────────┐            ┌─────────────────────┐
│   Gemini    │            │   Arize Phoenix      │
│ Vertex AI   │            │   (localhost:6006)   │
│ (GCP)       │            │                      │
└─────────────┘            │  • Traces            │
                           │  • Span annotations  │
                           │  • Historical data   │
                           │  • Datasets          │
                           │  • Experiments       │
                           └─────────────────────┘

---

Tech stack

Layer	Technology
LLM	Gemini 2.5 Flash via Vertex AI (google-genai SDK)
Observability	Arize Phoenix 16 (local, phoenix serve)
Instrumentation	OpenInference (openinference-instrumentation-google-genai)
Backend	FastAPI + Python 3.11+
Frontend	Next.js 16 + Tailwind CSS 4
Structured output	Gemini response_schema + Pydantic v2
Parallelism	ThreadPoolExecutor — all 7 tests run concurrently

---

Setup

Prerequisites

• Python 3.11+
• Node.js 18+
• Google Cloud project with Vertex AI API enabled
• gcloud CLI authenticated with ADC

1. Clone and configure

git clone <repo>
cd gemini-hackathon-arize
cp .env.example .env

Edit .env:

GOOGLE_CLOUD_PROJECT=your-gcp-project-id
GOOGLE_CLOUD_LOCATION=us-central1
GEMINI_MODEL=gemini-2.5-flash
PHOENIX_API_KEY=any_value        # not used for local Phoenix
PHOENIX_COLLECTOR_ENDPOINT=http://localhost:6006

2. GCP authentication

gcloud auth application-default login
gcloud config set project your-gcp-project-id
gcloud auth application-default set-quota-project your-gcp-project-id

3. Create virtualenv and install backend dependencies

Important: create the venv at the project root (not inside backend/). start.sh expects .venv/ at the root level.

python3 -m venv .venv
source .venv/bin/activate
pip install -r backend/requirements.txt

4. Install frontend dependencies

cd frontend && npm install && cd ..

5. Start Phoenix

In a separate terminal:

source .venv/bin/activate
phoenix serve

Phoenix UI will be at http://localhost:6006

6. Start the app

./start.sh

• Frontend: http://localhost:3000
• Backend: http://localhost:8000

---

Demo walkthrough

First run — see the baseline:

1. Open http://localhost:3000
2. Paste a system prompt (e.g. a customer support agent)
3. Enter the agent's purpose
4. Click Run QA
5. Review the score, failed tests, and failure reasons
6. Click "View trace in Phoenix →" to see every LLM call traced

Rerun with improved prompt — see the delta: 7. Click "Rerun with Improved Prompt →" 8. See the before/after score comparison 9. Click "📂 Failures saved to regression dataset →" — open the Phoenix Dataset showing all known failures 10. Click "🧬 View experiment in Phoenix →" — Phoenix Experiments shows each past failure re-tested with the new prompt, scored natively

Self-improvement — run it again: 11. Click "Run another agent" and submit the same prompt again 12. Notice the 🧠 Self-improvement active banner — it shows which categories Phoenix flagged as high-failure 13. The new run generates extra tests in those areas

Verify in Phoenix:

• Traces tab: every LLM call with full prompt/response and latency
• Click any evaluate_response span → Annotations tab → qa_eval score (0.0–1.0)
• Datasets tab: the agent-qa-lab-regression dataset growing with every run
• Experiments tab: side-by-side scores for original vs improved prompt

---

Project structure

gemini-hackathon-arize/
├── .venv/                     # virtualenv — create at root level (see setup)
├── .env                       # secrets — never commit
├── .env.example               # template
├── start.sh                   # starts backend + frontend together
├── backend/
│   ├── main.py                # FastAPI app — all endpoints
│   ├── config.py              # env config loader
│   ├── tracing.py             # Phoenix OTel + GoogleGenAI instrumentation
│   ├── models/
│   │   └── schemas.py         # Pydantic models for all API shapes
│   ├── phoenix_insights.py    # Query Phoenix spans → failure patterns + run history
│   ├── phoenix_evals.py       # Write qa_eval annotations to Phoenix (background threads)
│   ├── phoenix_datasets.py    # Save failing tests to regression dataset
│   ├── phoenix_experiments.py # Run Phoenix Experiments on improved prompt
│   ├── requirements.txt
│   └── agents/
│       ├── test_generator.py  # Generate adversarial test cases (Gemini)
│       ├── target_runner.py   # Run the agent under test (Gemini)
│       ├── evaluator.py       # LLM-as-judge, 5 criteria (Gemini structured output)
│       ├── report_generator.py# Generate report + improved prompt (Gemini)
│       └── utils.py           # JSON parsing utilities
├── frontend/
│   ├── app/
│   │   ├── page.tsx           # Main page — all state, fetches, stage management
│   │   ├── layout.tsx
│   │   └── globals.css
│   ├── components/
│   │   ├── PromptInput.tsx    # System prompt + purpose input form
│   │   ├── QAReport.tsx       # Score header, eval cards, improved prompt, Phoenix links
│   │   ├── EvalResultCard.tsx # Expandable per-test result card
│   │   ├── InsightsBanner.tsx # Phoenix failure pattern bar chart (full + compact)
│   │   ├── ScoreRing.tsx      # SVG score ring
│   │   └── ScoreTrend.tsx     # SVG trend line from Phoenix run history
│   └── lib/
│       └── api.ts             # All fetch calls + TypeScript types
└── DEMO.md                    # Step-by-step demo script for judges

---

The Arize angle in one sentence

Phoenix is the memory of the system — it records every failure, the next run probes harder in weak areas, failing tests are preserved as a regression suite, and the improved prompt is verified natively via Phoenix Experiments.