TraceLens is a friendly evaluation and regression-testing framework for AI agents. It turns agent runs into inspectable traces, graded outcomes, baseline comparisons, and CI-ready reliability signals.
迹镜是一个面向 AI Agent 的评测与回归检测框架。它把每次 agent run 转化成可观察的轨迹、可评分的结果、可比较的 baseline，以及可用于 CI 的可靠性信号。
📖 Documentation: https://ssf0409.github.io/tracelens/ • 📦 PyPI: pip install tracelens
Agents are non-deterministic, so "the tests pass" says little about whether an agent change is safe to ship. TraceLens gives a Python team a regression check that lives in its own repository:
- Repo-owned, local, no backend. Tasks, adapters, graders, baselines, and a
tracelens.yamlare files you commit; runs write JSON, Markdown, and HTML artifacts next to them. Nothing needs an account or a server, and CI is a plain job running the same command you run locally. - Inspectable evidence. Every run keeps its trials, transcripts, grader feedback, and provenance (which task content, graders, and settings produced the numbers, and which candidate was under test).
tracelens inspectexplains a failure from those files. - Explicit uncertainty. pass@k and pass^k separate capability from reliability, intervals come from a task-level bootstrap, numbers that were not measured are reported as unavailable rather than zero, and a gate that cannot be evaluated says so instead of passing.
- Harness failures stay separate from agent failures. Infra errors and grader crashes are counted on their own, so a broken eval never looks like a regression.
Use it when you need to answer questions like:
- Did this agent produce the right outcome, not just run without crashing?
- Is a flaky success still a real capability after 3–5 attempts?
- Did a prompt, model, tool, or infra change regress a baseline?
- Can CI block unsafe or lower-quality agent behavior before it ships?
It supports both subjective evaluation (LLM-as-judge for quality) and objective evaluation (schema validity, tool-use constraints, latency, budget, or domain-specific metrics).
Each claim above rests on a different kind of evidence; here is which:
| Claim | What it rests on |
|---|---|
| The documented workflow works from a fresh install | A CI job installs a freshly built wheel into a clean environment and drives init, run --config, baselines, the gate, an intentional regression, inspect, compare, a targeted rerun, an infra outage, a grader crash, malformed input, and checkpoint/resume through the console script. |
| The statistics do what the contract says | Hand-derived and independent-reference tests against the statistical contract (task-level bootstrap with multiplicity, order-independent pass^k, paired run comparison). |
| Integrations behave at their boundaries | Tests exercise the JSON/JSONL/CSV loaders, the HTTP adapter, the optional Hugging Face loader, the generated GitHub workflow, and tracelens.yaml. |
| TraceLens caught a real regression in a real project | Not yet published. The examples and the scaffold use simulated agents. A sanitized downstream case study is the open half of issue #33; until it exists, treat "catches regressions" as a tested mechanism, not an observed result. |
Hosted evaluation and observability platforms also run datasets, experiments, and CI checks; the difference is where the evidence lives and what is required, not whether evaluation exists. See TraceLens vs Adjacent Tools.
# Recommended: uv
uv pip install tracelens
# Or: plain pip
pip install tracelensFor the repository examples and local development tools:
git clone https://github.com/ssf0409/tracelens.git
cd tracelens
uv pip install -e ".[dev]"See Installation for extras
([llm], [http], [datasets]) and CI setup.
python examples/hello_world.py
tracelens report --results examples/reports/hello_world_report.json --format markdownExpected first output:
tracelens hello-world
--------------------
trials run : 9
pass rate : 100%
report json: examples/reports/hello_world_report.json
sample md : examples/reports/hello_world_report.md
The checked-in sample report shows the concrete pieces a real eval needs: tasks, trials, pass@k, pass^k, graders, baseline comparison, regression result, and CI summary.
To start inside your own project:
tracelens init .
tracelens run --config tracelens.yamltracelens init writes user-owned starter files under eval/, a tracelens.yaml holding the run settings, and a GitHub Actions workflow that runs the same command on every pull request. Flags on the command line override the file. It refuses to overwrite generated files unless you pass --force.
Four pieces — Task, Adapter, Grader, Runner — and a report:
import asyncio
from tracelens import (
Task, EvalSet, SimpleAdapter, CodeGrader,
EvaluationRunner, RunnerConfig, Transcript,
)
from tracelens.reporting.generator import ReportGenerator
# 1. Define tasks
eval_set = EvalSet(name="Math Suite", tasks=[
Task(name="Add 2+3", input_data={"a": 2, "b": 3}, metadata={"expected": 5}),
Task(name="Add 10+20", input_data={"a": 10, "b": 20}, metadata={"expected": 30}),
])
# 2. Wrap your agent
async def math_agent(input_data: dict) -> dict:
return {"answer": input_data["a"] + input_data["b"]}
adapter = SimpleAdapter(math_agent)
# 3. Write a grader
class MathGrader(CodeGrader):
def compute_metrics(self, transcript: Transcript, task: Task) -> dict[str, float]:
return {"correct": float(transcript.final_output["answer"] == task.metadata["expected"])}
def determine_pass(self, metrics: dict[str, float], task: Task) -> tuple[bool, float]:
return metrics["correct"] == 1.0, metrics["correct"]
# 4. Run and report
batch = asyncio.run(EvaluationRunner(adapter, [MathGrader("math")], RunnerConfig(num_runs=3)).run(eval_set))
print(ReportGenerator().render_markdown(ReportGenerator().build_report(batch)))Walkthrough: Getting Started (5 min). Ready for a non-toy agent? Evaluating a Real Agent.
The full, searchable docs live at https://ssf0409.github.io/tracelens/. Highlights:
Also: Build Your First Eval · User Guide · Loading Task Data · Evaluation Recipes · API Reference · Examples · Roadmap · Contributor Testing · Releasing.
TraceLens is MIT licensed and open to contributions. Start with CONTRIBUTING.md, then run the local verification gate:
make verify # lock check -> lint -> typecheck -> tests + coverageSecurity issues should be reported privately using SECURITY.md.
- Grade outcomes, not execution paths — focus on what the agent produced.
- Handle non-determinism — pass@k for capability, pass^k for reliability.
- Start with 20–50 real failure cases — build suites from actual issues.
- Read transcripts regularly — catch false signals and grader bugs.
- Calibrate with human evaluation — LLM graders drift without it.
- Separate harness failures from agent failures — track infra/grader error rates alongside pass rates.
Informed by Anthropic's Demystifying Evals for AI Agents.