RCAgentBench

A benchmark for measuring how tool design and agent instructions affect LLM agents doing root cause analysis on distributed traces.

The question

When a request fails in a distributed system, an engineer opens the trace and works backwards: which span was slow, which returned an error, and which of those is the cause rather than the consequence. Agents can do the same thing if you give them tools for it.

What nobody has measured carefully is how much the design of those tools changes the outcome. Does an agent do better with six small primitives it composes itself, or with one call that returns assembled evidence? Does a strict procedural instruction reduce malformed tool calls compared to an open-ended one? And how does the tool surface Jaeger actually ships compare to either?

RCAgentBench answers those questions with numbers instead of intuition.

How it works

Each scenario is a YAML file describing a call tree with normal timings and one seeded fault. The builder assembles real spans from it, records the pre-fault timings as service baselines, injects the fault, and recomputes durations.

scenario.yaml ──> span tree ──> tools ──> agent ──> submit_diagnosis ──> classify
                      │                                                     │
                 baselines captured                            position in the call tree
                 before the fault                               relative to the real fault

Because the baselines are captured before the fault is injected, every scenario stays trace-solvable: an agent can tell that 2,040ms is abnormal for a service whose baseline is 40ms without needing application logs. That constraint is the point. A scenario needing information outside the trace would penalise every agent equally and measure nothing.

Scenarios

Scenario	Difficulty	Fault	Why it is interesting
Database latency cascade	easy	2s added to a database query	Four spans are slow, one is slow on its own work
Product catalog failure	easy	500s from the catalog service	Latency is normal, only status distinguishes the cause
Shared dependency bottleneck	medium	800ms added to a shared service	Two sibling branches regress; the common dependency is the answer
Error propagation	hard	503 from the inventory service	Four services report errors and only one raised it
Timeout mistaken for failure	hard	3s added to a payment gateway	The only span owning an error code is the one that did nothing wrong
Distracting anomaly	hard	500ms added to inventory	A logger deviates further from baseline while contributing 26ms

The last two exist to punish a specific shortcut. Ranking spans by how far they deviate from normal and taking the worst is a reasonable first instinct, and it gets both of them wrong. In the timeout scenario the gateway is slow but returns cleanly, so the API above it times out and returns 504; the only span carrying an error code is the caller. In the distracting anomaly the logger is 8x its baseline and the inventory service is 4x, but the logger adds 26ms to a 757ms request and inventory adds 500ms.

Getting these right needs causal reasoning rather than ranking, which is what separates a good agent from an adequate one.

Failure modes, not just accuracy

Accuracy collapses every wrong answer into one bucket and throws away the part that tells you what to fix. An agent that blamed the frontend for an inventory failure followed the call chain to the wrong end. An agent that named a service absent from the trace invented it. Both score zero; they are not the same problem and they call for different fixes.

Every diagnosis is placed in exactly one category, decided by where the submitted service sits in the call tree relative to the real fault:

Outcome	Meaning
correct	Identified the originating service and fault type
wrong_fault_type	Right service, misread slow for failing
upstream_symptom	Blamed a caller that merely observed the failure
downstream_dependency	Blamed a callee beneath the fault, which was healthy
unrelated_service	Blamed a service on an unaffected branch
hallucinated_service	Named a service absent from the trace
abstained	Investigated but never committed to an answer

None of these depend on reading the agent's prose. Position in the tree is a fact about the trace, so the classification is reproducible.

The categories already earn their keep on the baseline. The rule-based floor fails the two hard scenarios in different ways: unrelated_service on the distracting anomaly and upstream_symptom on the timeout. A single accuracy number would have shown two failures and told you nothing about either.

Evidence quality

submit_diagnosis accepts span ids alongside the verdict, which makes a second axis measurable: can the agent point at what convinced it?

An answer is unsupported when it is correct but cites no span belonging to the faulty service. A configuration with high accuracy and low support is guessing well, and that stays invisible if you only count right answers. Cited ids are checked against the trace, so citing span-999 counts as invalid rather than as evidence.

Trajectory metrics

Metric	What it captures
Tool error rate	Share of calls that were malformed or unresolvable
Steps to evidence	Calls made before the faulty service's span was first seen
Tokens	Input plus output, summed across turns
Turn limit hits	Runs that ran out of turns without concluding
Accuracy spread	Variation across repeated passes

The last needs --runs. A configuration averaging 75% by alternating between 50% and 100% is not the same as one scoring 75% every time, and a single pass cannot tell them apart.

Results

The rule-based baseline, three passes over all six scenarios:

Configuration	Runs	Accuracy	Symptom confusion	Tool errors	Calls	Spread
heuristic	18	66.7%	16.7%	0%	2.0	0.0

Reproduce with python cli.py benchmark --heuristic --runs 3.

The model comparison has not been run yet. The harness supports it and the configurations are wired up, but I have not spent the API budget, so there are no numbers here to report. I would rather ship an empty column than a fabricated one.

The baseline is at 66.7% by design. It clears the four straightforward scenarios and fails both hard ones, which is the headroom a benchmark needs: a floor at 100% cannot rank anything, and a floor at 0% means the scenarios are unfair rather than difficult. Tests assert both halves of that.

Running it

pip install -e .            # harness, scenarios, and the rule-based baseline
pip install -e ".[model]"   # adds the API client for model-backed runs

python cli.py scenarios                      # print the assembled traces
python cli.py benchmark --heuristic --runs 3 # rule-based floor, no API key
python cli.py benchmark --runs 3             # full matrix, needs ANTHROPIC_API_KEY
python cli.py report                         # latest results as markdown
pytest                                       # 185 tests

benchmark runs the rule-based floor alongside the model by default, because a model score means little without knowing what simple rules already achieve. Pass --no-baseline to skip it.

The full matrix at --runs 3 is 3 tool sets x 2 instruction styles x 6 scenarios x 3 passes, so 108 model-backed evaluations plus 18 baseline runs. Narrow it with --tool-sets, --styles, and --limit.

Spending real money on it

A model-backed run costs money, so the harness is built not to waste it.

Every completed evaluation is flushed to a JSONL checkpoint as it finishes, and re-running the same command resumes from it. A network error on the 81st of 108 evaluations costs you the 81st, not the preceding 80. Configuration failures (bad key, malformed request) stop the run immediately rather than spending the remaining budget collecting the same error, while rate limits and server errors are retried by the client and, if they still fail, recorded as abstentions so the run continues.

Cost accrues in the output as runs complete, priced at published list rates. Start small:

python cli.py benchmark --limit 2 --tool-sets analytical --styles strict

That is a handful of evaluations, enough to see real trajectories and measure per-run token usage before committing to the full matrix.

--model selects what is under test and --effort sets thinking depth; both move cost substantially. Thinking is deliberately left at the model default rather than disabled, because with thinking off some models write a tool call into their visible text instead of emitting a tool call. The turn succeeds, the call never runs, and the harness would record an agent that did nothing. That is indistinguishable from a genuine reasoning failure, which is the one confound this benchmark cannot tolerate.

Tool sets under test

Granular gives the agent six primitives: list traces, get topology, list spans, get one span, get the critical path, and compare a service against its baseline. The agent assembles the picture itself.

Analytical gives it one call returning the topology, the critical path annotated with baseline comparisons, and error spans ranked by depth.

jaeger_mcp mirrors the tools Jaeger's MCP server actually exposes: get_services, get_span_names, search_traces, get_trace_topology, get_span_details, get_trace_errors. Two differences from the other sets are deliberate and are the interesting part. It has no baseline comparison, because Jaeger's surface has none, so an agent must judge whether a duration is abnormal unaided. And errors come from a separate call rather than arriving alongside timing, so an agent has to decide to go looking for them.

That third set is why the results transfer. A finding about granular versus analytical is a finding about two designs I invented; a finding about jaeger_mcp is a finding about the tools people are already using.

No set names a culprit. analyze_trace returns evidence and stops there, and a test asserts it contains no likely_root_cause field. A tool that handed over the answer would make the comparison measure nothing except which set was told what to say.

Instruction styles under test

skills/strict_rca.md is a six-step procedure that tells the agent to separate self time from total duration, check baselines before calling anything slow, and order errors by depth. skills/exploratory_rca.md is four short paragraphs that say to investigate and form a hypothesis.

The agent reads these files from disk. They are the variable being tested, so a hardcoded copy in the source would defeat the experiment.

Real Jaeger traces

traces/backend.py defines the interface tools read through:

trace_ids()      # which traces exist
index()          # catalogue with duration and error status
trace(trace_id)  # the spans

traces/jaeger.py implements it against Jaeger's trace JSON. Two conversions there do real work rather than renaming fields. Jaeger reports total span duration while the benchmark reasons about self time, so each span's self time is derived by subtracting its direct children; without that, a caller waiting on a slow callee looks exactly as slow as the callee and the timeout scenario stops being distinguishable. And error status has no single representation in practice, so an error tag, an OpenTelemetry status code, and an HTTP status of 400 or above are all accepted.

What is tested and what is not. The parsing is covered by fixtures in Jaeger's documented response shape, and a test drives the existing tools through a JaegerBackend end to end. The transport is injected as a callable and has not been run against a live Jaeger deployment, so the network path and the exact MCP envelope are unverified. Wiring a real client means supplying that callable, not changing the mapping.

Ground truth stays out of the backend protocol deliberately, since a real deployment has no expected service to hand over. The benchmark keeps reading that from the scenario files while spans come from wherever the backend gets them, which is what makes synthetic and real runs comparable.

Layout

traces/       span model, scenario builder, trace store, backend protocol, Jaeger adapter
agent/        tool schemas, Jaeger MCP tool set, tool execution, agent loop, baseline
evaluation/   outcome classification, metrics, report rendering
scenarios/    scenario definitions
skills/       instruction styles
experiments/  A/B runners
tests/        185 tests
cli.py        entry point

Relation to the Jaeger LFX mentorship

RCAgentBench is an independent prototype exploring the research problem in Jaeger's 2026 LFX mentorship project, "Benchmarking the AI Assistant's MCP Tools and Skills." It works on the same questions the proposal identifies: tool shape, Skill instructions, trace-solvable fault scenarios, and trajectory-level metrics. The implementation is deliberately independent of Jaeger's codebase.

It is not affiliated with the Jaeger project and is not an implementation of the mentorship deliverable.

The jaeger_mcp tool set is the closest point of contact: it mirrors the tool names Jaeger's MCP server exposes, so the same evaluation suite can compare Jaeger's real tool surface against alternative designs. The next integration step is pointing JaegerBackend at a running Jaeger instance, which would let the same scenarios run against synthetic and real spans and show whether findings from one hold in the other.

Next

• Run the model comparison and publish the numbers
• Point JaegerBackend at a live Jaeger instance and validate the MCP envelope
• Add scenarios where two services are genuinely ambiguous from the spans alone
• Support parallel sibling calls; the builder currently runs them in sequence
• Report confidence intervals once repeat counts justify them

License

MIT