telemetry: MLflow trace columns (Name/Session/User/Source) blank; token & cost omit cache tokens

[dhshah13]

What happens

Now that L2 OTLP export is live to the pilot MLflow (post-#3749, fullsend-ai dogfood runs landing in experiment 1), the experiment Traces view populates only a few columns. Across the 50 most recent real traces:

Column	Populated	Note
Name	0/50	blank on every trace
Session	0/50	can't group runs by issue/PR
User	0/50	triggering actor unknown
Source	0/50	repo / workflow / commit unknown
Request / Response	0/50	content — L3, expected empty
Tokens	50/50	derived from input_tokens/output_tokens only
Cost	50/50	undercounts — see below

Sample (tr-6d451616…, 218s run): tokenUsage {input: 30, output: 13536}, cost {input_cost: 0.00015, output_cost: 0.338}. The run consumed ~100k+ cached input tokens, but input_tokens: 30 (and thus input_cost: ~$0.0002) reflects only non-cached input — MLflow derives token/cost from gen_ai.usage.input_tokens/output_tokens and ignores the cache_read_input_tokens / cache_creation_input_tokens we emit.

Why it matters

Name / Session / User / Source are the columns that make the trace list navigable and groupable (per issue/PR, per actor, per repo). Cost is the pilot's headline metric; undercounting cached input by orders of magnitude undermines it.

Root cause

The exporter emits generic OTel attributes only — fullsend.*, gen_ai.usage.*, fullsend.cost_usd — and no mlflow.* trace-level attributes. MLflow auto-derives tokenUsage/cost from gen_ai.usage.input/output, but has no source for Name/Session/User/Source and does not fold cache tokens into the input figure.

Raw material we already emit (root span)

fullsend.agent, fullsend.work_item_id, gen_ai.request.model, gen_ai.usage.{input,output,cache_read,cache_creation}_input_tokens, fullsend.cost_usd (cache-aware, accurate), fullsend.tool_calls, fullsend.num_turns, fullsend.iterations. Not captured today: triggering actor and source repo/commit/run-url as span attributes.

Candidate mappings (illustrative, not prescriptive)

• Name ← fullsend.agent (optionally + work_item_id)
• Session ← fullsend.work_item_id (groups all runs for one issue/PR)
• User ← triggering actor (needs capture from CI context, e.g. GITHUB_ACTOR / issue author)
• Source ← repo + commit + run URL (needs capture from CI env)
• Tokens/Cost ← include cache tokens in the input figure, and/or surface fullsend.cost_usd directly

Open design question

Populating MLflow's Name/Session/User/Source requires emitting mlflow.*-prefixed trace attributes, which couples the backend-agnostic exporter to MLflow. Options to decide: emit mlflow.* attributes per ADR 0050; keep the exporter generic and accept blank MLflow columns; or map at a Collector/backend layer. The token/cost fidelity gap is partly upstream (MLflow's extractor ignores cache fields) and partly ours (we could emit a combined input count or the cost directly).

Out of scope

Request/Response = prompt/completion content = L3 (governance-gated). Not part of this issue.

Relates to #2862.

[dhshah13]

@ascerra — requesting your review. The blocking design call: should the exporter emit mlflow.* trace-level attrs to populate Name/Session/User/Source (couples the backend-agnostic exporter to MLflow, per ADR 0050), or stay generic and accept blank columns? That decision scopes the rest, including the cache-token cost fidelity fix.

[rh-hemartin]

Hello! I'm testing stuff and I can populate User and Session from OpenTelemetry (user.id and session.id). Also I found that our attributes are wrong for cache, they should be gen_ai.usage.cache_creation.input_tokens with a dot between cache_creation and input_tokens (see https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md and mlflow/mlflow#22818). Now they have an underscore.

Currently I do not know how to setup Name and Source. If we really want those we can have a variable that controls if those mlflow attributes are added. Something like "FULLSEND_MLFLOW_OTEL_ATTRIBUTES=true".

[fullsend-ai-triage]

Meeting update — 2026-07-23

Relevant to this issue: The team identified discrepancies between file-based reports and MLflow metrics as a blocking issue that must be resolved before enabling tracing for all agents (#5528).

• Investigation of data reporting discrepancies was assigned as a priority action item
• The team is restricting tracing activity to the non-production validation environment until data quality is confirmed
• Resolving these discrepancies gates the broader tracing rollout across all agents

Related PRs: #5442 addresses a related problem where collector header flags are silently ignored, potentially contributing to misrouted telemetry data

Next steps:

• Investigate root causes of discrepancies between file-based reports and MLflow metrics
• Validate data quality in the non-production environment before expanding tracing scope

Meeting notes

[dhshah13]

Span status is set independently of run outcome, which feeds the file-vs-MLflow discrepancy directly. Both status writes key solely on runErr == nil — agent span at internal/cli/run.go:1307-1311, root span at run.go:853-857 — and the agent span ends before the #2786 is_error transcript check runs (run.go:1337-1344), so its status can never reflect that check.

Verified live on main 3c57dda9 with three local minimal-explore runs:

Run	Transcript	Agent spans in run-telemetry.jsonl
auth failure (invalid_grant)	is_error=true, exit 1, $0, 0 tokens	Status OK, exit_code=1
Vertex 403 (billing)	is_error=true, exit 1, $0, 0 tokens	Status OK, exit_code=1
green control	success, 7 turns, $0.30	Status OK, exit_code=0

Status reads OK on failed and successful runs alike, so any consumer keying on span or trace status — MLflow trace state, dashboards, filters — counts failed iterations as successes. The root span's status is correct only when a validation loop exhausts and returns an error; harnesses without a validation_loop return nil and ship the root span Status OK alongside exit_code=1. No RecordError or exception events exist on any path.