telemetry: hierarchical work-graph correlation IDs for cost rollup (leaf / parent / feature)

[ascerra]

Summary

Emit forge-neutral, Level 1 correlation attributes on every agent root span so operators can search, group, and sum fullsend.cost_usd across related traces — not only for a single run, but for:

1. This work item (leaf / the entity that triggered the run)
2. The parent work item (one level up in the work graph)
3. The feature / root (top of the decomposition tree, typically two+ levels above the leaf)

This is a Fullsend telemetry capability, not an MLflow feature. Any OTLP backend (or offline run-telemetry.jsonl) must be able to answer “what did this task / epic / feature cost across triage → code → review → fix → retro?”

Why this is needed

Today each managed run already records accurate, cache-aware cost as fullsend.cost_usd and a per-run entity as fullsend.work_item_id (e.g. owner/repo#N). That is enough for cost of this job.

It is not enough for:

• Same work across issue → PR. GitHub issue numbers and PR numbers differ. Triage/code on issue #100 and review/fix/retro on PR #105 do not share work_item_id, so operators cannot sum “cost of this PR’s Fullsend chain” without manual join.
• Fan-out. Triage (or a future refine/decompose agent) may close one issue and open several children. One flat id cannot express both “cost of the closed parent decision” and “cost of each child.”
• Decomposition hierarchies. We will have agents that break large features into smaller work. On GitHub that may be sub-issues; on Jira that may be epic → story → task (or similar). Operators will want cost at leaf, parent, and feature independently.

Related but not a substitute:

• #5361 — MLflow Name/Session/User/Source blanks + token/cost display fidelity. Adjacent enrichment; does not define hierarchical join keys or issue≠PR lifecycle semantics.
• W3C TRACEPARENT — correct for stitching one dispatched pipeline into a single OTEL trace; wrong for independent GHA/GitLab jobs over days that should remain separate traces and still be summable.
• ADR 0050 Level 3 content capture (e.g. #6429) — prompts/completions/reasoning. This issue is Level 1 metadata (always-on attributes), not content.

Goals

A. Leaf lifecycle (same work unit across forge shape changes)

For the happy path “triage → code → review → (fix)* → retro” on one work unit, every related root span carries a stable leaf lifecycle id that does not change when the forge entity flips from issue → PR (or Jira equivalent).

fullsend.work_item_id remains “the entity that triggered this run.” The lifecycle id is the join key across those runs.

B. Hierarchical rollup (decomposition / sub-work)

Every leaf run also carries enough ancestry to answer three cost questions:

Rollup	Answers	GitHub-ish examples	Jira-ish examples
Work item (leaf)	Cost of this task	issue, PR, or sub-issue	story, task, bug
Parent	Cost of the container that owns these leaves	parent issue of sub-issues	epic (or immediate parent)
Feature / root	Cost of the whole decomposition	top-level issue / feature root	initiative or feature-root epic

Illustrative attribute names (exact names are an implementation decision; prefer portable OTel + fullsend.*):

• fullsend.work_item_id — already exists (this run’s trigger)
• fullsend.parent_work_item_id — immediate parent in the work graph (when known)
• fullsend.feature_id or fullsend.root_work_item_id — top of the decomposition tree (when known)
• Optional: fullsend.work_item_path — ordered ancestors if nesting can exceed two levels

C. Backend-agnostic emission

• Emit from the CLI / managed path as Level 1 span attributes (same class as today’s work_item_id / cost_usd).
• Do not require mlflow.* attributes in core. Orgs may use MLflow, Phoenix, Jaeger, a collector, or JSONL only.
• If we also set OTel session.id so GenAI UIs (including MLflow’s Session column) light up, document which rollup key maps to session.id (leaf lifecycle vs feature). One Session column cannot express a three-level tree; hierarchy stays on explicit attributes.

D. Queryability (acceptance of the capability)

Given traces for a completed flow, an operator (or script) can:

1. Filter/search by leaf lifecycle id and sum fullsend.cost_usd across triage + N code/fix + review + retro.
2. Filter by parent id and sum cost of all child leaves under that parent.
3. Filter by feature/root id and sum cost of the whole decomposition.

Authoritative cost is always fullsend.cost_usd, not a backend’s derived Cost column (cache tokens are often omitted — see #5361).

Non-goals

• Baking vendor score/Assessment APIs or product-specific env into managed workflows (ADR 0087 / tool-agnostic core).
• Replacing TRACEPARENT for same-job dispatched children.
• Level 3 content capture.
• A same-job eval-measure scorer that invents cross-run totals from one artifact — chain rollup is a derived product over a store of traces (or offline JSONL), not EM-001.

Design questions to resolve in implementation / ADR notes

1. Issue → PR continuity: Prefer keeping the same leaf lifecycle id when a PR is opened from an issue (PR continues the work unit; it is not a fork).
2. Fan-out (1→N): When an agent closes #100 and opens #101/#102/#103, prefer fork with parent link (each child gets a new leaf lifecycle id; parent/feature ids preserved or linked via previous / parent attributes) over forcing one id across unrelated children. Document how to answer both “cost of the triage decision on #100” and “cost of child #101.”
3. Resolution at run time: How does the runner learn parent/feature when the event only names the leaf? (Forge APIs differ: GitHub sub-issues vs Jira epic link / parent. Agents that create children must stamp new leaves.)
4. Depth: Is parent + feature enough, or do we need an explicit path for deeper trees?
5. session.id mapping: Which single key (if any) is duplicated into OTel session.id for UI grouping?

Proposed acceptance criteria

• Spec (short ADR section or tracing guide update) defines leaf lifecycle vs work_item_id vs parent vs feature/root, including issue→PR and 1→N fork rules.
• Root spans for managed agent runs emit the agreed Level 1 attributes when resolvable; missing ancestry is explicit (absent or sentinel), not silently wrong.
• Documented query pattern: group by each key and SUM(fullsend.cost_usd) using OTLP backend search or offline run-telemetry.jsonl.
• Dogfood evidence: at least one real multi-agent chain (issue → PR) shares a leaf lifecycle id; at least one decomposed / parented example populates parent and feature/root.
• No mlflow.* (or other vendor) attributes required in core for this to work.
• Cross-link from #5361: Session column enrichment may consume session.id once mapping is decided; this issue owns the hierarchy semantics.

Related

• ADR 0050 — framework-native OTEL; Level 1 metadata; TRACEPARENT for dispatched pipelines
• ADR 0087 — eval measurements as derived products; tool-agnostic export
• #5361 — MLflow column / cost display enrichment (related consumer)
• #2368 — harness content SHA / join-adjacent attrs
• Docs: docs/guides/infrastructure/distributed-tracing.md (fullsend.work_item_id as today’s primary cross-run key)

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 7:00 PM UTC · Completed 7:04 PM UTC

Commit: ff93b07 · View workflow run →

[fullsend-ai-triage]

Triage assessment

Severity: medium · Category: feature

This is a well-specified feature request with clear acceptance criteria. The five design questions documented in the issue are architectural decisions to resolve during implementation — they do not block triage.

Related work

• telemetry: MLflow trace columns (Name/Session/User/Source) blank; token & cost omit cache tokens #5361 (open) — MLflow column enrichment. Complementary consumer; once session.id mapping is decided here, telemetry: MLflow trace columns (Name/Session/User/Source) blank; token & cost omit cache tokens #5361 can use it for the Session column. Not a prerequisite in either direction.
• feat: add fullsend.harness.content_sha trace attribute for config change correlation #2368 (open) — Harness content SHA attribute. Adjacent join key, not blocking.
• feat(telemetry): implement ADR 0050 Level 3 content capture #6429 (open PR) — Level 3 content capture. Different telemetry level; no dependency.

Proposed test case

// TestHierarchicalCorrelationAttributes verifies that when a run is
// triggered with parent/feature context, the root span carries the
// expected Level 1 hierarchy attributes.
//
// Setup: Create a mock forge context where issue #101 has parent #100
// and feature root #50. Trigger a managed run for #101.
//
// Assertions:
// - Root span has fullsend.work_item_id = "owner/repo#101"
// - Root span has fullsend.lifecycle_id (non-empty, stable identifier)
// - Root span has fullsend.parent_work_item_id = "owner/repo#100"
// - Root span has fullsend.root_work_item_id = "owner/repo#50"
//
// Edge cases:
// - When no parent exists, parent attribute is absent (not empty string)
// - When parent == root, both attributes have the same value
// - Issue-to-PR transition preserves lifecycle_id
// - Fan-out: child issues get new lifecycle_ids but inherit parent/root
// - run-telemetry.jsonl contains the same attributes as OTLP export

Implementation notes

The primary implementation surface is the CLI managed path (Go code under cmd//internal/), specifically where root span attributes are set. The distributed tracing guide (docs/guides/infrastructure/distributed-tracing.md) and ADR 0050 will need updates to document the new attributes and query patterns.

---

Labels: This is a feature request for the runner/telemetry component (span attribute emission in the CLI managed path), at medium priority since it enables cost analysis but is not a regression.

---

Next steps:

• /fs-code — agent creates a PR to implement this issue
• /fs-code <your instruction> — agent implements with your specific guidance