Skip to content
106 changes: 106 additions & 0 deletions docs/ADRs/0090-forge-pointer-correlation-for-agent-traces.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
---
title: "90. Harness snapshot and forge pointer correlation for agent traces"
status: Accepted
relates_to:
- operational-observability
topics:
- observability
- telemetry
- opentelemetry
- forge
---

# 75. Harness snapshot and forge pointer correlation for agent traces

Date: 2026-07-22

## Status

Accepted

## Context

[ADR 0050](0050-distributed-tracing-instrumentation.md) defines how fullsend
generates traces (OTel Go SDK; Level 1 = `run-telemetry.jsonl`; Level 2 =
OTLP). [Operational observability](../problems/operational-observability.md)
requires forge **pointers** (repo + SHA, change id, pipeline run) rather than
duplicating diffs, and a stable answer to: “what harness/config produced this
run?” Forge-neutral hosting concepts remain those in
[ADR 0005](0005-forge-abstraction-layer.md); this ADR only decides how join
pointers are recorded on a run.

After the OTel SDK migration, `run-summary.json` was removed; the sole Level 1
telemetry file today is `run-telemetry.jsonl`. Eval wrappers that `export`
ambient CI variables are the wrong layer: harness identity is a **run-time**
concern of `fullsend run`, and downstream tools must not scrape CI env as the
source of truth. Cross-project join field names live in a shared join contract;
this ADR decides only what **fullsend** writes. Literal OTel attribute names
below are **illustrative** pending that contract
([ABEvalFlow #57](https://github.com/RHEcosystemAppEng/ABEvalFlow/pull/57));
the follow-up implementation issue freezes the final strings.

## Options

**A. Root-span attributes only (no snapshot file).** Join keys exist in OTLP
backends, but offline/forensic consumers must parse spans, and keys disappear
when span export is disabled.

**B. Embed a snapshot event in `run-telemetry.jsonl`.** One Level 1 file, but
couples the harness/config contract to telemetry format and exporter semantics.

**C. Eval-script / ambient CI env exports.** Easy to prototype; unstable across
local runs, child dispatches, and forges; encourages scraping rather than an
explicit artifact.

**D. Dedicated `harness-snapshot.json` plus mirrored root-span attributes
(chosen).** Stable local run-start contract for offline consumers, plus
backend-friendly correlation for OTLP.

## Decision

Every `fullsend run` records run-start join keys by writing
**`harness-snapshot.json`** next to `run-telemetry.jsonl` **and**, when tracing
is enabled, mirroring those keys on the root `run` span (illustrative OTel
namespaces today: `vcs.*`, `cicd.*`, plus harness content SHA and forge
platform — finalize literal names with the shared join contract / follow-up
issue). The file holds pointers and a config fingerprint only (harness
identity, content hash, forge/CI pointers when known) — no diffs, prompts, or
skill bodies. Forge/CI fields are filled at write time (`FULLSEND_*` overrides
first, then standard CI env). Consumers read the JSON (or stores that ingest
it), not ambient CI env.

When tracing is enabled, write the snapshot after the root span exists and
include `trace_id` / `traceparent` so the file joins to `run-telemetry.jsonl`
and OTLP. When `OTEL_SDK_DISABLED=true`, still write the snapshot (config/forge
contract, not span export) but **omit** `trace_id` / `traceparent` and skip
root-span attribute mirroring — there is no usable trace context.

## Consequences

- Operators and downstream loggers can join a run to forge/CI via a single
decided artifact; when tracing is on, `trace_id` joins to `run-telemetry.jsonl`.
- OTLP backends receive join keys without parsing the JSON when tracing is on;
local JSON remains the forensic / handoff contract.
- Dispatched child runs must inherit forge context via env so child snapshots
are complete.
- GitHub/GitLab coverage depends on CI vars or `FULLSEND_*` overrides — both
have in-tree `forge.Client` implementations today (see `internal/forge/github/`
and `internal/forge/gitlab/`). Forgejo is defined as a future forge target in
[ADR 0005](0005-forge-abstraction-layer.md) and needs a `forge.Client`
implementation before this join-key behavior applies there.
(Note: ADR 0005 predates the GitLab client landing and still groups GitLab
with Forgejo as unimplemented; the in-tree code is the authoritative source
for current forge support status.)
- Implementation will be tracked in a **new** follow-up issue filed after this
ADR merges (per repo convention). Do not cite closed #5449. Detailed field
lists for cross-project consumers stay in the shared join contract, not
duplicated here.

## Related

- [ADR 0050](0050-distributed-tracing-instrumentation.md)
- [ADR 0005](0005-forge-abstraction-layer.md)
- Shared join contract (pending merge; not yet on `main`):
[ABEvalFlow #57](https://github.com/RHEcosystemAppEng/ABEvalFlow/pull/57)
- [#2368](https://github.com/fullsend-ai/fullsend/issues/2368)
- [#294](https://github.com/fullsend-ai/fullsend/issues/294)
1 change: 1 addition & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -303,6 +303,7 @@ Observability is a cross-cutting concern that touches every other component. Eac
- JSONL reasoning trace exposure: raw JSONL conversation transcripts are extracted from sandboxes and stored with owner-scoped access. Credential scanning acts as an invariant check on [ADR 0017](ADRs/0017-credential-isolation-for-sandboxed-agents.md)'s isolation model. Agents handling data from protected sources beyond the target repo can opt in to JSONL suppression via configuration ([ADR 0021](ADRs/0021-jsonl-reasoning-trace-exposure.md)).
- Event-driven stage dispatch remains traceable end-to-end in the GitHub Actions UI by using synchronous `workflow_call` dispatch (see [ADR 0041](ADRs/0041-synchronous-workflow-call-event-dispatch.md)).
- Distributed tracing: framework-native OpenTelemetry instrumentation with zero-configuration baseline. Every run produces `run-telemetry.jsonl` locally; optional live OTLP export to any compatible backend. W3C trace context propagation links multi-agent pipelines into unified traces. OTEL GenAI semantic conventions enable LLM-aware backends ([ADR 0050](ADRs/0050-distributed-tracing-instrumentation.md)).
- Decided (implementation follow-up after merge): harness snapshot join keys — every run will write `harness-snapshot.json` (harness fingerprint + forge/CI pointers) and, when tracing is enabled, mirror those keys on the root span so offline and OTLP consumers can correlate without scraping CI env ([ADR 0090](ADRs/0090-forge-pointer-correlation-for-agent-traces.md)).

**Open questions:**

Expand Down
6 changes: 6 additions & 0 deletions docs/guides/infrastructure/distributed-tracing.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,12 @@ Level 3 adds prompt/completion content to spans.

## Environment variables

> **Planned:** `harness-snapshot.json` (run-start harness fingerprint and
> forge/CI pointers, mirrored on the root span) is decided in
> [ADR 0090](../../ADRs/0090-forge-pointer-correlation-for-agent-traces.md).
> Implementation will be tracked in a follow-up issue after that ADR merges.
> Until that ships, only `run-telemetry.jsonl` is guaranteed on disk.

### Endpoint configuration

| Variable | Purpose | Notes |
Expand Down