feat(eval)!: add eval measurements and EM-001 trace_fitness scorer

action.yml

@@ -409,6 +409,60 @@ runs:
           "${STATUS_FLAGS[@]+"${STATUS_FLAGS[@]}"}" \
           "${MINT_FLAGS[@]+"${MINT_FLAGS[@]}"}"
 
+    # Eval measurements (fail-open): score run-telemetry.jsonl with the agents
+    # measurement manifest. Same job as fullsend run; never fails the agent.
+    # Writes eval-measurements.jsonl when at least one new score row is produced
+    # (tool-agnostic artifact); missing telemetry/manifest skips with no file.
+    # Manifest trust: prefer a local override from the PR base SHA (same
+    # trusted tip reusable-* workflows check out for kill-switch config).
+    # Path prefix comes from inputs.fullsend-dir (install layout: per-repo
+    # `.fullsend/` or per-org workspace root); file bytes come from the
+    # trusted ref — never from a PR-head working tree. When no trusted
+    # override exists, fetch SHA-pinned agents@v0 (no --fullsend-dir).
+    - name: Eval measurements
+      if: always() && inputs.agent != '__install_only__'
+      continue-on-error: true
+      shell: bash
+      env:
+        AGENT: ${{ inputs.agent }}
+        FULLSEND_DIR: ${{ inputs.fullsend-dir }}
+        # For GetRef of agents@v0 (SHA pin). Not sent to raw.githubusercontent.com.
+        GH_TOKEN: ${{ inputs.github_token }}
+        PR_BASE_SHA: ${{ github.event.pull_request.base.sha }}
+      run: |
+        set -euo pipefail
+        MEASURE_ARGS=(--agent "${AGENT}" --output-dir "${GITHUB_WORKSPACE}/output")
+        TRUSTED_REF="${PR_BASE_SHA:-}"
+        if [[ -z "${TRUSTED_REF}" ]]; then
+          TRUSTED_REF="${GITHUB_SHA:-}"
+        fi
+        FULLSEND_DIR="${FULLSEND_DIR:-.fullsend}"
+        MEASURE_REL=""
+        case "${FULLSEND_DIR}" in
+          "${GITHUB_WORKSPACE}"/*)
+            MEASURE_REL="${FULLSEND_DIR#"${GITHUB_WORKSPACE}"/}"
+            ;;
+          /*)
+            MEASURE_REL=""
+            ;;
+          *)
+            MEASURE_REL="${FULLSEND_DIR}"
+            ;;
+        esac
+        MEASURE_REL="${MEASURE_REL#./}"
+        MEASURE_REL="${MEASURE_REL%/}"
+        if [[ -n "${TRUSTED_REF}" && -n "${MEASURE_REL}" ]] \
+          && git cat-file -e "${TRUSTED_REF}:${MEASURE_REL}/eval/measurements/${AGENT}.yaml" 2>/dev/null; then
+          MEASURE_FILE="${GITHUB_WORKSPACE}/output/.fullsend-measure-${AGENT}.yaml"
+          mkdir -p "${GITHUB_WORKSPACE}/output"
+          git show "${TRUSTED_REF}:${MEASURE_REL}/eval/measurements/${AGENT}.yaml" > "${MEASURE_FILE}"
+          MEASURE_ARGS+=(--registry "${MEASURE_FILE}")
+        fi
+        # Binary resolves SHA-pinned agents@v0 when --registry is unset
+        # (allowlist/hash/audit) and scores only platform run-telemetry.jsonl
+        # at the top of each runDir.
+        fullsend eval-measure "${MEASURE_ARGS[@]}"
+
     - name: Upload fullsend artifacts
       if: always() && inputs.agent != '__install_only__'
       uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1

docs/.vitepress/config.ts

@@ -260,6 +260,7 @@ export default defineConfig({
             { text: "Standalone Mint", link: "/guides/infrastructure/standalone-mint" },
             { text: "Private Repositories", link: "/guides/infrastructure/private-repositories" },
             { text: "Tracing Reference", link: "/guides/infrastructure/distributed-tracing" },
+            { text: "Eval Measurements", link: "/guides/infrastructure/eval-measurements" },
             { text: "Advanced Setup", link: "/guides/infrastructure/advanced-setup" },
             {
               text: "Layered Config Reference",

docs/ADRs/0050-distributed-tracing-instrumentation.md

@@ -156,6 +156,14 @@ artifact. OTLP export also changed from post-hoc directory upload to live
 span export via the OTel SDK's batch processor. The core decision (three-level
 opt-in, OTel-native, W3C propagation) is unchanged.
 
+**2026-08-10 — Eval measurements ([ADR 0087](0087-eval-measurements-online-trace-scoring.md)):**
+online scoring of wild-run traces writes `eval-measurements.jsonl`
+beside telemetry when at least one new score is produced (tool-agnostic). Distinct from functional eval fixtures
+([ADR 0051](0051-agent-eval-harness-for-test-infrastructure.md)).
+
+> **Planned:** portable remote score export follows the same OTLP
+> configuration as this ADR — no vendor score adapters in core.
+
 **2026-08-18 — Remove duplicate token/cost from root span (3278b059):**
 `gen_ai.request.model` and `gen_ai.usage.*` token attributes moved to agent
 spans only; the root span keeps `fullsend.cost_usd` and `fullsend.tool_calls`

docs/ADRs/0087-eval-measurements-online-trace-scoring.md

@@ -0,0 +1,151 @@
+---
+title: "87. Eval measurements as online trace scoring with portable export"
+status: Accepted
+relates_to:
+  - operational-observability
+  - testing-agents
+topics:
+  - observability
+  - evaluation
+  - opentelemetry
+---
+
+# 87. Eval measurements as online trace scoring with portable export
+
+Date: 2026-08-10
+
+## Status
+
+Accepted
+
+## Context
+
+Agent runs already emit OpenTelemetry traces as `run-telemetry.jsonl`, with
+optional live OTLP export when `OTEL_EXPORTER_OTLP_*` is set
+([ADR 0050](0050-distributed-tracing-instrumentation.md)). Separately,
+[ADR 0051](0051-agent-eval-harness-for-test-infrastructure.md) owns the
+**functional** eval harness: curated fixtures / scenarios in
+`fullsend-ai/agents` `eval/<agent>/` that gate agent PRs. Those fixtures do
+not score wild production runs.
+
+Operators also need an **online / trend** layer on wild traces (completeness
+first; quality signals later). Fullsend must stay **backend-agnostic**: orgs
+already choose Phoenix, MLflow, Jaeger, or another OTLP collector for traces.
+Baking a single product’s Assessments/Quality API into the core CLI or managed
+workflows would force a tool decision on every install.
+
+Adjacent telemetry work (not competing with this score path):
+
+- **Level 3 content capture** ([ADR 0050](0050-distributed-tracing-instrumentation.md);
+  activation draft closed without merge in
+  [#5947](https://github.com/fullsend-ai/fullsend/pull/5947)): first ship
+  reads Level 1/2 metadata in `run-telemetry.jsonl` (fitness foundation).
+  Content-aware scorers on prompt/completion bodies are the intended next
+  layer once Level 3 is implemented. Measure CLI is host-side after sandbox
+  exit.
+- **Span status from run outcome**
+  ([#5944](https://github.com/fullsend-ai/fullsend/pull/5944), merged):
+  OTLP Status (and `fullsend.transcript_error`) become the reliable
+  success/failure signal. EM-001 only checks that `exit_code` is **present**
+  (fitness). Outcome scorers must key on Status, not `exit_code == 0`.
+- **Observer / lessons → fixtures** (draft closed without merge in
+  [#2423](https://github.com/fullsend-ai/fullsend/pull/2423)): narrative
+  analysis and golden-set promotion remain a sibling idea. This ADR is
+  same-job deterministic scoring on traces.
+- **Harness snapshot / forge join keys**
+  ([#5524](https://github.com/fullsend-ai/fullsend/pull/5524), open):
+  sibling artifact for harness fingerprint and forge/CI pointers beside
+  telemetry. Complementary join/identity layer; primary run facts belong on
+  the OTEL trace (Level 1), while measurements stay a derived sibling file.
+
+## Options
+
+1. **Local JSONL only** — portable offline artifact; no remote scores from
+   fullsend itself.
+2. **Backend-native APIs in core** (e.g. one vendor’s Assessments API) —
+   couples every managed workflow to that product’s auth and schema.
+3. **Local JSONL + same OTLP path as agent traces for remote** — scores travel
+   with the endpoint/headers orgs already configure for ADR 0050; no second
+   vendor stack in core.
+
+## Decision
+
+Introduce **eval measurements**: deterministic scorers that read
+`run-telemetry.jsonl` after `fullsend run` in the **same** managed job
+(`fullsend eval-measure` in `action.yml`), **fail-open**. Functional eval
+scenarios remain ADR 0051 / `eval/<agent>/`; measurements never block
+delivery.
+
+In plain terms: eval measurements are the concept of scoring traces.
+[OTEL primary facts](../glossary.md#otel-primary-facts) are what happened
+on the run (the OTEL trace / `run-telemetry.jsonl`).
+[OTEL derived products](../glossary.md#otel-derived-products) are scores
+computed from that trace (`eval-measurements.jsonl`). Measurements never
+rewrite primary facts, and they are [fail-open](../glossary.md#fail-open).
+
+Scores land in a tool-agnostic `eval-measurements.jsonl` (plus a
+small idempotency ledger) next to `run-telemetry.jsonl` whenever at least
+one new measurement row is produced (including `label: skip`). Remote score export
+will use the same `OTEL_EXPORTER_OTLP_*` configuration as ADR 0050 — no
+vendor-specific score adapters in core. `fullsend` owns the parser, scorers,
+CLI, and GHA step; `fullsend-ai/agents` owns per-agent measurement manifests
+(`eval/measurements/<agent>.yaml`) that declare which scorers to enable.
+Stock-agent defaults resolve from `agents@v0` at runtime; local files are for
+override, opt-out, or custom agents only. Activation is **two-step**: merge
+measurement manifests into `fullsend-ai/agents` **and** cut a `v0.x.y` release
+that re-points the floating `v0` tag. Merging alone does not activate managed
+jobs. Tracking: [#6384](https://github.com/fullsend-ai/fullsend/issues/6384).
+Until that release lands, GHA/GitLab `eval-measure` wiring is provisional
+(clean skip when the remote manifest is missing). Local `FULLSEND_DIR`
+manifests are exercised in unit tests today.
+
+The first scorer is `trace_fitness` (catalog id `em-001`) — span-tree and
+attribute fitness so later scorers can trust the trace. EM-001 reads
+OpenTelemetry GenAI attribute names (`gen_ai.*` constants in
+`internal/evalmeasure`). `gen_ai.system` was renamed to `gen_ai.provider.name`
+in semconv v1.37.0; `modelOK` accepts either so `em-001@1` survives the
+emitter migration. Other upstream renames remain an `em-001` version bump.
+Pre-script-skipped runs, runs with no `agent` span (never reached an
+iteration), and runs where agent spans flushed but the root `run` span never
+ended (hard kill / timeout) record `label: skip` and are excluded from
+pass/(pass+fail).
+
+### Versioning (per measurement, not platform “v1”)
+
+There is no product-wide “eval measurements v1” switch. “First ship” just
+means only one scorer is enabled yet. Each manifest entry carries:
+
+| Field | Meaning |
+|---|---|
+| `id` | Stable catalog id (`em-001`). New measurement concept → new id. |
+| `scorer` | Go dispatch name (`trace_fitness`). |
+| `version` | Integer **contract** version of that measurement’s checks / pass rule. |
+
+Scores and the idempotency ledger key on `id@version` (e.g. `em-001@1`).
+Bump `version` when pass/fail semantics change so trends do not mix eras.
+Add a check that does not change the pass definition → same version is fine.
+Entirely new signal → new `em-NNN` (and usually a new `scorer` string).
+
+## Consequences
+
+- Every measured run produces a reviewable, backend-agnostic score file beside
+  telemetry; missing manifests skip cleanly and measure failure never fails
+  the agent job. GitHub Actions is the first-ship managed path (uploads
+  `output/`). GitLab CI calls the same fail-open `eval-measure` CLI under
+  `$CI_PROJECT_DIR/output` with `artifacts: when: always`. Stock manifests
+  fetch from public `agents@v0` even without `GH_TOKEN` (rate-limited); a
+  token is recommended on shared runners.
+- Core stays tool-agnostic: no product-specific score env vars in managed
+  workflows; remote scores follow OTEL when that path lands.
+- Functional scenarios (gate) and eval measurements (trend) stay separate;
+  retro can recommend either a manifest scorer or a scenario fixture.
+- Level 1/2 metadata scorers (EM-001) are the foundation; Level 3 content
+  capture expands what scorers *can* assert (quality / LLM-judge style) once
+  implemented — it does not replace this same-job path.
+- Per-measurement versioning (`id@version`) lets pass/fail semantics evolve
+  without mixing trend eras.
+- Pre-script skipped runs (`fullsend.prescript.skipped=true` on the root span),
+  runs with no `agent` span (never reached an iteration), and runs where agent
+  spans flushed but the root `run` span never ended (hard kill / timeout) are
+  excluded from EM-001: the scorer writes `label: skip` instead of failing a run
+  that never produced a full telemetry contract.

docs/architecture.md

@@ -319,11 +319,15 @@ Observability is a cross-cutting concern that touches every other component. Eac
 - JSONL reasoning trace exposure: raw JSONL conversation transcripts are extracted from sandboxes and stored with owner-scoped access. Credential scanning acts as an invariant check on [ADR 0017](ADRs/0017-credential-isolation-for-sandboxed-agents.md)'s isolation model. Agents handling data from protected sources beyond the target repo can opt in to JSONL suppression via configuration ([ADR 0021](ADRs/0021-jsonl-reasoning-trace-exposure.md)).
 - Event-driven stage dispatch remains traceable end-to-end in the GitHub Actions UI by using synchronous `workflow_call` dispatch (see [ADR 0041](ADRs/0041-synchronous-workflow-call-event-dispatch.md)).
 - Distributed tracing: framework-native OpenTelemetry instrumentation with zero-configuration baseline. Every run produces `run-telemetry.jsonl` locally; optional live OTLP export to any compatible backend. W3C trace context propagation links multi-agent pipelines into unified traces. OTEL GenAI semantic conventions enable LLM-aware backends ([ADR 0050](ADRs/0050-distributed-tracing-instrumentation.md)).
+- Eval measurements: the concept of scoring traces ([fail-open](glossary.md#fail-open)). [OTEL primary facts](glossary.md#otel-primary-facts) stay on the run trace (`run-telemetry.jsonl`); [OTEL derived products](glossary.md#otel-derived-products) are the scores (`eval-measurements.jsonl`) ([ADR 0087](ADRs/0087-eval-measurements-online-trace-scoring.md)). See [Eval Measurements](guides/infrastructure/eval-measurements.md).
+
+  > **Planned:** portable remote score export via the same OTLP configuration as agent traces ([ADR 0087](ADRs/0087-eval-measurements-online-trace-scoring.md)). Not yet implemented.
 
 **Open questions:**
 
 - What signals matter most — cost, latency, token usage, action logs, decision traces, or something else?
 - ~~How do we balance detailed tracing (useful for debugging) with the volume of data agents will produce?~~ Decided in [ADR 0050](ADRs/0050-distributed-tracing-instrumentation.md): instrument all lifecycle steps comprehensively; volume is managed by backends not by suppressing data at the source.
+- ~~How do we score wild agent traces for trends without a second export stack?~~ Decided in [ADR 0087](ADRs/0087-eval-measurements-online-trace-scoring.md): eval measurements write local JSONL beside telemetry when at least one new score row is produced (including `label: skip`); portable remote export uses the same OTLP config as traces (planned). The JSONL is absent (not empty) when telemetry/manifest is missing, no traces match, or every candidate is already in the ledger.
 - What is the retention and access model for agent logs? Who can see what? (JSONL trace access model decided in [ADR 0021](ADRs/0021-jsonl-reasoning-trace-exposure.md); retention policy and broader log access remain open.)
 - How does observability interact with the security requirement that "every action is logged, attributable, and reviewable"? (See [security-threat-model.md](problems/security-threat-model.md).)
 - Is there a real-time monitoring requirement (agent is stuck, agent is behaving anomalously), or is observability primarily forensic?

docs/cli/README.md

@@ -27,6 +27,7 @@ Download the latest binary from [GitHub Releases](https://github.com/fullsend-ai
 | `fullsend run` | Execute an agent locally in a sandbox. See [running agents locally](../guides/user/running-agents-locally.md). |
 | `fullsend lock [agent-name]` | Pin remote dependencies to `lock.yaml` |
 | `fullsend scan` | Run security scanners on agent input/output |
+| `fullsend eval-measure` | Score wild-run traces into `eval-measurements.jsonl`. See [Eval measurements](../guides/infrastructure/eval-measurements.md). |
 
 ## Global flags