feat(traces): record model, reasoning effort, and token usage per session event

README.md

@@ -523,6 +523,8 @@ For VFS-capable runtimes (claude-code/codex) the `hivemind-goals` skill creates
 
 Per-agent integration mechanisms (marketplace plugin, hooks, skills, native extension) and monorepo structure: **[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md)**.
 
+What each trace row records — model, reasoning effort, stop reason, token usage (incl. cache tokens and cost) and per-harness extras, with the capture matrix per agent: **[harnesses/TRACE_MODEL_USAGE.md](harnesses/TRACE_MODEL_USAGE.md)**.
+
 ## Roadmap
 
 - **Trajectory export for fine-tuning.** Because traces are stored in Deeplake's tensor format, they're export-ready as PyTorch datasets. Teams running their own open-source models can fine-tune on their org's accumulated trajectories. A handful of advanced customers are already doing this against the trajectories their Claude Code and Codex agents generated.

harnesses/TRACE_MODEL_USAGE.md

@@ -0,0 +1,143 @@
+# Trace model / token-usage enrichment
+
+Every captured session event (user prompt, tool call, assistant turn) is written
+as one row in the `sessions` table, with the event payload in the JSONB `message`
+column. On top of the base event, each row is enriched — where the agent exposes
+it — with **which model** produced the turn, its **reasoning effort**, why the
+turn **stopped**, full **token usage** (including cache tokens and **money
+cost**), and a per-harness **`usage_extra`** bag for fields that don't generalize.
+
+This lets "tokens/cost per model" be a plain `GROUP BY message->>'model'` query.
+
+## Where the code lives
+
+- Extraction / normalization: [`src/notifications/model-usage.ts`](../src/notifications/model-usage.ts)
+  - `parseClaudeTurnMeta(transcriptPath)` — Claude Code
+  - `parseCodexTurnMeta(transcriptPath, fallbackModel)` — Codex
+  - `normalizeSdkUsage` / `sdkTurnMeta` — Pi / OpenClaw (in-process SDK message)
+  - `readClaudeEffortLevel(cwd)` — Claude effort from settings
+- Wiring (per harness):
+  - Claude Code — `src/hooks/capture.ts` (assistant_message / Stop)
+  - Codex — `src/hooks/codex/capture.ts`
+  - Pi — `harnesses/pi/extension-source/hivemind.ts` (`message_end`; inlines the
+    normalizer because pi ships raw `.ts` with no shared-module imports)
+  - OpenClaw — `harnesses/openclaw/src/index.ts` (`agent_end` auto-capture)
+  - Hermes — `src/hooks/hermes/capture.ts` (reads `extra.model` / `extra.platform`)
+
+## Stored shape (JSONB `message`)
+
+```jsonc
+{
+  // ...base event fields (session_id, type, content, timestamp, ...)
+  "model": "claude-opus-4-8",
+  "reasoning_effort": "medium",        // null when unset / not applicable
+  "stop_reason": "end_turn",           // end_turn | tool_use | max_tokens | stop | toolUse | ...
+  "token_usage": {                     // the most recent turn
+    "input_tokens": 131,
+    "output_tokens": 403,
+    "cache_read_tokens": 357059,
+    "cache_creation_tokens": 2102,
+    "reasoning_output_tokens": 160,    // Codex
+    "total_tokens": 216408,            // Codex / SDK
+    "cost": {                          // Pi / OpenClaw — fractional dollars
+      "input": 0.165, "output": 0.0001, "cache_read": 0, "cache_creation": 0, "total": 0.1651
+    }
+  },
+  "token_usage_total": { /* ...same shape */ },  // Codex only — whole-session cumulative
+  "usage_extra": { /* per-harness, see below */ }
+}
+```
+
+All fields are **optional** and only present when the agent exposes them (absent
+keys stay absent — never defaulted to 0/empty).
+
+## Capture matrix (verified against real transcripts)
+
+| Field | Claude Code | Codex | Pi | OpenClaw | Cursor | Hermes |
+|---|:--:|:--:|:--:|:--:|:--:|:--:|
+| `model` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
+| `reasoning_effort` | ✅ (settings) | ✅ | — | — | — | — |
+| `stop_reason` | ✅ | — ¹ | ✅ | ✅ | — | — |
+| token: input / output | ✅ | ✅ | ✅ | ✅ | — | — |
+| token: cache_read | ✅ | ✅ | ✅ | ✅ | — | — |
+| token: cache_creation | ✅ | — ² | ✅ | ✅ | — | — |
+| token: reasoning_output | — | ✅ | — | — | — | — |
+| token: total | — ³ | ✅ | ✅ | ✅ | — | — |
+| `token_usage.cost` ($) | — | — | ✅ | ✅ | — | — |
+| `token_usage_total` (cumulative) | — | ✅ | — | — | — | — |
+| `usage_extra` | tier / speed / cache-TTL / server_tool_use | model_context_window / rate_limits | — | — | — | platform |
+| custom fields | — | — | — | — | duration / loop_count / status | — |
+
+**Notes.** ¹ Codex emits only per-turn `status:"completed"`, not a stop reason.
+² Codex has no cache-write concept. ³ Claude doesn't report a grand total in
+`usage`; it isn't synthesized. Cursor's transcript has **no** token/cost data;
+Hermes' hook payload carries **no** token/cost (only `model` + `platform`).
+
+### Source of each field, per harness
+
+| Harness | Source |
+|---|---|
+| Claude Code | transcript `~/.claude/projects/.../*.jsonl` (last assistant line: `message.model`, `message.usage`, `message.stop_reason`); effort from `settings.json` `effortLevel` |
+| Codex | rollout `~/.codex/sessions/.../rollout-*.jsonl` (`turn_context` → model/effort; `token_count` → usage/total/quota) |
+| Pi | in-process `message_end` SDK message (`model`, `stopReason`, `usage` incl. `cost`) |
+| OpenClaw | in-process `agent_end` SDK message (`model`, `stopReason`, `usage` incl. `cost`) |
+| Cursor | hook payload `model` (+ its own `duration`/`loop_count`/`status`) |
+| Hermes | hook payload `extra.model` / `extra.platform` (NousResearch/hermes-agent `shell_hooks.py`) |
+
+## Aggregation
+
+```sql
+-- Per-model output tokens (Claude / any agent that reports per-turn usage)
+SELECT message->>'model' AS model,
+       SUM((message->'token_usage'->>'output_tokens')::bigint) AS out_tokens
+FROM sessions
+WHERE agent = 'claude_code'
+GROUP BY 1;
+
+-- Per-model cost (Pi / OpenClaw)
+SELECT message->>'model' AS model,
+       SUM((message->'token_usage'->'cost'->>'total')::numeric) AS usd
+FROM sessions
+WHERE message->'token_usage'->'cost' IS NOT NULL
+GROUP BY 1;
+```
+
+`token_usage_total` (Codex) is a **whole-session cumulative across all models**,
+not per-model — roll it up as `MAX(total_tokens)` per session, not `SUM`. For
+per-model Codex totals, sum the per-turn `token_usage` and dedup by `turn_id`
+(a turn's tool events share one snapshot).
+
+## Design notes
+
+- **Best-effort.** Any read/parse failure returns null and capture proceeds
+  unchanged — enrichment never blocks or breaks a trace write.
+- **Tail read (perf).** The parsers read only the **file tail** (default 1 MiB)
+  via a single `openTranscript()` — model/usage live in the most recent turn at
+  the end of the file — so each event is ~O(1), not O(filesize) (and not O(n²)
+  across a Codex session that parses on every user/tool row). If a single record
+  exceeds the window, a whole-file fallback runs from the **same fd/snapshot**.
+  See the file's `readWindowLines` / `openTranscript` for boundary + short-read
+  handling. Residual TOCTOU (transcript rewritten between the two reads) is
+  unreachable for append-only session logs.
+- **Redaction guard.** Storing model ids exposed a false positive: the secret
+  entropy backstop shredded long dated slugs like `claude-haiku-4-5-20251001`.
+  `src/hooks/shared/redact.ts` `looksLikeSecret` exempts provider model ids
+  (lowercase, segments ≤12 chars, incl. cloud-prefixed Bedrock slugs) while a
+  high-entropy secret wearing a model prefix is still masked.
+- **Reasoning effort for Claude** is the user's `effortLevel` setting
+  (low/medium/high/xhigh), not a per-message field — read from settings at
+  capture time; allowlisted so an arbitrary settings value can't poison analytics.
+- **Numeric guards.** Token counts must be non-negative safe integers; cost is a
+  non-negative finite number (fractional). Invalid values are dropped.
+
+## Adding a new harness
+
+If a runtime's transcript/payload carries model/usage, wire it the same way:
+- Payload has `model` → set it on the trace meta directly.
+- Transcript file with usage → add a `parse<Agent>TurnMeta`.
+- In-process SDK message with `{model, usage, stopReason}` → reuse `sdkTurnMeta`.
+- Anything that doesn't generalize → put it under `usage_extra`, don't fabricate.
+
+Tests: `tests/shared/notifications-model-usage.test.ts` (extraction),
+`tests/shared/redact.test.ts` (model-id guard), plus per-harness capture tests.
+E2E harness: `scripts/trace-model-usage-e2e.mjs`.

harnesses/openclaw/src/index.ts

@@ -67,6 +67,7 @@ import { readVirtualPathContent } from "../../../src/hooks/virtual-table-query.j
 import { tryEmbedStandalone, _setSpawnImpl } from "../../../src/embeddings/standalone-embed-client.js";
 import { embeddingSqlLiteral } from "../../../src/embeddings/sql.js";
 import { redactSecrets } from "../../../src/hooks/shared/redact.js";
+import { sdkTurnMeta } from "../../../src/notifications/model-usage.js";
 // Resolve sibling skillify-worker.js path at runtime via import.meta.url. The
 // openclaw plugin is bundled to harnesses/openclaw/dist/index.js, then installed to
 // ~/.openclaw/extensions/hivemind/dist/index.js by install-openclaw.ts. The
@@ -1482,7 +1483,7 @@ export default definePluginEntry({
     // Auto-capture: store new messages in sessions table (same format as CC capture.ts)
     if (config.autoCapture !== false) {
       hook("agent_end", async (event) => {
-        const ev = event as { success?: boolean; session_id?: string; channel?: string; messages?: Array<{ role: string; content: string | Array<{ type: string; text?: string }> }> };
+        const ev = event as { success?: boolean; session_id?: string; channel?: string; messages?: Array<{ role: string; content: string | Array<{ type: string; text?: string }>; model?: unknown; usage?: unknown; stopReason?: unknown }> };
         if (!captureEnabled || !ev.success || !ev.messages?.length) return;
         try {
           const dl = await getApi();
@@ -1515,12 +1516,16 @@ export default definePluginEntry({
             if (!text.trim()) continue;
 
             const ts = new Date().toISOString();
+            // Tag assistant rows with model + token usage from the SDK message
+            // (openclaw exposes both on the message object; no reasoning effort).
+            const modelMeta = msg.role === "assistant" ? sdkTurnMeta(msg.model, msg.usage, msg.stopReason) : undefined;
             const entry = {
               id: crypto.randomUUID(),
               type: msg.role === "user" ? "user_message" : "assistant_message",
               session_id: sid,
               content: text,
               timestamp: ts,
+              ...(modelMeta ?? {}),
             };
             // Mask secrets before the payload is embedded or stored.
             const line = redactSecrets(JSON.stringify(entry));

harnesses/pi/extension-source/hivemind.ts

@@ -149,6 +149,44 @@ function applyDirConfig(creds: Creds, cwd: string): { creds: Creds; collect: boo
   return { creds: { ...withEnv, orgId, orgName, workspaceId }, collect: true, routed };
 }
 
+// Inline copy of normalizeSdkUsage / sdkTurnMeta (shared version lives in
+// src/notifications/model-usage.ts, but pi extensions ship as raw .ts with no
+// shared-module imports — kept in lockstep with that file). Maps the pi/SDK
+// usage object {input, output, cacheRead, cacheWrite, totalTokens} onto the
+// normalized token_usage keys used across every agent's trace rows.
+function piModelMeta(message: any): { model?: string; stop_reason?: string; token_usage?: Record<string, unknown> } {
+  const out: { model?: string; stop_reason?: string; token_usage?: Record<string, unknown> } = {};
+  if (typeof message?.model === "string" && message.model) out.model = message.model;
+  if (typeof message?.stopReason === "string" && message.stopReason) out.stop_reason = message.stopReason;
+  const u = message?.usage;
+  if (u && typeof u === "object") {
+    const t: Record<string, unknown> = {};
+    const putInt = (k: string, v: unknown) => {
+      if (typeof v === "number" && Number.isSafeInteger(v) && v >= 0) t[k] = v;
+    };
+    putInt("input_tokens", u.input);
+    putInt("output_tokens", u.output);
+    putInt("cache_read_tokens", u.cacheRead);
+    putInt("cache_creation_tokens", u.cacheWrite);
+    putInt("total_tokens", u.totalTokens);
+    if (u.cost && typeof u.cost === "object") {
+      const cost: Record<string, number> = {};
+      putFloatInto(cost, "input", u.cost.input);
+      putFloatInto(cost, "output", u.cost.output);
+      putFloatInto(cost, "cache_read", u.cost.cacheRead);
+      putFloatInto(cost, "cache_creation", u.cost.cacheWrite);
+      putFloatInto(cost, "total", u.cost.total);
+      if (Object.keys(cost).length > 0) t.cost = cost;
+    }
+    if (Object.keys(t).length > 0) out.token_usage = t;
+  }
+  return out;
+}
+
+function putFloatInto(target: Record<string, number>, key: string, v: unknown): void {
+  if (typeof v === "number" && Number.isFinite(v) && v >= 0) target[key] = v;
+}
+
 // Inline copies of decodeJwtPayload + healDriftedOrgToken (the shared helpers
 // live in src/commands/auth.ts, but pi extensions ship as raw .ts with no
 // shared-module imports — kept in lockstep with that file).
@@ -1531,6 +1569,7 @@ export default function hivemindExtension(pi: ExtensionAPI): void {
         session_id: sessionId,
         content: text,
         timestamp: new Date().toISOString(),
+        ...piModelMeta(message),
       });
     } catch (e: any) {
       logHm(`message_end: writeSessionRow swallowed: ${e?.message ?? e}`);