feat: add OpenTelemetry tracing for agent execution

[pavelanni]

Summary

Add OpenTelemetry (OTel) instrumentation to DocsClaw so that agent execution — LLM calls, tool invocations, skill loading, and A2A delegation — produces distributed traces that can be viewed in any OTel-compatible backend (Jaeger, Tempo, Datadog, etc.).

Motivation

Observability is a gap compared to frameworks like Google ADK that provide built-in tracing. Rather than adopting a framework-specific tracing solution, OTel provides a universal standard that works with any backend and any agent framework in a multi-agent system.

This matters for:

• Debugging: understanding why an agent took 30 seconds (was it the LLM call? a tool timeout? skill loading?)
• Multi-agent visibility: tracing A2A delegation chains across agents
• Production readiness: operators need to understand agent behavior without reading logs

Proposed instrumentation points

Component	Span name	Key attributes
pkg/tools/loop.go	agent.loop.iteration	iteration number, tool calls made
internal/anthropic/	llm.complete	model, token count, stop reason
pkg/tools/	tool.execute.<name>	tool name, duration, success/error
pkg/skills/	skill.load	skill name, source (OCI/ConfigMap)
internal/bridge/	a2a.delegate	target agent, message ID

Implementation notes

• Use go.opentelemetry.io/otel SDK — lightweight, no heavy dependencies
• Export via OTLP (gRPC or HTTP) configured through standard OTEL_EXPORTER_* env vars
• Traces should propagate through A2A calls via W3C Trace Context headers
• Keep it optional — if no exporter is configured, tracing is a no-op

Related

• Closes gap identified in feat: explore Google ADK as an alternative agent framework #33 (ADK evaluation)
• Prometheus metrics already exist in internal/metrics/ — traces complement, not replace

[pavelanni]

Addition: Token usage and cost tracking

Beyond distributed tracing, we need per-request token accounting in llm.Response. This feeds into OTel span attributes, Prometheus metrics, context compaction (#5 Phase 3), and the eval harness (#35).

Proposed Usage struct

type Usage struct {
    InputTokens      int
    OutputTokens     int
    CacheReadTokens  int
    CacheWriteTokens int
    TotalTokens      int
}

Add Usage field to llm.Response. Both the Anthropic and OpenAI providers already receive this data in API responses — we just need to propagate it.

Cost calculation (optional, Phase 2)

Model pricing as $/million tokens in a registry. Calculate per-request cost from usage × pricing. This is lower priority than raw token counts but valuable for enterprise cost attribution.

Integration points

• llm.Response.Usage — populated by each provider from API response
• OTel spans: llm.tokens.input, llm.tokens.output, llm.tokens.total as span attributes
• Prometheus: docsclaw_llm_tokens_total{direction="input|output"} counter
• Context compaction (Memory and context persistence #5): uses TotalTokens for accurate window tracking instead of char estimation
• Eval harness (feat: add evaluation harness for agent behavior testing #35): reports token usage per scenario

Identified from pi-mono's Usage type which tracks input/output/cache tokens with cost calculation across all 28+ providers.