feat: add evaluation harness for agent behavior testing

[pavelanni]

Summary

Add a test harness that validates agent behavior through repeatable scenarios — not just code correctness (unit tests) but whether the agent produces useful, accurate output for given inputs.

Motivation

Agent quality depends on prompt engineering, skill design, and tool orchestration — none of which are covered by go test. Frameworks like Google ADK emphasize evaluation as a first-class concern. DocsClaw should have its own eval framework without depending on external frameworks.

Proposed design

An eval scenario is a YAML file describing input, expected behavior, and pass criteria:

name: executive-summary
description: Agent should produce a structured summary from a report
config-dir: testdata/executive-assistant
input: |
  Summarize this Q1 report: Revenue $4.2M, up 12%.
  Key driver: Enterprise segment grew 25%.
  Risk: Two large renewals pending in Q2.
expect:
  contains:
    - "revenue"
    - "enterprise"
    - "risk"
  max_length: 500
  has_sections:
    - "Summary"
    - "Key decisions"

A docsclaw eval subcommand runs scenarios against a live or mocked LLM:

docsclaw eval scenarios/       # run all scenarios
docsclaw eval scenarios/hr/    # run HR-specific scenarios

Key features

• Deterministic mode: fixed seed / temperature 0 for reproducibility
• Golden file comparison: save a known-good output, diff against it
• Scoring rubric: LLM-as-judge for subjective quality (optional)
• CI integration: exit code reflects pass/fail for pipeline gating
• Cost tracking: report token usage per scenario

Related

• Closes gap identified in feat: explore Google ADK as an alternative agent framework #33 (ADK evaluation)
• Scenarios in docs/demo/ could become the first eval cases

[pavelanni]

Note: Mock LLM provider for deterministic testing

Pi-mono includes a "faux provider" — a mock LLM that returns scripted responses with simulated token usage. This pattern would directly support the eval harness:

// pkg/llm/faux/faux.go
type FauxProvider struct {
    Responses []Response // scripted responses returned in order
}

func (f *FauxProvider) CompleteWithTools(ctx context.Context, messages []Message, 
    tools []ToolDefinition) (*Response, error) {
    // Return next scripted response, simulate tool calls if configured
}

Benefits:

• Deterministic eval scenarios without API calls or costs
• Unit tests for the agentic loop (pkg/tools/loop.go) without mocking HTTP
• CI-friendly — no API keys needed for behavior tests

This could be implemented as a third provider registered via init(), following the existing Anthropic/OpenAI pattern.

From pi-mono's packages/ai/src/providers/faux.ts.