feat: fixture-based evaluation harness

.github/workflows/eval.yml

@@ -0,0 +1,41 @@
+name: Eval
+
+on:
+  workflow_dispatch:
+    inputs:
+      case:
+        description: "Run a specific case (leave empty for all)"
+        required: false
+        default: ""
+
+permissions:
+  contents: read
+
+jobs:
+  eval:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+      - uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
+      - run: uv python install 3.12
+      - run: uv sync --extra dev
+
+      - name: Run evals
+        env:
+          MODEL_API_BASE: ${{ secrets.MODEL_API_BASE }}
+          MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }}
+          MODEL_NAME: ${{ secrets.MODEL_NAME }}
+        run: |
+          args="--verbose"
+          if [ -n "${{ inputs.case }}" ]; then
+            args="$args --case ${{ inputs.case }}"
+          fi
+          uv run python evals/run.py $args
+
+      - name: Upload results
+        if: always()
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
+        with:
+          name: eval-results
+          path: evals/results/
+          if-no-files-found: ignore

evals/README.md

@@ -0,0 +1,62 @@
+# Evaluation Harness
+
+Fixture-based evaluation for code-to-docs prompt quality. Tests run against
+a real LLM endpoint and check assertions rather than comparing golden text.
+
+## Running
+
+```bash
+# Set endpoint credentials
+export MODEL_API_BASE="https://your-endpoint/v1"
+export MODEL_API_KEY="your-key"
+export MODEL_NAME="your-model"
+
+# Run all cases
+uv run python evals/run.py
+
+# Run a single case
+uv run python evals/run.py --case issue-52-deletion --verbose
+```
+
+## Adding a Case
+
+Create a directory under `evals/fixtures/` with:
+
+```
+evals/fixtures/my-case/
+  diff.patch          # The code diff to analyze
+  before/             # Doc files in their original state
+    docs/guide.md
+    docs/api.md
+  expectations.yaml   # Assertions to check
+  instructions.txt    # (optional) User instructions
+```
+
+### expectations.yaml format
+
+```yaml
+# Files that should be updated (not return NO_UPDATE_NEEDED)
+selected:
+  - docs/guide.md
+
+# Files that should NOT be updated
+not_selected:
+  - docs/unrelated.md
+
+# Content checks on the generated output
+content_checks:
+  docs/guide.md:
+    contains:
+      - "new-flag"
+    not_contains:
+      - "deleted heading"
+    heading_present:
+      - "## Configuration"
+
+# If true, all files should return NO_UPDATE_NEEDED
+expect_no_update: false
+```
+
+Assertions are preferred over golden files because model output varies
+across runs and model versions. Check for structural properties (headings
+present, keywords included, sections preserved) rather than exact text.

evals/fixtures/cli-reference-update/before/docs/cli.md

@@ -0,0 +1,24 @@
+# CLI Reference
+
+## Usage
+
+```
+mytool [options] <input>
+```
+
+## Options
+
+| Flag | Description |
+|------|-------------|
+| `--verbose`, `-v` | Enable verbose output |
+| `--help` | Show help message |
+
+## Examples
+
+```bash
+# Basic usage
+mytool data.csv
+
+# Verbose mode
+mytool -v data.csv
+```

evals/fixtures/cli-reference-update/diff.patch

@@ -0,0 +1,16 @@
+diff --git a/src/cli.py b/src/cli.py
+index 1112222..3334444 100644
+--- a/src/cli.py
++++ b/src/cli.py
+@@ -35,6 +35,12 @@ def build_parser():
+         "--verbose", "-v",
+         action="store_true",
+         help="Enable verbose output",
+     )
++    parser.add_argument(
++        "--output-format",
++        choices=["json", "text", "csv"],
++        default="text",
++        help="Output format for results (default: text)",
++    )
+     return parser

evals/fixtures/cli-reference-update/expectations.yaml

@@ -0,0 +1,12 @@
+selected:
+  - docs/cli.md
+
+content_checks:
+  docs/cli.md:
+    contains:
+      - "--output-format"
+      - "json"
+      - "csv"
+    heading_present:
+      - "# CLI Reference"
+      - "## Options"

evals/fixtures/issue-52-deletion/before/docs/auth-guide.md

@@ -0,0 +1,42 @@
+# Authentication Guide
+
+## Overview
+
+This guide covers the authentication system used by the application.
+
+## Token Validation
+
+Use `AuthManager.validate_token()` to check whether a JWT token is valid:
+
+```python
+auth = AuthManager()
+if auth.validate_token(token):
+    print("Token is valid")
+```
+
+## Rate Limiting
+
+The auth system enforces rate limits on token validation requests.
+By default, each client is limited to 100 validations per minute.
+
+To configure:
+
+```python
+auth = AuthManager(rate_limit=200)
+```
+
+## Error Handling
+
+When validation fails, the system logs the failure reason. Common causes:
+
+- Expired token
+- Invalid signature
+- Malformed payload
+
+## Troubleshooting
+
+If you encounter persistent validation failures, check:
+
+1. Clock synchronization between services
+2. Key rotation schedule
+3. Token issuer configuration

evals/fixtures/issue-52-deletion/diff.patch

@@ -0,0 +1,19 @@
+diff --git a/src/auth.py b/src/auth.py
+index aaa1111..bbb2222 100644
+--- a/src/auth.py
++++ b/src/auth.py
+@@ -45,6 +45,15 @@ class AuthManager:
+     def validate_token(self, token):
+         """Validate a JWT token."""
+         return self._decode(token) is not None
++
++    def refresh_token(self, token):
++        """Refresh an expired token.
++
++        Returns a new token with an extended expiry, or None if the
++        original token is invalid.
++        """
++        payload = self._decode(token, allow_expired=True)
++        if payload:
++            return self._encode(payload)
++        return None

evals/fixtures/issue-52-deletion/expectations.yaml

@@ -0,0 +1,12 @@
+selected:
+  - docs/auth-guide.md
+
+content_checks:
+  docs/auth-guide.md:
+    contains:
+      - "refresh_token"
+    # These sections must survive; the model should not delete them
+    heading_present:
+      - "## Rate Limiting"
+      - "## Error Handling"
+      - "## Troubleshooting"

evals/fixtures/no-update-needed/before/docs/api-reference.md

@@ -0,0 +1,23 @@
+# API Reference
+
+## Authentication
+
+### POST /auth/login
+
+Authenticates a user and returns a JWT token.
+
+**Request body:**
+```json
+{"username": "admin", "password": "secret"}
+```
+
+**Response:**
+```json
+{"token": "eyJ..."}
+```
+
+## Users
+
+### GET /users
+
+Returns a list of all users.

evals/fixtures/no-update-needed/diff.patch

@@ -0,0 +1,14 @@
+diff --git a/src/internal/cache.py b/src/internal/cache.py
+index eee5555..fff6666 100644
+--- a/src/internal/cache.py
++++ b/src/internal/cache.py
+@@ -22,7 +22,7 @@ class LRUCache:
+     def get(self, key):
+         if key in self._store:
+             self._hits += 1
+-            return self._store[key]
++            value = self._store.pop(key)
++            self._store[key] = value  # move to end
++            return value
+         self._misses += 1
+         return None

evals/fixtures/no-update-needed/expectations.yaml

@@ -0,0 +1,3 @@
+# The diff changes an internal cache implementation detail.
+# The API reference doc has nothing to do with it.
+expect_no_update: true

evals/fixtures/short-doc-legitimate-edit/before/docs/config.md

@@ -0,0 +1,9 @@
+# Configuration
+
+## Default Settings
+
+| Key | Default | Description |
+|-----|---------|-------------|
+| `timeout` | `30` | Request timeout in seconds |
+| `retries` | `3` | Number of retry attempts |
+| `log_level` | `INFO` | Logging verbosity |

evals/fixtures/short-doc-legitimate-edit/diff.patch

@@ -0,0 +1,10 @@
+diff --git a/src/config.py b/src/config.py
+index ccc3333..ddd4444 100644
+--- a/src/config.py
++++ b/src/config.py
+@@ -10,6 +10,7 @@ DEFAULTS = {
+     "timeout": 30,
+     "retries": 3,
+     "log_level": "INFO",
++    "max_connections": 50,
+ }

evals/fixtures/short-doc-legitimate-edit/expectations.yaml

@@ -0,0 +1,9 @@
+selected:
+  - docs/config.md
+
+content_checks:
+  docs/config.md:
+    contains:
+      - "max_connections"
+    heading_present:
+      - "# Configuration"