Feat/eval metrics

.github/workflows/ci.yml

@@ -0,0 +1,37 @@
+name: CI
+
+on:
+  pull_request:
+    branches: [main]
+  push:
+    branches: [main]
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ci-${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  quality:
+    name: Tests and lint
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+          cache: pip
+          cache-dependency-path: pyproject.toml
+
+      - name: Install project
+        run: python -m pip install -e ".[dev]"
+
+      - name: Run quality checks
+        run: make check

.gitignore

@@ -16,6 +16,9 @@ data/*
 !data/corpus/
 !data/corpus/**
 
+# Generated eval reports
+eval/results/
+
 # OS / editor
 .DS_Store
 .idea/

Makefile

@@ -1,4 +1,4 @@
-.PHONY: db-up db-down db-reset install fetch-corpus ingest eval test fmt
+.PHONY: db-up db-down db-reset install fetch-corpus ingest eval eval-retrieval test fmt check
 
 db-up:      ## start Postgres+pgvector
 	docker compose up -d
@@ -27,8 +27,16 @@ ingest:     ## build the index from ./data/corpus
 eval:       ## run the evaluation harness
 	PYTHONPATH=src python -m eval.evaluate
 
+eval-retrieval: ## fast retrieval-only eval (no chat/judge calls)
+	PYTHONPATH=src python -m eval.evaluate --retrieval-only
+
 test:       ## run unit tests
 	pytest
 
 fmt:        ## format & lint
 	ruff format . && ruff check --fix .
+
+check:      ## run the same quality checks as CI
+	ruff format --check .
+	ruff check .
+	pytest

README.md

@@ -46,6 +46,7 @@ make install
 make fetch-corpus              # optional refresh; committed corpus works offline
 make ingest                    # upsert chunks from ./data/corpus
 make eval                      # 20 grounded questions
+make eval-retrieval            # fast: retrieval metrics only, no chat calls
 ```
 
 Schema changes require a fresh volume: `make db-reset`.
@@ -72,17 +73,48 @@ Postgres + pgvector (HNSW) + generated `tsvector` for hybrid retrieval, Ollama
 
 ## Results
 
-| Configuration | Retrieval | Judge quality |
-| ------------- | --------- | ------------- |
-| hybrid + llama3.2, loose prompt | 100% (20/20) | 0.65 |
-| hybrid + qwen2.5, grounded / low-temp prompt | 100% (20/20) | 0.67 |
+Retrieval (`hybrid`, `nomic-embed-text`, top-5):
+
+| Hit rate@5 | Precision@1 | MRR | nDCG@5 |
+| ---------- | ----------- | --- | ------ |
+| 100% | 80% | 0.900 | 0.926 |
+
+Generation:
+
+| Configuration | Judge quality |
+| ------------- | ------------- |
+| llama3.2, loose prompt | 0.65 |
+| qwen2.5:7b, grounded / low-temp prompt | 0.67 |
 
 Retrieval was already solid. Tightening the prompt and swapping chat models
 barely moved the LLM-as-judge score — which is a bit expected when the same
 local stack is grading itself. Answers *look* cleaner by eye; the judge just
 doesn’t capture that well. Eval also tracks citation precision / citation
 source hits now so we’re not leaning only on the vibe score.
 
+### Eval metrics and artifacts
+
+The retrieval report is source-level: repeated chunks from one document count as
+one ranked source. It prints:
+
+- **Hit rate@k** — whether any expected source appeared in the top-k.
+- **Precision@1** — whether the first source was relevant.
+- **MRR** — rewards putting the first relevant source near the top.
+- **nDCG@k** — rewards relevant sources appearing higher in the ranking.
+
+Each run writes a full JSON report and a flat per-question CSV under
+`eval/results/` (gitignored). The report includes the model, retrieval mode,
+chunking settings, top-k, code revision, question-set hash, summary metrics,
+and per-question rankings.
+
+```bash
+make eval-retrieval                         # quick retrieval iteration
+make eval                                   # full generation + judge run
+python -m eval.evaluate --k 10              # try another top-k
+python -m eval.evaluate --no-artifacts      # print only
+python -m eval.evaluate --output-dir /tmp/eval-runs
+```
+
 ## What I'd improve next
 
 - Metadata filters (category) at query time