Add pipeline verification, freshness tracking, and confidence tiers

[JMRussas]

Summary

• Verify command: Index health checks (schema, sources, embeddings, FTS5) + configurable search quality tests from config.json
• Stale detection: Two-level check (repo git commits + file SHA-256 hashes), now also checks upstream for unpulled commits and detects embedding model drift
• Freshness dashboard: New pipeline.py freshness command — unified report showing index age, embed model consistency, per-source chunk counts, and aggregated issues
• Confidence tiers: Search results classified as HIGH/MEDIUM/LOW based on configurable thresholds, with optional low-confidence filtering
• Gotcha annotations: Anti-hallucination warnings displayed as CAUTION on search results
• Provenance metadata: Stores embed_model, embed_dimensions, indexed_at, and per-repo git commits in index_metadata table
• Database sources: db_sources config for indexing from external SQLite databases via SQL queries
• Hybrid search: BM25 + vector fusion via Reciprocal Rank Fusion (RRF)
• License: Switched to AGPL v3

Test plan

• python -m pytest tests/ -q — 94 tests pass
• Run python pipeline.py freshness on a RAG with a built index to verify dashboard output
• Run python pipeline.py stale to verify upstream check and model drift detection
• Rebuild a RAG and confirm embed_model/embed_dimensions appear in index_metadata

Generated by Claude Code · Claude Opus 4.6

[JMRussas]

Review fixes (3a9fbd6):

1. Fixed ConnectError break bug — when Ollama is unreachable, semantic search tests now set an ollama_available flag and skip remaining search queries. Lookup tests (pure SQLite) continue running instead of being silently skipped.

2. Renamed shadowed "rows" variable to "lookup_rows" in the lookup test block for clarity.

Note: This PR still bundles hybrid search/reranker changes with the verify command. Since this repo uses squash merge, the final commit will be clean, but future PRs should keep features separate.

[JMRussas]

Self-review defect fixes (13dcc33):

3 issues found and fixed:

1. cmd_freshness KeyError (pipeline.py:1161) — config["ollama"] accessed directly instead of config.get("ollama", {}). Would crash on minimal configs. Fixed to match cmd_stale's pattern.

2. Dead stale_source_tags variable (pipeline.py:996-1030) — Set populated but never read in cmd_stale. Removed.

3. Confidence tiers broken for hybrid search (server.py:383-400) — RRF scores (~0.008–0.03) were compared against cosine similarity thresholds (0.85/0.65), making every hybrid result "LOW" confidence. Added _rrf_confidence_thresholds() that computes rank-calibrated cutoffs: top-3 = HIGH, top-10 = MEDIUM. The search function now selects thresholds based on whether hybrid/reranker is active.

[JMRussas]

Review fix round 2 (ceff8f7):

4 remaining issues fixed:

1. sentence-transformers removed from requirements.txt — optional dep (~500MB PyTorch/ONNX) only needed when reranker.enabled=true. Already lazy-imported in reranker.py. Now documented as install-on-demand comment.

2. Upstream detection made robust — was checking git fetch --dry-run stderr for any output (false positives from progress messages). Now does a proper comparison: fetches remote refs, then compares local HEAD vs @{u} (upstream tracking ref).

3. Reranker confidence thresholds fixed — reranker assigns 1/(rank+1) scores (1.0, 0.5, 0.33...) which is a completely different range than RRF 1/(k+rank+1). Was using RRF thresholds for both. Now: reranker uses 0.25/0.08 (top-3/top-10), RRF uses rank-calibrated, cosine uses config thresholds.

4. cmd_gotcha no longer reads sys.argv directly — now receives chunk_id and gotcha_text as function parameters, matching the pattern of all other cmd_ functions. Argument parsing stays in main().