pipeline-correctness-fixes: cache-or-LLM audit + 250+ commits backlog

.claude/agents/code-reviewer.md

@@ -7,7 +7,7 @@ model: sonnet
 
 You are a code reviewer specializing in multi-agent LangGraph systems.
 
-Read @docs/rules.md for the full list of constraints and conventions before reviewing.
+Read @docs/rules.md and `.claude/rules/` for the full list of constraints and conventions before reviewing.
 
 Review code for:
 

.claude/agents/db-auditor.md

@@ -1,6 +1,6 @@
 ---
 description: "Scans test files for accidental production database references. Prevents the 2026-03-25 incident."
-tools: Read, Grep, Glob, LS
+tools: Read, Grep, Glob
 disallowedTools: Write, Edit, Bash
 model: haiku
 maxTurns: 10

.claude/agents/memory-debugger.md

@@ -9,7 +9,7 @@ You are a memory system debugger for this multi-agent project.
 
 Read @docs/hooks.md for the 5-tier memory architecture table and tool permission model.
 
-When debugging `shared/memory_layer.py`:
+When debugging `shared/memory_layer/` (package: `_manager.py`, `_stores.py`, `_entries.py`, `_pattern.py`, `_router.py`, `_sqlite_store.py`, `_qdrant_store.py`, `_neo4j_store.py`):
 - Check `MemoryManager.get_context_for_agent()` for injection logic
 - Verify memory is PUSHED to agents (agents should never query memory directly)
 - Check retrieval scoring and relevance filtering

.claude/agents/pattern-explorer.md

@@ -18,5 +18,7 @@ Key patterns in this project:
 - **Peer Debate** (`patterns/peer_debate.py`): Sequential pipeline then cross-critique debate rounds
 - **Dynamic Swarm** (`patterns/dynamic_swarm.py`): Task analyzer + priority queue + executor dispatch
 - **Enhanced Swarm** (`patterns/enhanced_swarm.py`): Dynamic swarm + factory + GRPO + persona evolution
+- **Plan and Execute** (`patterns/plan_and_execute.py`): Upfront planning then sequential step execution
+- **Map Reduce** (`patterns/map_reduce.py`): Parallel map across sub-tasks then reduce/merge results
 
 Always ground explanations in the actual code. Quote specific functions and line numbers.

.claude/agents/reviewer.md

@@ -1,6 +1,6 @@
 ---
 description: "Reviews code changes against project rules, checks dispatch sync, verifies DB safety in tests."
-tools: Read, Grep, Glob, LS
+tools: Read, Grep, Glob
 disallowedTools: Write, Edit, Bash
 model: opus
 maxTurns: 15

.claude/commands/add-intent.md

@@ -10,9 +10,11 @@ When adding a new command/intent, multiple files must be updated in sync. This c
 4. Ask: What agent function will handle it? (existing file or needs new agent)
 
 Then update ALL of these files:
-- `jobpulse/dispatcher.py` — add to AGENT_MAP + correct *_INTENTS set
-- `jobpulse/swarm_dispatcher.py` — add to AGENT_MAP + correct *_INTENTS set
-- `shared/nlp_classifier.py` — add regex patterns + embedding examples for the new intent
+- `jobpulse/handler_registry.py` — add handler to the shared handler map
+- `jobpulse/intent_registry.py` — add intent to the correct intent group
+- `jobpulse/command_router.py` — add to Intent enum + classification logic
+- Verify both `jobpulse/dispatcher.py` AND `jobpulse/swarm_dispatcher.py` pick it up via `get_handler_map()`
+- `jobpulse/nlp_classifier.py` — add embedding examples for the new intent (do NOT add regex — regex tier is legacy)
 - Create test in `tests/` for dispatch routing (both dispatchers) + NLP classification
 
 Finally, run `/check-dispatch` to verify everything is in sync.

.claude/commands/fix-pipeline-bugs.md

@@ -0,0 +1,64 @@
+# /fix-pipeline-bugs — Run one session of the pipeline-bugs fix plan
+
+> **LAUNCH WITH `--dangerously-skip-permissions`** — this command edits files,
+> runs `git commit`, and runs `python -m jobpulse.runner job-process-url`
+> against live ATS pages in dry-run mode. Without that flag every tool call
+> blocks on a permission prompt and the session can't run unattended.
+>
+> ```bash
+> claude --dangerously-skip-permissions
+> # then in the session:
+> /fix-pipeline-bugs
+> ```
+
+## What this does
+
+One invocation = **one session** of the 18-session plan that systematically
+fixes everything in `docs/audits/pipeline-bugs.md`. The runner:
+
+1. **Detects state** by greping `git log` for `fix(pipeline-bugs-S<n>):`
+   commits. Next session = `max(n) + 1`. Starts from S1 if no prior commits.
+2. **Loads the protocol** for that session from `docs/audits/pipeline-bugs-runner.md`.
+3. **Pulls a live ATS URL** from production DBs (`data/applications.db`,
+   `data/form_experience.db`, `data/screening_cache.db`) — only if the
+   session needs one.
+4. **Implements the fix + a regression test** that fails on the bug *pattern*,
+   not just the instance.
+5. **Runs a live `dry_run=True` reproducer** against the real URL when applicable
+   (browser headed via `chrome-pw`, `JOB_AUTOPILOT_AUTO_SUBMIT=false`,
+   `JOBPULSE_FAST_FILL=true`).
+6. **Commits** with `fix(pipeline-bugs-S<n>): ...` and **marks the row** in
+   `docs/audits/pipeline-bugs.md` with `✅ FIXED <commit-hash>`.
+
+## Protocol
+
+Read and follow `docs/audits/pipeline-bugs-runner.md` exactly. That file is
+the durable prompt — this command is the entry point.
+
+After reading it:
+
+1. Run the **state detection** block.
+2. Announce the session number + scope to the user (≤ 3 lines).
+3. Execute steps 1-7 of the per-session protocol in `pipeline-bugs-runner.md`.
+4. End with the **post-session checklist** before declaring done.
+
+## When to stop
+
+A session ends when **any** of these is true:
+
+- ✅ Acceptance criteria met → commit + mark FIXED + report.
+- 🛑 Live reproducer requires user input (Telegram captcha, account creation,
+  paid ATS interaction) → stop, ask user, do **not** commit partial work.
+- 🛑 Fix touches > 2 subsystems → stop, ask user whether to split.
+- 🛑 Wider regression sweep introduces > 1 new failure → stop, revert, advisor.
+
+If you're tempted to "just keep going to the next session" — don't. One
+session per invocation. The state-detection in the next invocation will pick
+up cleanly.
+
+## Reading order on first invocation
+
+1. This file (you are here).
+2. `docs/audits/pipeline-bugs-runner.md` — full session protocol + 18-session table.
+3. `docs/audits/pipeline-bugs.md` — the bug catalog you are draining.
+4. `docs/audits/audit-followup-worklist.md` — per-subsystem source detail.

.claude/mistakes.md

@@ -4,6 +4,13 @@ Append on error. Re-check before committing. Use `semantic_search "mistake <topi
 
 ---
 
+- [2026-04-30] Ollama probe must verify model availability, not just server reachability — empty models list = cloud fallback
+- [2026-04-30] Vision fallback must use a vision-capable cloud model (gpt-4o-mini), not local text-only model
+- [2026-04-30] PageReasoner smart_llm_call() requires (llm, messages) positional args, not keyword-only
+- [2026-04-30] Login pages with embedded CAPTCHA/hCaptcha classified as verification_wall — suppress wall score when login/signup signals coexist
+- [2026-04-30] PageReasoner caches `abort` from failed LLM calls — stale cache blocks retry on same page
+- [2026-04-30] Auth handlers must read actual page content, not follow hardcoded flows — Oracle Cloud email-only page crashed because handler assumed password field exists
+- [2026-04-30] PageReasoner must be PRIMARY decision-maker, not fallback — hardcoded classifier→handler chains break on any page layout the code hasn't seen
 - [2026-04-04] Use MCP tools before Explore agents — saves ~190k tokens per exploration
 - [2026-04-03] Route jobs by `classify_action()` not `determine_match_tier()` — tier is display only
 - [2026-04-01] Regex: specific patterns BEFORE general. Test against all question types
@@ -20,3 +27,31 @@ Append on error. Re-check before committing. Use `semantic_search "mistake <topi
 - [2026-03-24] Never wait for Telegram replies in Claude Code — poll API directly
 - [2026-03-24] Never use Events API for commits — use Commits API per-repo
 - [2026-03-24] Never rewrite a file without grepping for all function names used elsewhere
+
+## DB drop-rate alert — user_profile.screening_defaults (2026-05-10)
+
+DB ``user_profile.screening_defaults`` returned data that was dropped from 7/10 consumed lookups
+(70.0% drop rate over the last 1 day(s)).
+
+Top drop reason: ``option_misalignment`` (7 occurrences).
+
+**OPRAL investigation prompt:**
+- **Observe**: pull a sample of dropped values from
+  ``data/db_observability.db`` where ``db_name='user_profile' AND table_name='screening_defaults' AND status='dropped'``
+- **Plan**: trace which call site produced the lookup and which downstream
+  consumer dropped it. Use ``mcp__code-intelligence__callers_of`` on the
+  accessor.
+- **Reason**: is the data wrong-shape (option_misalignment, validation_failed),
+  or is the consumer buggy?
+- **Act**: fix the source (rewrite stored row, retrain pattern, replace stale
+  default) or the consumer (improve alignment, surface a clearer match).
+- **Learn**: emit ``adaptation`` or ``correction`` signal once fixed; verify
+  drop rate drops on next daily summary.
+
+Sample dropped rows (max 5):
+- field='Are you willing to commute to the office?' intended='Yes, willing to commute to any UK office' actual='' reason='option_misalignment'
+- field='Are you willing to commute to the office?' intended='Yes, willing to commute to any UK office' actual='' reason='option_misalignment'
+- field='Are you willing to relocate to the office in London?' intended='Yes, within the UK' actual='' reason='option_misalignment'
+- field='Are you willing to relocate to the office in London?' intended='Yes, within the UK' actual='' reason='option_misalignment'
+- field='Are you willing to commute to the office?' intended='Yes, willing to commute to any UK office' actual='' reason='option_misalignment'
+

.claude/rules/error-handling.md

@@ -1,5 +1,5 @@
 ---
-paths: ["jobpulse/**/*.py", "shared/**/*.py"]
+paths: ["jobpulse/**/*.py", "shared/**/*.py", "patterns/**/*.py", "mindgraph_app/**/*.py"]
 description: "Structured error handling for all agent code"
 ---
 
@@ -30,3 +30,6 @@ All errors in agent code MUST return structured error context, not generic strin
 - Distinguish access failures (needing retry) from valid empty results (successful query, no matches)
 - Only validate at system boundaries (user input, external APIs) — don't add error handling for impossible internal scenarios
 - Trust internal code and framework guarantees — no defensive coding against things that can't happen
+
+## OPRAL Error Loop
+On every error: follow the OPRAL loop (Observe → Plan → Reason → Act → Learn). See CLAUDE.md.

.claude/rules/jobpulse.md

@@ -3,13 +3,17 @@
 ## Claude = Orchestrator, Not Doer (MANDATORY)
 When running the pipeline, invoke the actual AI agents — don't bypass them with ad-hoc scripts. Agents only learn from runs they performed. Observe, diagnose, direct. Manual scripts only as fallback for issues agents can't handle yet — then feed corrections into agent DBs so it's autonomous next time.
 
+## OPRAL Error Loop (MANDATORY)
+On every error/issue: **Observe** (capture error + context) → **Plan** (trace root cause via MCP) → **Reason** (why did this fail? which learning system prevents recurrence?) → **Act** (fix surgically, re-run with real data, route to correct DB) → **Learn** (emit signal, verify DB persisted, confirm agent handles it autonomously next run). Every error must make the system smarter — if it can recur, the fix is incomplete.
+
 ## Dual Dispatcher Invariant
 When adding a new intent or agent:
-1. Add handler to jobpulse/dispatcher.py AGENT_MAP
-2. Add SAME handler to jobpulse/swarm_dispatcher.py AGENT_MAP
-3. Add intent string to the correct *_INTENTS set in both files
-4. Add NLP examples to shared/nlp_classifier.py
-5. Add tests for the new intent in both dispatch paths
+1. Add handler to jobpulse/handler_registry.py handler map
+2. Add intent to jobpulse/intent_registry.py in the correct intent group
+3. Add routing in jobpulse/command_router.py (Intent enum + classification)
+4. Verify both jobpulse/dispatcher.py AND jobpulse/swarm_dispatcher.py pick it up via get_handler_map()
+5. Add NLP examples to jobpulse/nlp_classifier.py (embedding examples, NOT regex)
+6. Add tests for the new intent in both dispatch paths
 NEVER update only one dispatcher. This has caused production bugs (see mistakes.md 2026-03-30).
 
 ## Telegram Message Handling
@@ -67,6 +71,11 @@ Regex MUST NOT be used for semantic work: intent routing, question categorizatio
 ## Real Data + Wiring Verification (MANDATORY)
 Every new agent/feature: test with real URLs, real APIs, real DB queries (never mocks or stale fixtures). Then verify end-to-end that all downstream systems fire — hooks, signals, DB writes, learning chains, Notion syncs, Telegram notifications. Not wired = not done.
 
+## API Pitfalls (from incidents)
+- GitHub: use Commits API per-repo, never Events API (Events API misses commits and has aggressive rate limits)
+- GitHub: `pushed_at` timestamp filtering uses `>=` or `<`, never `==` (timestamps have sub-second precision)
+- ATS numeric fields: plain integers only — no currency symbols, commas, ranges, or units
+
 ## Database Safety
 - Production DBs live in data/*.db — tests MUST NEVER touch these
 - All test fixtures use tmp_path or monkeypatch DB_PATH