diff --git a/.claude/agents/code-reviewer.md b/.claude/agents/code-reviewer.md index c3ea8111..b5c0db9b 100644 --- a/.claude/agents/code-reviewer.md +++ b/.claude/agents/code-reviewer.md @@ -7,7 +7,7 @@ model: sonnet You are a code reviewer specializing in multi-agent LangGraph systems. -Read @docs/rules.md for the full list of constraints and conventions before reviewing. +Read @docs/rules.md and `.claude/rules/` for the full list of constraints and conventions before reviewing. Review code for: diff --git a/.claude/agents/db-auditor.md b/.claude/agents/db-auditor.md index 0393fe02..07c475e6 100644 --- a/.claude/agents/db-auditor.md +++ b/.claude/agents/db-auditor.md @@ -1,6 +1,6 @@ --- description: "Scans test files for accidental production database references. Prevents the 2026-03-25 incident." -tools: Read, Grep, Glob, LS +tools: Read, Grep, Glob disallowedTools: Write, Edit, Bash model: haiku maxTurns: 10 diff --git a/.claude/agents/memory-debugger.md b/.claude/agents/memory-debugger.md index 62a4288b..441e0ec3 100644 --- a/.claude/agents/memory-debugger.md +++ b/.claude/agents/memory-debugger.md @@ -9,7 +9,7 @@ You are a memory system debugger for this multi-agent project. Read @docs/hooks.md for the 5-tier memory architecture table and tool permission model. -When debugging `shared/memory_layer.py`: +When debugging `shared/memory_layer/` (package: `_manager.py`, `_stores.py`, `_entries.py`, `_pattern.py`, `_router.py`, `_sqlite_store.py`, `_qdrant_store.py`, `_neo4j_store.py`): - Check `MemoryManager.get_context_for_agent()` for injection logic - Verify memory is PUSHED to agents (agents should never query memory directly) - Check retrieval scoring and relevance filtering diff --git a/.claude/agents/pattern-explorer.md b/.claude/agents/pattern-explorer.md index d05b6d32..9d9cccd2 100644 --- a/.claude/agents/pattern-explorer.md +++ b/.claude/agents/pattern-explorer.md @@ -18,5 +18,7 @@ Key patterns in this project: - **Peer Debate** (`patterns/peer_debate.py`): Sequential pipeline then cross-critique debate rounds - **Dynamic Swarm** (`patterns/dynamic_swarm.py`): Task analyzer + priority queue + executor dispatch - **Enhanced Swarm** (`patterns/enhanced_swarm.py`): Dynamic swarm + factory + GRPO + persona evolution +- **Plan and Execute** (`patterns/plan_and_execute.py`): Upfront planning then sequential step execution +- **Map Reduce** (`patterns/map_reduce.py`): Parallel map across sub-tasks then reduce/merge results Always ground explanations in the actual code. Quote specific functions and line numbers. diff --git a/.claude/agents/reviewer.md b/.claude/agents/reviewer.md index 3b45f2e8..4f7a2140 100644 --- a/.claude/agents/reviewer.md +++ b/.claude/agents/reviewer.md @@ -1,6 +1,6 @@ --- description: "Reviews code changes against project rules, checks dispatch sync, verifies DB safety in tests." -tools: Read, Grep, Glob, LS +tools: Read, Grep, Glob disallowedTools: Write, Edit, Bash model: opus maxTurns: 15 diff --git a/.claude/commands/add-intent.md b/.claude/commands/add-intent.md index 2b70be06..634a6e5e 100644 --- a/.claude/commands/add-intent.md +++ b/.claude/commands/add-intent.md @@ -10,9 +10,11 @@ When adding a new command/intent, multiple files must be updated in sync. This c 4. Ask: What agent function will handle it? (existing file or needs new agent) Then update ALL of these files: -- `jobpulse/dispatcher.py` — add to AGENT_MAP + correct *_INTENTS set -- `jobpulse/swarm_dispatcher.py` — add to AGENT_MAP + correct *_INTENTS set -- `shared/nlp_classifier.py` — add regex patterns + embedding examples for the new intent +- `jobpulse/handler_registry.py` — add handler to the shared handler map +- `jobpulse/intent_registry.py` — add intent to the correct intent group +- `jobpulse/command_router.py` — add to Intent enum + classification logic +- Verify both `jobpulse/dispatcher.py` AND `jobpulse/swarm_dispatcher.py` pick it up via `get_handler_map()` +- `jobpulse/nlp_classifier.py` — add embedding examples for the new intent (do NOT add regex — regex tier is legacy) - Create test in `tests/` for dispatch routing (both dispatchers) + NLP classification Finally, run `/check-dispatch` to verify everything is in sync. diff --git a/.claude/commands/fix-pipeline-bugs.md b/.claude/commands/fix-pipeline-bugs.md new file mode 100644 index 00000000..44d79dd7 --- /dev/null +++ b/.claude/commands/fix-pipeline-bugs.md @@ -0,0 +1,64 @@ +# /fix-pipeline-bugs — Run one session of the pipeline-bugs fix plan + +> **LAUNCH WITH `--dangerously-skip-permissions`** — this command edits files, +> runs `git commit`, and runs `python -m jobpulse.runner job-process-url` +> against live ATS pages in dry-run mode. Without that flag every tool call +> blocks on a permission prompt and the session can't run unattended. +> +> ```bash +> claude --dangerously-skip-permissions +> # then in the session: +> /fix-pipeline-bugs +> ``` + +## What this does + +One invocation = **one session** of the 18-session plan that systematically +fixes everything in `docs/audits/pipeline-bugs.md`. The runner: + +1. **Detects state** by greping `git log` for `fix(pipeline-bugs-S):` + commits. Next session = `max(n) + 1`. Starts from S1 if no prior commits. +2. **Loads the protocol** for that session from `docs/audits/pipeline-bugs-runner.md`. +3. **Pulls a live ATS URL** from production DBs (`data/applications.db`, + `data/form_experience.db`, `data/screening_cache.db`) — only if the + session needs one. +4. **Implements the fix + a regression test** that fails on the bug *pattern*, + not just the instance. +5. **Runs a live `dry_run=True` reproducer** against the real URL when applicable + (browser headed via `chrome-pw`, `JOB_AUTOPILOT_AUTO_SUBMIT=false`, + `JOBPULSE_FAST_FILL=true`). +6. **Commits** with `fix(pipeline-bugs-S): ...` and **marks the row** in + `docs/audits/pipeline-bugs.md` with `✅ FIXED `. + +## Protocol + +Read and follow `docs/audits/pipeline-bugs-runner.md` exactly. That file is +the durable prompt — this command is the entry point. + +After reading it: + +1. Run the **state detection** block. +2. Announce the session number + scope to the user (≤ 3 lines). +3. Execute steps 1-7 of the per-session protocol in `pipeline-bugs-runner.md`. +4. End with the **post-session checklist** before declaring done. + +## When to stop + +A session ends when **any** of these is true: + +- ✅ Acceptance criteria met → commit + mark FIXED + report. +- 🛑 Live reproducer requires user input (Telegram captcha, account creation, + paid ATS interaction) → stop, ask user, do **not** commit partial work. +- 🛑 Fix touches > 2 subsystems → stop, ask user whether to split. +- 🛑 Wider regression sweep introduces > 1 new failure → stop, revert, advisor. + +If you're tempted to "just keep going to the next session" — don't. One +session per invocation. The state-detection in the next invocation will pick +up cleanly. + +## Reading order on first invocation + +1. This file (you are here). +2. `docs/audits/pipeline-bugs-runner.md` — full session protocol + 18-session table. +3. `docs/audits/pipeline-bugs.md` — the bug catalog you are draining. +4. `docs/audits/audit-followup-worklist.md` — per-subsystem source detail. diff --git a/.claude/mistakes.md b/.claude/mistakes.md index 5b594f3d..ae1db0e4 100644 --- a/.claude/mistakes.md +++ b/.claude/mistakes.md @@ -4,6 +4,13 @@ Append on error. Re-check before committing. Use `semantic_search "mistake =` or `<`, never `==` (timestamps have sub-second precision) +- ATS numeric fields: plain integers only — no currency symbols, commas, ranges, or units + ## Database Safety - Production DBs live in data/*.db — tests MUST NEVER touch these - All test fixtures use tmp_path or monkeypatch DB_PATH diff --git a/.claude/rules/jobs.md b/.claude/rules/jobs.md index 87223abc..d8f8a44f 100644 --- a/.claude/rules/jobs.md +++ b/.claude/rules/jobs.md @@ -5,11 +5,14 @@ - Logs to stdout; cron streams to Telegram. On ambiguity: STOP, tell human. ## Rate Limits -LinkedIn 20/day (guest API scan, Playwright Easy Apply only) | Greenhouse/Lever 15/day headed | Indeed/Workday/Generic 15/day | Reed 15/day API | Total 50/day +LinkedIn 15/day (session break 30min every 5 apps, `LINKEDIN_SESSION_CAP=5`) | Greenhouse/Lever 7/day headed | Indeed 8/day | Workday 5/day | Reed 7/day API | TotalJobs 4/day | Generic 5/day | Total 30/day +Session breaks: `SESSION_BREAK_EVERY=5`, `SESSION_BREAK_MINUTES=10` Safety: `JOB_AUTOPILOT_AUTO_SUBMIT=false` default, `JOB_AUTOPILOT_MAX_DAILY=10` ## Application Engine -- Playwright CDP to real Chrome. `PlaywrightAdapter` default for ALL platforms. +- `PlaywrightDriver` (`playwright_driver.py`): Core CDP driver — connects to real Chrome, human-like input, field interaction. +- `PlaywrightAdapter` (`playwright_adapter.py`): ATS adapter extending BaseATSAdapter, default for ALL platforms. +- `driver_protocol.py`: Driver interface protocol shared by both. - Platform strategies (`ats_adapters/strategy.py`): container hints, field ranges, screening defaults - Container-scoped CDP scan (`getPartialAXTree`). `FormExperienceDB` stores selectors/timing per domain. - `FAST_FILL=true` skips delays (Claude Code sessions) @@ -34,6 +37,7 @@ GitHub data already synced by 3am cron. Only Notion Skill Tracker needs live syn - Verify all GitHub URLs. No "JD Match" row. Headers: teal #1a5276. ## Pre-Screen Pipeline +- Route jobs via `classify_action()`, not `determine_match_tier()` — tier is display-only, not a routing signal - Gate 0 (`recruiter_screen.py`): title + keyword filter, pre-LLM - Gates 1-3 (`skill_graph_store.py`): kill signals, must-haves, competitiveness. Hybrid skill extraction (582 taxonomy → LLM fallback <10 skills) - Cross-platform dedup: same company+title = one job. K1 seniority kill: ≥3yr (not ≥5) @@ -57,6 +61,13 @@ Unverified skills → Notion "Pending" → user marks "I Know"/"Don't Know" → - Screening: ScreeningPipeline (cache + intent + alignment) → LLM fallback → SQLite cache. - All platforms → NativeFormFiller + `get_strategy(platform)` +**Verification primitives** (post 2026-05 hardening): +- `NavigationActionExecutor.execute()` reads back every fill, retries once on mismatch, returns `ExecutorResult`. +- `FormNavigator._verify_action(pre, post, action_kind)` is the shared verifier — `_phase_act` and `AuthHandler` both call it. +- `PageAction.expected_outcome` is a contract — set it correctly when extending the reasoner prompt. +- On ghost click: cache invalidation + reflection via `reason_with_failure`. Don't bypass — these run even on auth pages now. +- Low-confidence (`< 0.7`) actions trigger a screenshot-based page-type cross-check; disagreement invalidates the cache. + ## Form Scoping (`field_scanner.py`) 3-tier: Learned → Auto-detect (common ancestor JS) → Strategy hint. `validate_field_scan()` rejects noise. Self-healing: stale selectors auto-deleted. Scoped CDP → falls back to `getFullAXTree`. @@ -85,8 +96,9 @@ Fires after EVERY submission (both auto and manual paths). Three concerns: 3. Notion update → Applied status, dates, links Non-blocking. Runs before anti-detection delay. -## Real Data + Wiring Verification (MANDATORY) +## Real Data + Wiring + OPRAL (MANDATORY) Every new job pipeline feature: test with real job URLs, real profile data, real ATS pages (never mocks or stale fixtures). Then verify the full chain fires — `post_apply_hook` → `CorrectionCapture` → `AgentRulesDB` → `strategy_reflector` → `OptimizationEngine` → `AgentPerformanceDB` → Notion update. Not wired = not done. +On error: **Observe → Plan → Reason → Act → Learn**. Capture error context → trace root cause → determine which DB prevents recurrence → fix + re-run real data → emit signal + verify DB persisted + confirm autonomous on next run. ## PDF Upload Sanitized via PyMuPDF (`_sanitize_pdf()`). `set_input_files()` uses `{name, mimeType: "application/pdf", buffer}` — never bare paths. @@ -98,11 +110,29 @@ Sanitized via PyMuPDF (`_sanitize_pdf()`). `set_input_files()` uses `{name, mime ## Cover Letter Lazy via `cl_generator` callback. `build_dynamic_points()` maps projects→skills. `polish_points_llm()` ~$0.002. Static fallback. +## Security Wall Bypass (MANDATORY) +When any security wall is detected (Cloudflare, Turnstile, reCAPTCHA, hCaptcha, 403, 429), use Playwright's FULL capabilities to bypass it automatically. Human fallback is MANDATORY if auto-bypass fails. + +**6-stage bypass pipeline** (`_navigator.py:_bypass_verification_wall`): +1. **Auto-wait** (15s) — Cloudflare JS challenges auto-resolve in 3-10s. Poll every 3s. +2. **Human simulation** — Mouse movement, scrolling, random delays via Playwright `page.mouse.move()`, `page.evaluate("window.scrollBy()")`. +3. **Turnstile checkbox** — Locate Cloudflare iframe → enter content frame → click checkbox/challenge element. +4. **Page reload** — `page.reload(wait_until="domcontentloaded")` clears transient challenges. +5. **Second reload** — `page.reload(wait_until="networkidle")` with longer wait. +6. **Human fallback (MANDATORY)** — Telegram alert with URL + wall type. Poll 120s. Confirm cleared. If still blocked after 120s, skip job + Telegram notification. + +**Never abort without human fallback.** Even if all 5 auto-stages fail, the human MUST be asked. The pipeline MUST wait for human response before giving up. + +**Platform bypass** (`platform_bypass.py`): When aggregators (Indeed/LinkedIn/TotalJobs/Reed/Glassdoor) block persistently after all 6 stages, resolve the direct ATS URL instead. Resolution order: cached mapping → FormExperienceDB → known ATS board patterns (httpx HEAD) → Playwright web search. Stores results in NavigationLearner, GotchasDB, OptimizationEngine, ExperienceMemory, TrajectoryStore. Wired in `_navigator.py` after `_bypass_verification_wall()` returns `solved=False` on aggregator domains. + +**Detection**: `playwright_driver.py:get_snapshot()` inline JS detects Cloudflare selectors, text patterns, and iframe URLs. `page_analysis/classifier.py` weights `verification_wall_present` at 6.0. + ## Verification Wall Learning Universal detector (Turnstile/reCAPTCHA/hCaptcha/403/429). 17 signals per session. Statistical correlation (zero LLM). LLM every 5th block (~$0.002). Cooldown: 2hr→4hr→48hr exponential. Reset on success. Telegram alert on 3rd block. Adaptive params by risk level. ## Platform Quirks +- **LinkedIn**: Navigate to `/jobs/` first, then specific URL. Easy Apply badge can be `` not `