Disk-index-rag

.claude/settings.local.json

@@ -0,0 +1,19 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(git remote *)",
+      "Read(//Applications/**)",
+      "Read(//Users/alspenceramitojr/Applications/**)",
+      "Bash(pip3 list *)",
+      "Bash(python3 -c \"import cv2; print\\('opencv ok'\\)\")",
+      "Bash(python3 -c \"import pyttsx3; print\\('pyttsx3 ok'\\)\")",
+      "Bash(pip3 install *)",
+      "Bash(python3 -c \"import cv2, pyttsx3; print\\('deps ok'\\)\")",
+      "Bash(say -v '?')",
+      "Bash(pip show *)",
+      "Bash(pip install *)",
+      "Bash(python *)",
+      "Bash(python3 *)"
+    ]
+  }
+}

.cursor/plans/disk_layer_orchestrator_integration_2aac001c.plan.md

@@ -0,0 +1,106 @@
+---
+name: Disk layer orchestrator integration
+overview: Generalize the Spotlight+Cactus file resolver from a single `resume_path` into a role-based `resolved_local_files` map, run resolution at orchestrator startup (or keep a thin duplicate in `main` for clarity), and wire the visual planner plus executor so browser/local actions consume the right paths by key—enabling contextual disk access for multiple goals without hardcoding task-specific layout.
+todos:
+  - id: model-resolved-map
+    content: Introduce resolved_local_files (role→path), keep resume_path derived for YC compatibility; adjust file_resolve for multi-role + FIND_FILE key
+    status: completed
+  - id: orchestrator-hook
+    content: Call resolver at start of Orchestrator.run; merge into collected_data; upload_file resolves via file_role helper
+    status: completed
+  - id: planner-disk-context
+    content: Add disk_context to visual_planner prompt; document file_role in params for upload_file
+    status: completed
+  - id: optional-reveal
+    content: Add reveal_in_finder action + FilesystemExecutor helper (Phase B)
+    status: completed
+  - id: tests
+    content: Tests for multi-role resolution and orchestrator path selection
+    status: completed
+isProject: false
+---
+
+# Connect disk resolution to the orchestrator
+
+## Current state
+
+```mermaid
+flowchart LR
+  main[main.py] --> parse[parse_intent]
+  parse --> enrich[enrich_intent_with_resolved_files]
+  enrich --> slots["intent.slots resume_path only"]
+  slots --> orch[Orchestrator.run]
+  orch --> planner[visual_planner.choose_next_action]
+  orch --> exec["_execute_action upload_file"]
+  exec -->|"resume_path"| browser[yc_apply_fill]
+```
+
+- [`intent/file_resolve.py`](intent/file_resolve.py) is **resume-centric**: it gates on `_needs_resume_path()` and always writes `intent.slots["resume_path"]` (even for `FIND_FILE`, which is semantically wrong).
+- [`orchestrator/orchestrator.py`](orchestrator/orchestrator.py) never calls the resolver; it assumes upstream filled `resume_path` for `upload_file`.
+- [`orchestrator/visual_planner.py`](orchestrator/visual_planner.py) passes `intent_slots` + `collected_data` in JSON to Cactus but has **no explicit "disk summary"** field—paths are easy to miss or conflate.
+
+## Target shape
+
+**Single source of truth for resolved paths:** a small map on the intent (or merged into `collected_data` at orchestrator start):
+
+- `resolved_local_files: dict[str, str]` — **role → absolute path** (e.g. `"resume"`, `"attachment"`, `"deck"`, `"target"`).
+- Legacy **`resume_path`** can remain as a **derived convenience**: if `"resume"` exists in the map, set `slots["resume_path"]` for older YC browser code paths, or migrate call sites to `resolved_local_files["resume"]`.
+
+**Resolver behavior:**
+
+- Replace `_needs_resume_path` / single output with **file roles to resolve**, derived from:
+  - `intent.goal`,
+  - `intent.uses_local_data` (extend usage of tags like `resume`, `attachment`, `document`, …),
+  - optional parser slots such as `file_hints: list[{"role", "description"}]` (future-friendly).
+- For each role, run the **same** pipeline (Spotlight + predicate rounds + pick + refine on low confidence) with a **role-specific transcript hint** (e.g. append `"Role: attachment — user asked: …"`).
+- **FIND_FILE**: resolve the user’s query into one path and store under `resolved_local_files["found"]` (or `target`) instead of overloading `resume_path`.
+
+## Where to hook resolution (orchestrator connection)
+
+**Recommended:** call resolution **inside** [`Orchestrator.run`](orchestrator/orchestrator.py) **before** the first `choose_next_action`, e.g. `await resolve_local_paths_for_intent(intent, transcript)` which:
+
+1. Mutates `intent.slots` / attaches `resolved_local_files`.
+2. **`state.collected_data.update(...)`** so every planner step sees stable keys: `resolved_local_files`, plus backward-compatible `resume_path` if present.
+
+**Optional:** keep [`main.py`](main.py) calling the same function for logging UX ("parsed → resolved → execute") or remove duplicate and rely only on orchestrator—pick one to avoid double resolution (guard with `if "resolved_local_files" not in intent.slots` or idempotent resolver).
+
+## Visual planner: contextual disk access
+
+- Extend [`_build_prompt`](orchestrator/visual_planner.py) with a **compact, non-huge** block, e.g. `disk_context`:
+  - list of `{role, basename, parent_folder, age_hint}` — **not** full paths if you want safer logs; full paths stay in `collected_data` for execution only.
+- Update the instructions so Cactus knows **`upload_file` must reference which role** (`params.file_role: "resume"`) and the orchestrator resolves to a real path.
+- Extend [`ALLOWED_ACTION_TYPES`](orchestrator/visual_planner.py) / `_execute_action` minimally for general tasks:
+  - **`reveal_in_finder`** (macOS `open -R path`) — safe, useful after FIND_FILE.
+  - Later: **`attach_to_mail`** or pass `resolved_local_files` into existing Mail flows—only if you add AppleScript support.
+
+## Executor wiring
+
+- [`_execute_action`](orchestrator/orchestrator.py): change `upload_file` to resolve path as:
+  - `params.get("file_role")` → `data["resolved_local_files"][role]`, else fall back to `resume_path` / `resolved_local_files["resume"]`.
+- Add small helper `_path_for_file_action(collected_data, params) -> str`.
+- [`_run_local`](orchestrator/orchestrator.py) `find_file`: prefer `resolved_local_files` over raw `find_by_alias` when alias map is empty.
+- [`executors/local/filesystem.py`](executors/local/filesystem.py) (or `applescript.py`): add `reveal_in_finder(path: str)` using `subprocess` / `open -R`.
+
+## Parser and schema (light touch)
+
+- [`intent/schema.py`](intent/schema.py): document `slots` keys for `resolved_local_files` (no strict TypedDict required initially).
+- [`intent/parser.py`](intent/parser.py) / rules: optionally emit **file role hints** in `slots` when user says "attach my deck" (e.g. `uses_local_data` includes `document` and `slots["file_query"]` text)—feeds resolver prompts without new goals.
+
+## Tests
+
+- Unit: resolver writes multiple keys when mocked; orchestrator `upload_file` picks path by `file_role`.
+- Contract: `test_core_contracts` / smoke: intent with `resolved_local_files` only—no crash.
+
+## Phasing
+
+| Phase | Scope |
+|-------|--------|
+| **A** | Data model `resolved_local_files`, refactor `file_resolve` to multi-role, FIND_FILE uses `found` key, orchestrator calls resolver once + merges into `collected_data`, `upload_file` uses `file_role`. |
+| **B** | Visual planner `disk_context` block + prompt text + optional `reveal_in_finder` action + executor. |
+| **C** | Parser hints + Mail/Calendar attachments (if desired). |
+
+## Risks / constraints
+
+- **Token size:** keep planner `disk_context` summarized (basenames + age), not full directory dumps.
+- **Double work:** ensure resolver runs once per command (orchestrator-only vs main-only).
+- **Privacy:** logs should prefer basename over full path unless `FILE_RESOLVE_DEBUG`.

Ali/.gitignore

@@ -0,0 +1,31 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+.venv/
+venv/
+dist/
+build/
+.env
+*.log
+.DS_Store
+*.wav
+*.tmp
+
+# Node (vendored browser sub-agent)
+node_modules/
+package-lock.json.bak
+executors/browser/agent/server/dist/
+executors/browser/agent/dist/
+
+# Laptop-wide disk index (built locally, never committed)
+# Default location is ~/.cache/ali/ which is outside the repo, but protect
+# against relative ALI_INDEX_DIR overrides too.
+.cache/
+ali-index/
+index.db
+index.db-wal
+index.db-shm
+vectors.bin
+vectors_meta.json
+profile.json
+manifest.json

Ali/PLAN.md

@@ -0,0 +1,138 @@
+# Build Plan — YC Voice Agent
+Hackathon: April 18–19, 2026 | Submissions close: 11am Sunday
+
+---
+
+## Timeline
+
+### Saturday (Day 1)
+| Time | What |
+|---|---|
+| 10am–1pm | Setup, permissions, Cactus orientation, skeleton |
+| 1pm–6pm | Core loop: voice → intent → orchestrator → executor |
+| 6pm–10pm | Site adapters (YC Apply first), confirmation gate |
+| 10pm–2am | Polish demo flow, error handling, recovery |
+
+### Sunday (Day 2)
+| Time | What |
+|---|---|
+| 9am–11am | Full demo run-throughs, fix blockers |
+| 11am | Submissions close |
+| 1pm–2pm | Demo presentation |
+
+---
+
+## Build Sequence
+
+### Phase 0 — Saturday morning (before lunch)
+- [ ] All 4 teammates clone repo, confirm Python env works
+- [ ] Run `bash scripts/setup_macos_perms.sh` on the demo machine
+- [ ] Get Cactus installed, confirm Gemma 4 loads
+- [ ] Confirm Chrome persistent context opens to a logged-in session
+- [ ] Confirm `osascript` can send a test iMessage
+
+### Phase 1 — Voice capture (Layer 1)
+- [ ] `voice/capture.py` — hold Option key → start recording, release → stop
+- [ ] Raw audio bytes written to temp file
+- [ ] `voice/transcribe.py` — pipe audio to Cactus/Gemma 4 STT
+- [ ] Fallback path: if Cactus STT fails, use `faster-whisper` locally
+- [ ] Test: speak "apply to YC" → get transcript string
+
+### Phase 2 — Intent parsing (Layer 2)
+- [ ] `intent/schema.py` — define `IntentObject` dataclass and `KnownGoal` enum
+- [ ] `intent/parser.py` — prompt Gemma 4 with transcript, parse JSON response
+- [ ] Prompt design: classify goal, extract slots, flag resource needs
+- [ ] Test: transcript "apply to YC Fall 2026 using my resume" → correct intent object
+- [ ] Fallback: if parse fails, ask user to rephrase (don't crash)
+
+### Phase 3 — Orchestrator (Layer 3)
+- [ ] `orchestrator/state.py` — TaskState object: goal, plan, step_index, collected_data, status
+- [ ] `orchestrator/plans.py` — hardcoded plan for `apply_to_job`, `send_message`, `add_calendar_event`
+- [ ] `orchestrator/router.py` — given intent, return ordered list of executor calls
+- [ ] `orchestrator/orchestrator.py` — run plan, handle step failures, emit state updates
+- [ ] Test: intent object → correct plan selected → first step executed
+
+### Phase 4A — Local executor (Layer 4A)
+- [ ] `executors/local/filesystem.py` — named alias lookup (resume, cover letter, etc.)
+- [ ] `executors/local/applescript.py` — send iMessage, send Mail, create Calendar event
+- [ ] `executors/local/shell.py` — gated shell with allowlist
+- [ ] Test: "text Hanzi I'll be late" → iMessage sends (to test number first!)
+
+### Phase 4B — Browser executor (Layer 4B)
+- [ ] `executors/browser/browser.py` — Playwright persistent context, Chrome profile path
+- [ ] `executors/browser/adapters/yc_apply.py` — **primary demo adapter**
+  - Navigate to apply.ycombinator.com
+  - Map intent slots to form fields
+  - Upload resume PDF
+  - Pause before submit → confirmation gate
+- [ ] `executors/browser/adapters/linkedin.py` — Easy Apply flow (stretch)
+- [ ] `executors/browser/dom_agent.py` — general fallback (stretch/v2)
+- [ ] Test: run YC adapter on staging/personal account, confirm form fills correctly
+
+### Phase 5 — Confirmation gate (Layer 5)
+- [ ] `ui/confirmation.py` — modal dialog: show pending action text + Yes/No buttons
+- [ ] Voice confirmation: "yes" or "send it" → approve; "no" or "cancel" → abort
+- [ ] Wire into orchestrator: all write actions pause here before executing
+- [ ] `ui/menu_bar.py` — macOS menu bar icon, push-to-talk button, status indicator
+
+### Phase 6 — Integration + demo polish
+- [ ] `main.py` wires all layers in a loop
+- [ ] Full demo flow works end-to-end: voice → YC form filled → confirmation → submit
+- [ ] Second demo flow: voice → iMessage sent
+- [ ] Error states handled gracefully (no crashes on stage)
+- [ ] Status display shows current step in orchestrator
+
+---
+
+## Demo Script (rehearse this)
+
+1. "Watch — I'm going to apply to YC using just my voice."
+2. Hold Option key. Say: *"Apply to the YC Fall 2026 batch using my resume."* Release.
+3. Show transcript appears. Show intent object parsed locally.
+4. Browser opens, navigates to apply.ycombinator.com, fields start filling.
+5. Confirmation dialog: "I'm about to submit your YC application with resume.pdf. Send it?"
+6. Say "yes" → submits (or cancel on stage to not actually submit).
+7. "Everything that just happened — the parsing, the intent extraction — ran on my laptop. Nothing left my machine."
+
+---
+
+## Risk Register
+
+| Risk | Likelihood | Mitigation |
+|---|---|---|
+| Cactus STT unstable | Medium | Whisper fallback ready Day 1 |
+| Gemma 4 intent parsing unreliable | Medium | Hardcode 3 demo intents with fallback JSON |
+| YC Apply form changes | Low | Test adapter Saturday afternoon |
+| Chrome profile path wrong | Low | Config file, test Sunday 9am |
+| macOS permission dialogs during demo | High | Grant all permissions Saturday, test reboot |
+| Orchestrator infinite loop | Medium | Max steps limit + hard timeout |
+| Demo machine ≠ dev machine | Low | All run on single MacBook, no server deps |
+
+---
+
+## Division of Labor (suggestion)
+
+| Person | Layer |
+|---|---|
+| Alspencer | Orchestrator + main loop |
+| Hanzi | Browser executor + YC Apply adapter |
+| Ethan | Voice capture + Cactus/Gemma 4 integration |
+| Korin | UI (menu bar, confirmation gate) + demo polish |
+
+Overlap: everyone can touch intent/schema.py and plans.py — they're the connective tissue.
+
+---
+
+## Key Decisions
+
+**Push-to-talk over wake word** — wake word is flaky and will die on stage. Option key hold is legible, demonstrable, and debuggable.
+
+**State machine over agent loop** — every step is inspectable and interruptible. No runaway agents on stage.
+
+**Site adapters over general DOM agent** — 50-100 lines of Playwright per site is unbeatable for demo reliability. General agent is v2.
+
+**Hardcoded plans for known flows** — the LLM fills in slots, not the whole recipe. This is what makes v1 reliable.
+
+**Persistent Chrome context** — no re-login, no cookie setup, no OAuth dance. You drive the browser you're already logged into.
+
+**Confirmation gate on all writes** — this is not a limitation. It's the feature that makes the agent trustable and makes the demo pauseable.