diff --git a/README.md b/README.md index 018e905..190fe1d 100644 --- a/README.md +++ b/README.md @@ -1,32 +1,48 @@ # Vaultmind -**Vaultmind** is a local AI tool for Obsidian. It transforms your static notes into an active knowledge base by generating automated insights, daily briefings, and study aids. All powered by [Ollama](https://ollama.com). +AI toolkit for Obsidian. Transforms your static notes into an active knowledge base, insights, briefings, study aids, vault chat, and automated organization. Runs locally with Ollama or via NVIDIA NIM for cloud speed. [![vaultmind demo](https://img.youtube.com/vi/hqVmcqMPpUE/maxresdefault.jpg)](https://www.youtube.com/watch?v=hqVmcqMPpUE) + --- -## Key Features +## What it does -- **Weekly/Monthly Insights:** Multi-lens analysis (Emotional, Productivity, Patterns) of your recent activity. -- **Daily Morning Briefings:** Summarizes yesterday's progress and extracts pending tasks for today. -- **Intelligent Study Recaps:** Automatically generates spaced-repetition questions and finds connections to older notes. -- **Smart Ingestion:** A two-pass engine that converts raw text dumps and transcripts into structured, tagged Obsidian notes. +| Script | Description | +| :--- | :--- | +| `generate_insights.py` | Multi-lens weekly analysis with persistent AI memory | +| `morning_briefing.py` | Daily briefing with pending tasks and suggested focus | +| `study_recap.py` | Spaced repetition questions from your study notes | +| `txt_to_notes.py` | Converts any text into structured, linked Obsidian notes | +| `chat.py` | Persistent vault-aware conversation with SQLite memory | +| `auto_tagger.py` | Scans and injects missing tags without touching existing metadata | +| `moc_generator.py` | Builds Maps of Content to connect orphan notes | --- -## Hardware Requirements & Ollama models +## Models & Backends + +### Local — Ollama (privacy first) + +Runs entirely on your machine. No data leaves your vault. + +| Model | Best for | +| :--- | :--- | +| `deepseek-r1:8b` | Recommended default — best reasoning, lowest hallucination | +| `llama3.1:8b` | Faster, slightly less accurate | +| `llama3.2:3b` | Edge devices and systems without a dedicated GPU | -Vaultmind performance depends entirely on your local LLM runner (Ollama). +> Not sure what your hardware can handle? Check the [LLM Hardware Requirements Guide](https://onyx.app/llm-hardware-requirements). -> **Tip:** If you are unsure what your specific hardware can handle, check the [LLM Hardware Requirements Guide](https://onyx.app/llm-hardware-requirements) for a detailed breakdown of parameters vs. VRAM. +### Cloud — NVIDIA NIM (speed & low-end hardware) -While Ollama supports hundreds of models, Vaultmind has been specifically tested with the following: +Offloads processing to NVIDIA's infrastructure. Useful if you don't have a GPU or want to run 70B+ models. -* **DeepSeek-R1 (8B):** The recommended default. Superior for the `generate_insights.py` script due to its advanced reasoning and low hallucination rate. -* **Llama 3.1 (8B):** Runs a little faster but with a little less accuracy. -* **Llama 3.2 (3B):** Optimized (kinda) for edge devices and systems without dedicated GPUs. +- Tested model: `meta/llama-3.3-70b-instruct` +- Requires an API key and internet connection +- Free tier has rate limits, but you're unlikely to hit them for normal vault use -**Disclaimer:** Using models not listed above may result in unexpected output formats, conversational "chatter", or failure to parse the JSON planning phase in `txt_to_notes.py`. +[Get your NVIDIA NIM API key](https://build.nvidia.com/explore/discover) --- @@ -34,55 +50,72 @@ While Ollama supports hundreds of models, Vaultmind has been specifically tested ### 1. Prerequisites -- [Ollama](https://ollama.com) installed and running -- Python 3.10 or higher +- Python 3.10+ +- [Ollama](https://ollama.com) installed (if running locally) -### 2. Setup +### 2. Clone and install ```bash -# Clone the repository git clone https://github.com/paulobarb/vaultmind.git cd vaultmind -# Create virtual environment python -m venv venv source venv/bin/activate # Windows: venv\Scripts\activate -# Install dependencies -pip install requests +pip install requests pyyaml ``` -### 3. Initialize Model - -Choose a model: +### 3. Choose your backend +**Option A — Local (Ollama)** ```bash ollama pull deepseek-r1:latest ``` +In `config.py`, set `USE_NVIDIA_NIM = False` and verify the model name matches. + +**Option B — Cloud (NVIDIA NIM)** + +In `config.py`, set `USE_NVIDIA_NIM = True` and paste your API key: +```python +NVIDIA_API_KEY = "your_key_here" +``` --- ## Configuration -Vaultmind is customizable via `config.py`. +All settings live in `config.py`. The most important ones: | Setting | Description | | :--- | :--- | -| `VAULT_PATH` | Absolute path to your Obsidian vault | -| `NUM_CTX` | Context window size — increase for long notes, decrease if you encounter VRAM spillover | -| `TEMPERATURE` | `0.2` for factual briefings, `0.7` for more creative insights | -| `EXCLUDED_FOLDERS` | List of folders the AI should ignore (e.g. Templates, Archive) | +| `USE_NVIDIA_NIM` | `True` for cloud, `False` for local Ollama | +| `VAULT_PATH` | Path to your Obsidian vault | +| `NUM_CTX` | Context window size — increase for larger models or long notes | +| `TEMPERATURE` | `0.2` for factual output, `0.7` for creative insights | +| `EXCLUDED_FOLDERS` | Folders the AI should ignore (e.g. Templates, Archive) | +| `SAVE_CHAT_HISTORY` | `True` to persist chat history in SQLite, `False` for amnesiac mode | + +> If you upgrade to larger models (35B, 70B, or NVIDIA NIM), increase `NUM_CTX` and `MAX_NOTE_CHARS` for better results. + +All prompts are in `prompts.json`. --- ## The Toolkit -### ☀️ Morning Briefing (`morning_briefing.py`) +### ☀️ Morning Briefing + +Splits your recent notes into "Yesterday" (what you executed) and "Today" (what you planned). Finds unfinished checkboxes and generates a suggested focus so you never start the day cold. + +```bash +python morning_briefing.py +``` + +--- -The Daily Momentum Builder +### 📊 Weekly Insights -- It categorizes notes into "Yesterday" (execution) and "Today" (planning). It scans for unfinished tasks (empty checkboxes) and unresolved thoughts. -- It provides a "Suggested Focus," acting as a bridge between the work you finished and the work you haven't started yet, ensuring you never wake up with a "cold start" in your vault. +Runs your notes through parallel analysis lenses (Therapist, Coach, Pattern Detector, Strengths, Connections). Reads `AI_State.md` at startup, a compressed history of previous weeks and compares it against your current notes to detect drift and track long-term progress. Updates the state file after each run. ```bash python generate_insights.py @@ -90,69 +123,112 @@ python generate_insights.py --- -### 📊 Weekly Insights (`generate_insights.py`) +### 🧠 Study Recap + +Auto-detects notes modified in your last study session and lets you refine the list. Cross-references them with your full vault to find hidden connections, then generates spaced repetition questions to fight the forgetting curve. + +```bash +python study_recap.py +``` + +--- + +### 📥 Text to Notes -It uses a Parallel Processing model to run your notes through multiple "lenses" (like a Therapist or a Pattern Detector) simultaneously to save time. +Two-pass ingestion engine for messy data. transcripts, braindumps, articles. -- It reads a file called AI_State.md at startup. This file contains a compressed history of who you were last week. -- The AI compares your current notes against that state to identify "drift"—checking if you are actually making progress on your goals or just repeating the same cycles. It then updates the state file, making the AI following your journey every time you run it. +- **Pass 1 (Plan):** AI reads the raw text and creates a blueprint — how many notes, what titles, which existing vault tags to reuse +- **Pass 2 (Write):** Writes the actual content with `[[Wikilinks]]` to your existing knowledge base ```bash -python generate_insights.py +python txt_to_notes.py my_transcript.txt ``` --- -### 🧠 Study Recap (`study_recap.py`) +### 💬 Vault Chat -This is an Interactive Tool designed to fight the "forgetting curve". Unlike the other scripts, this one waits for your input. +Conversation engine with access to your vault. Saves history to a local SQLite database so the AI maintains context across sessions. Commands: -- It auto-detects notes modified in a specific window (e.g., your last 24 hours of studying) and lets you manually refine the list. It then cross-references these with the rest of your vault to find "hidden connections." -- The Result: It generates a dedicated recap note filled with Spaced Repetition questions. It transforms passive reading into active testing. +``` +/memories list saved exchanges +/forget delete a specific exchange +/clear wipe all history +/exit quit +``` ```bash -python study_recap.py +python chat.py ``` --- -### 📥 TXT to Notes (txt_to_notes.py`) +### 🏷️ Auto Tagger -This script handles the "messy" data—transcripts, braindumps, or long articles. It uses a Plan-then-Execute architecture to prevent the AI from getting lost in long texts. +Scans for notes missing tags and injects them without touching existing metadata. Supports non-English characters (á, ç, ú). Two modes: -- Pass 1 (Planning): The AI reads the raw text and creates a JSON "blueprint." It decides how many notes are needed, what the titles should be, and which existing tags from your vault to reuse. -- Pass 2 (Writing): Using that blueprint, it writes the actual content, automatically creating [[Wikilinks]] between the new notes and your existing knowledge base. +- **Vault-wide:** finds all untagged notes automatically +- **Targeted:** force re-tag a specific file by name ```bash -python txt_to_notes.py my_transcript.txt +python auto_tagger.py ``` --- -## Customization +### 🗺️ MOC Generator -Edit `prompts.json` to change how the AI speaks or what it focuses on. +Builds a Map of Content for your vault in three modes: + +- **Tag/Keyword** - finds all notes containing a specific tag or keyword (e.g. `#productivity`) +- **Anchor Note** - finds all notes that link to a specific note (e.g. `MyNote`) +- **Folder Index** - indexes all notes inside a specific folder (e.g. `Projects/MyProject`) + +Generates a structured index note with automatic `[[Wikilinks]]`. +```bash +python moc_generator.py +``` --- -## Automation (Linux / macOS) +## Automation -Add scripts to your crontab to have briefings waiting every morning (only works if you have your machine 24/7 on): +Add to crontab for automatic daily and weekly runs (machine must be on): ```bash -# Run morning briefing at 7:00 AM daily +# Morning briefing — every day at 7am 0 7 * * * /path/to/vaultmind/venv/bin/python /path/to/vaultmind/morning_briefing.py -# Run weekly insights every Sunday at 8:00 AM +# Weekly insights — every Sunday at 8am 0 8 * * 0 /path/to/vaultmind/venv/bin/python /path/to/vaultmind/generate_insights.py ``` --- -## Limitations & Accuracy +## Project structure + +``` +vaultmind/ +├── core/ # Shared AI logic and backend routing +├── data/ # Local SQLite databases +├── config.py # All settings — start here +├── prompts.json # All AI prompts — edit to customize behavior +├── auto_tagger.py +├── chat.py +├── generate_insights.py +├── moc_generator.py +├── morning_briefing.py +├── study_recap.py +└── txt_to_notes.py +``` + +--- + +## Limitations -- **Context Limits:** Very long notes may require increasing `NUM_CTX` in `config.py`. -- **Local Speed:** Performance is tied to your hardware. Large-scale analysis is significantly faster with a dedicated GPU. +- **AI variability:** Output depends on your model. Always verify especially the Pending section in briefings. +- **Hardware:** CPU only or weak GPU machines will be slow on multi-pass scripts like `txt_to_notes.py`. +- **Personal tool:** Built for a specific workflow. Edit `prompts.json` or the scripts to fit yours. --- diff --git a/auto_tagger.py b/auto_tagger.py new file mode 100644 index 0000000..194a30a --- /dev/null +++ b/auto_tagger.py @@ -0,0 +1,126 @@ +# ============================================================= +# vaultmind — auto_tagger.py +# ============================================================= +# Scans the vault for notes missing structural tags (no frontmatter, +# missing 'tags' key, or empty tags list) and automatically +# generates them using AI based on the note's content. +# ============================================================= + +import sys +import re +import yaml +from pathlib import Path +from config import VAULT_PATH, EXCLUDED_FOLDERS +from core.ai_backend import get_backend, run_startup_checks +from core.tagger_logic import collect_vault_tags, get_ai_tags + +# --- SETUP & PATHS --- +SCRIPT_DIR = Path(__file__).parent.resolve() +sys.path.insert(0, str(SCRIPT_DIR)) + +# --- COLORS --- +CYAN, GREEN, YELLOW, DIM, BOLD, RESET = "\033[96m", "\033[92m", "\033[93m", "\033[2m", "\033[1m", "\033[0m" +RED = "\033[31m" + +def has_no_tags(content: str) -> bool: + """ + Checks if a note is missing tags structurally. + Returns True if: + 1. No frontmatter exists. + 2. Frontmatter exists but has no 'tags' key. + 3. Frontmatter has 'tags' but it's empty. + """ + fm_match = re.match(r'^---(.*?)---', content, re.DOTALL) + if not fm_match: + return True + + try: + data = yaml.safe_load(fm_match.group(1)) + if not data or 'tags' not in data: + return True + if not data['tags'] or len(data['tags']) == 0: + return True + return False + except Exception: + return True # If YAML is broken, consider it "untagged" + +def process_note(path: Path, existing_tags: list, backend: str): + """Adds tags while preserving existing non-tag properties (word, link, etc).""" + content = path.read_text(encoding="utf-8", errors="ignore") + + # Separate Frontmatter and Body + fm_match = re.match(r'^---(.*?)---', content, re.DOTALL) + if fm_match: + try: + metadata = yaml.safe_load(fm_match.group(1)) or {} + except Exception: # <-- FIXED: E722 No bare 'except' + metadata = {} + body = content[fm_match.end():].strip() + else: + metadata = {} + body = content.strip() + + # Get AI Tags + print(f" {DIM}· Consulting AI...{RESET}", end="\r") + ai_suggestions = get_ai_tags(body, existing_tags, backend) + + # Filter out technical loop-terms + final_tags = [t for t in ai_suggestions if t.lower() != "untagged"] + if not final_tags: + final_tags = ["needs-review"] + + # Update Metadata + metadata['tags'] = final_tags + + # Rebuild File + new_fm = yaml.dump(metadata, sort_keys=False, allow_unicode=True).strip() + new_content = f"---\n{new_fm}\n---\n\n{body}" + + path.write_text(new_content, encoding="utf-8") + print(f" {GREEN}· Applied: {BOLD}{', '.join(final_tags)}{RESET}") + +def main(): + backend = get_backend() + run_startup_checks() + vault = Path(VAULT_PATH).expanduser().resolve() + + print(f"\n{CYAN}{BOLD}VAULT TAG MANAGER{RESET}") + print(f"{DIM}1. Auto-tag all notes missing properties/tags{RESET}") + print(f"{DIM}2. Tag a specific note by name{RESET}") + + choice = input(f"\n{YELLOW}Select > {RESET}").strip() + existing_tags = collect_vault_tags() + + if choice == '1': + targets = [] + for p in vault.rglob("*.md"): + if any(skip in str(p) for skip in EXCLUDED_FOLDERS): + continue + content = p.read_text(encoding="utf-8", errors="ignore") + + if has_no_tags(content): + targets.append(p) + + if not targets: + print(f"{GREEN}All notes are properly tagged!{RESET}\n") + return # <-- FIXED: E702 Split semicolon into two lines + + print(f"{DIM}• Found {len(targets)} note(s) missing tags. Processing...{RESET}") + for i, path in enumerate(targets): + print(f" {CYAN}[{i+1}/{len(targets)}]{RESET} {BOLD}{path.name}{RESET}") + process_note(path, existing_tags, backend) + + print(f"\n{GREEN}Vault tagging complete.{RESET}\n") + + elif choice == '2': + query = input(f"{YELLOW}Enter filename > {RESET}").strip() + clean_query = query[:-3].lower() if query.lower().endswith(".md") else query.lower() + target_path = next((p for p in vault.rglob("*.md") if p.stem.lower() == clean_query), None) + + if target_path: + process_note(target_path, existing_tags, backend) + else: + print(f"{RED}File not found.{RESET}") + +if __name__ == "__main__": + main() \ No newline at end of file diff --git a/chat.py b/chat.py index 3ba8941..41fdb1f 100644 --- a/chat.py +++ b/chat.py @@ -27,13 +27,11 @@ SAVE_CHAT_HISTORY, HISTORY_LIMIT, ) -from ai_backend import call_ai, get_backend, run_startup_checks +from core.ai_backend import call_ai, get_backend, run_startup_checks SCRIPT_DIR = Path(__file__).parent sys.path.insert(0, str(SCRIPT_DIR)) - -# --- COLORS --- CYAN = "\033[96m" GREEN = "\033[92m" YELLOW = "\033[93m" @@ -42,7 +40,7 @@ BOLD = "\033[1m" RESET = "\033[0m" -DB_FILE = SCRIPT_DIR / "chat_history.db" +DB_FILE = Path(__file__).parent / "data" / "chat_history.db" TEMP_CHAT = TEMPERATURES.get("chat") or TEMPERATURES.get("default") or 0.2 @@ -151,11 +149,9 @@ def retrieve_context(query: str, all_paths: list[Path], backend: str) -> tuple: Returns: Tuple of (context_blocks, matched_filenames) """ - # step 1: keyword force — catch exact filename matches query_words = [w for w in re.findall(r'\w+', query.lower()) if len(w) > 3] forced = [p for p in all_paths if any(w in p.name.lower() for w in query_words)] - # step 2: librarian — ask AI to rank all files print(f"{DIM} ranking files...{RESET}", end="\r") file_list = "\n".join(p.name for p in all_paths) librarian_prompt = ( @@ -166,7 +162,6 @@ def retrieve_context(query: str, all_paths: list[Path], backend: str) -> tuple: raw = call_ai(librarian_prompt, backend, temperature=0.0) suggested = [clean_filename(line) for line in raw.splitlines() if line.strip()] - # step 3: read content of matched files context_blocks, matched_names = [], [] seen = set() diff --git a/config.py b/config.py index 7c73eef..e695808 100644 --- a/config.py +++ b/config.py @@ -4,7 +4,7 @@ # -- NVIDIA NIM -- USE_NVIDIA_NIM = False # Change to True if you want to use NVIDIA NIM -NVIDIA_API_KEY = "nvapi-xxxxxxx" # Your API KEY here +NVIDIA_API_KEY = "nvapi-xxxxxxxxxx" # Your API KEY here NVIDIA_MODEL = "meta/llama-3.3-70b-instruct" # Chose your model # meta/llama-3.3-70b-instruct was the only model tested @@ -39,7 +39,7 @@ VAULT_PATH = "~/Obsidian" # Path to your Obsidian vault. MAX_FILE_SIZE = 1_000_000 # 1MB max per file, skip larger ones -MAX_NOTE_CHARS = 15000 # Max characters read per note +MAX_NOTE_CHARS = 8000 # Max characters read per note MAX_CONTENT_NOTES = 12 # Max files sent to AI per query # add any folder you want to ignore here @@ -48,7 +48,8 @@ "Insights", "Study Recaps", "Captures", - # "Templates", + "MOC", + "Templates", # "Archive", ] @@ -66,6 +67,8 @@ INSIGHT_TITLE_FORMAT = "{date} {period} Insight.md" # Change here how would you like your title # Placeholders: {date}, {period} +INSIGHTS_DIR_NAME = "Insights" # Set the folder name for Insights notes + # Words that trigger specific tags CANDIDATES = { "productivity": ["productiv", "task", "goal", "work", "focus"], @@ -78,15 +81,31 @@ # "finances": ["money", "budget", "spend", "invest", "finance"], } + # -- Morning Briefing (morning_briefing.py) -- BRIEFING_TITLE_FORMAT = "{date} Morning Briefing.md" # Change here how would you like your title # Placeholders: {date} +BRIEFING_DIR_NAME = "Briefings" # Set the folder name for Morning Briefings + + # -- Study Recap (study_recap.py) -- HOURS_BACK = 24 # How many hours back to auto-detect notes RECAP_TITLE_FORMAT = "{date} ({time}) Study Recap — {subject}.md" # Change here how would you like your title # Placeholders: {date}, {time}, {subject} +STUDY_DIR_NAME = "Study Recaps" # Set the folder name for Study Recap + + # -- TXT to Notes (txt_to_notes.py) -- TXT_TITLE_FORMAT = "{title}.md" # Change here how would you like your title - # Placeholders: {title} \ No newline at end of file + # Placeholders: {title} + +CAPTURES_DIR_NAME = "Captures" # Set the folder name for TXT to Notes + + +# -- MOC Generator (moc_generator.py) -- +MOC_DIR_NAME = "MOC" # Change here how would you like your title + # Placeholders: {title} + +MOC_TITLE_FORMAT = "{title} MOC" # Set the folder name for MOC Generator \ No newline at end of file diff --git a/ai_backend.py b/core/ai_backend.py similarity index 100% rename from ai_backend.py rename to core/ai_backend.py diff --git a/core/tagger_logic.py b/core/tagger_logic.py new file mode 100644 index 0000000..f28cc46 --- /dev/null +++ b/core/tagger_logic.py @@ -0,0 +1,66 @@ +# ============================================================= +# vaultmind — tagger_logic.py +# ============================================================= +# Shared tagging engine. Centralizes vault tag collection and +# AI-based tag generation. +# ============================================================= + +import re +import json +from pathlib import Path +from config import MAX_FILE_SIZE, VAULT_PATH +from core.ai_backend import call_ai + +PROMPTS_PATH = Path(__file__).parent.parent / "prompts.json" + +with PROMPTS_PATH.open(encoding="utf-8") as f: + PROMPTS = json.load(f) + +def format_tag(text: str) -> str: + text = re.sub(r'^(tags?:|here are.*?:|output:)\s*', '', text, flags=re.IGNORECASE) + + text = text.lower().replace(" ", "-") + return re.sub(r'[^\w-]', '', text) + +def collect_vault_tags() -> list[str]: + """Scans the vault to find every tag you've already used.""" + tags = set() + vault = Path(VAULT_PATH).expanduser().resolve() + for path_obj in vault.rglob("*.md"): + try: + if path_obj.stat().st_size > MAX_FILE_SIZE: + continue + content = path_obj.read_text(encoding="utf-8", errors="ignore") + + # Match frontmatter tags + fm = re.match(r"^---\n(.*?)\n---", content, re.DOTALL) + if fm: + for line in fm.group(1).splitlines(): + m = re.match(r"\s*-\s*(.+)", line) + if m: + tags.add(m.group(1).strip().lower()) + + # Match inline #hashtags + for t in re.findall(r"#([a-zA-Z][a-zA-Z0-9_/-]+)", content): + tags.add(t.lower()) + except Exception: + continue + + return sorted([format_tag(t) for t in tags if format_tag(t)]) + +def get_ai_tags(content: str, existing_tags: list, backend: str) -> list[str]: + """Ask the AI for a simple list of 3-4 relevant tags using the JSON prompt.""" + tags_hint = ", ".join(existing_tags[:80]) + + # Use the JSON template + template = PROMPTS["auto_tagger"]["prompt"] + prompt = template.format(content=content[:4000], tags_hint=tags_hint) + + raw = call_ai(prompt, backend, temperature=0.0) + + # Clean up and split + raw_list = raw.replace("\n", ",").split(",") + ai_tags = [format_tag(t.strip()) for t in raw_list if t.strip()] + + final_tags = sorted(list(set([t for t in ai_tags if t])))[:4] + return final_tags if final_tags else ["untagged"] \ No newline at end of file diff --git a/generate_insights.py b/generate_insights.py index 0edec83..ddce2a3 100644 --- a/generate_insights.py +++ b/generate_insights.py @@ -21,14 +21,15 @@ import datetime from pathlib import Path from concurrent.futures import ThreadPoolExecutor, as_completed -from config import MAX_FILE_SIZE, VAULT_PATH, DAYS_BACK, MAX_NOTE_CHARS, EXCLUDED_FOLDERS, CANDIDATES, TEMPERATURES, INSIGHT_TITLE_FORMAT -from ai_backend import get_backend, call_ai, run_startup_checks +from config import MAX_FILE_SIZE, VAULT_PATH, DAYS_BACK, MAX_NOTE_CHARS, EXCLUDED_FOLDERS, CANDIDATES, TEMPERATURES, INSIGHT_TITLE_FORMAT, INSIGHTS_DIR_NAME +from core.ai_backend import get_backend, call_ai, run_startup_checks SCRIPT_DIR = Path(__file__).parent sys.path.insert(0, str(SCRIPT_DIR)) VAULT_PATH = Path(VAULT_PATH).expanduser().resolve() -INSIGHT_FOLDER = VAULT_PATH / "Insights" +INSIGHT_FOLDER = VAULT_PATH / INSIGHTS_DIR_NAME +INSIGHT_FOLDER.mkdir(parents=True, exist_ok=True) STATE_FILE = INSIGHT_FOLDER / "AI_State.md" # --- COLORS --- diff --git a/moc_generator.py b/moc_generator.py new file mode 100644 index 0000000..0e380dc --- /dev/null +++ b/moc_generator.py @@ -0,0 +1,193 @@ +# ============================================================= +# vaultmind — moc_generator.py +# ============================================================= +# Multi-Strategy Map of Content (MOC) Generator. +# Organizes notes via Tags, Anchor Links, or Folder Structures. +# ============================================================= + +import sys +import re +import json +import datetime +from pathlib import Path +from config import VAULT_PATH, EXCLUDED_FOLDERS, MOC_DIR_NAME, MOC_TITLE_FORMAT +from core.ai_backend import get_backend, call_ai, run_startup_checks + +# --- SETUP & PATHS --- +SCRIPT_DIR = Path(__file__).parent.resolve() +sys.path.insert(0, str(SCRIPT_DIR)) + +# --- LOAD PROMPTS --- +PROMPTS_PATH = SCRIPT_DIR / "prompts.json" +with PROMPTS_PATH.open(encoding="utf-8") as f: + PROMPTS = json.load(f) + +# --- COLORS & CONSTANTS --- +CYAN, GREEN, YELLOW, DIM, BOLD, RESET = "\033[96m", "\033[92m", "\033[93m", "\033[2m", "\033[1m", "\033[0m" +RED = "\033[31m" + +VAULT = Path(VAULT_PATH).expanduser().resolve() +MOC_FOLDER = VAULT / MOC_DIR_NAME + +# ============================================================= +# SEARCH STRATEGIES +# ============================================================= + +def get_all_notes() -> list[Path]: + notes = [] + for path in VAULT.rglob("*.md"): + if any(skip in str(path) for skip in EXCLUDED_FOLDERS) or MOC_DIR_NAME in str(path): + continue + notes.append(path) + return notes + +def find_by_tag(topic: str) -> list[dict]: + relevant = [] + search_term = topic.replace("#", "").lower() + for path in get_all_notes(): + try: + content = path.read_text(encoding="utf-8", errors="ignore") + if search_term in content.lower() or search_term in path.name.lower(): + relevant.append({"title": path.stem, "snippet": content[:400]}) + except Exception: + continue + return relevant + +def find_by_anchor(anchor_title: str) -> list[dict]: + relevant = [] + anchor_title_clean = anchor_title.replace(".md", "") + link_pattern = re.compile(r"\[\[([^|\]]+).*?\]\]") + outgoing_links = set() + + anchor_found = False + for path in get_all_notes(): + if path.stem.lower() == anchor_title_clean.lower(): + anchor_found = True + content = path.read_text(encoding="utf-8", errors="ignore") + matches = link_pattern.findall(content) + outgoing_links.update([m.strip() for m in matches]) + break + + if not anchor_found: + print(f"{RED}Could not find an anchor note named '{anchor_title_clean}'.{RESET}") + return [] + + for path in get_all_notes(): + if path.stem.lower() == anchor_title_clean.lower(): + continue + try: + content = path.read_text(encoding="utf-8", errors="ignore") + is_incoming = f"[[{anchor_title_clean}]]" in content or f"[[{anchor_title_clean}|" in content + is_outgoing = path.stem in outgoing_links + + if is_incoming or is_outgoing: + relevant.append({"title": path.stem, "snippet": content[:400]}) + except Exception: + continue + return relevant + +def find_by_folder(folder_rel_path: str) -> list[dict]: + relevant = [] + target_folder = VAULT / folder_rel_path + if not target_folder.exists() or not target_folder.is_dir(): + print(f"{RED}Folder not found: {target_folder}{RESET}") + return [] + + for path in target_folder.rglob("*.md"): + try: + content = path.read_text(encoding="utf-8", errors="ignore") + relevant.append({"title": path.stem, "snippet": content[:400]}) + except Exception: + continue + return relevant + +# ============================================================= +# AI GENERATION +# ============================================================= + +def generate_moc_content(topic: str, context_type: str, notes: list, backend: str) -> str: + notes_data = "\n".join([f"TITLE: {n['title']} | SNIPPET: {n['snippet']}" for n in notes]) + + if context_type == "anchor": + context_prompt = "These notes are part of a localized graph cluster connected to the anchor note." + elif context_type == "folder": + context_prompt = "These notes reside in the same directory. Create a structured index for this folder." + else: + context_prompt = "These notes share a common theme or tag." + + template = PROMPTS["moc_generator"]["prompt"] + prompt = template.format( + topic=topic, + context_prompt=context_prompt, + notes_data=notes_data + ) + + return call_ai(prompt, backend, temperature=0.1) + +# ============================================================= +# MAIN INTERFACE +# ============================================================= + +def main(): + backend = get_backend() + run_startup_checks() + + print(f"\n{CYAN}{BOLD}VAULT MOC ARCHITECT{RESET}") + print(f"{DIM}1. Search by Tag/Keyword (e.g., #productivity){RESET}") + print(f"{DIM}2. Anchor Note Graph (e.g., MyNote){RESET}") + print(f"{DIM}3. Folder Index (e.g., Projects/MyProject){RESET}") + + choice = input(f"\n{YELLOW}Select strategy (1/2/3) > {RESET}").strip() + + if choice == '1': + topic = input(f"{YELLOW}Enter Keyword/Tag > {RESET}").strip() + print(f"{DIM}• Scanning vault for '{topic}'...{RESET}", end="\r") + notes = find_by_tag(topic) + context_type = "tag" + filename_base = topic.replace('#', '') + + elif choice == '2': + topic = input(f"{YELLOW}Enter Exact Anchor Note Title > {RESET}").strip() + print(f"{DIM}• Mapping connections for '[[{topic}]]'...{RESET}", end="\r") + notes = find_by_anchor(topic) + context_type = "anchor" + filename_base = f"{topic} Network" + + elif choice == '3': + topic = input(f"{YELLOW}Enter Folder Path (relative to Vault) > {RESET}").strip() + print(f"{DIM}• Reading contents of '{topic}/'...{RESET}", end="\r") + notes = find_by_folder(topic) + context_type = "folder" + filename_base = Path(topic).name + + else: + print(f"{RED}Invalid choice.{RESET}") + sys.exit(1) + + if not notes: + print(f"\n{RED}No notes found for this query.{RESET}\n") + return + + print(f"{DIM}• Organizing {len(notes)} notes into an MOC...{' '*15}{RESET}") + moc_body = generate_moc_content(topic, context_type, notes, backend) + + # --- FILENAME GENERATION --- + MOC_FOLDER.mkdir(parents=True, exist_ok=True) + + # Sanitize and format the title + clean_title = re.sub(r'[\\/*?:"<>|]', "", filename_base).title() + final_title = MOC_TITLE_FORMAT.format(title=clean_title) + filepath = MOC_FOLDER / f"{final_title}.md" + + # --- SAVE --- + now = datetime.datetime.now().strftime("%Y-%m-%d") + frontmatter = f"---\ntags:\n - moc\ncreated: {now}\n---\n\n# {final_title}\n\n" + + filepath.write_text(frontmatter + moc_body, encoding="utf-8") + + print(f"{DIM}{'─'*45}{RESET}") + print(f"{GREEN}│ Map Created: {final_title}.md{RESET}") + print(f"{DIM}│ Check the '{MOC_DIR_NAME}' folder.{RESET}\n") + +if __name__ == "__main__": + main() \ No newline at end of file diff --git a/morning_briefing.py b/morning_briefing.py index 03f99e9..744e712 100644 --- a/morning_briefing.py +++ b/morning_briefing.py @@ -15,8 +15,8 @@ import sys import json import datetime -from config import EXCLUDED_FOLDERS, MAX_FILE_SIZE, VAULT_PATH, MAX_NOTE_CHARS, TEMPERATURES, BRIEFING_TITLE_FORMAT -from ai_backend import get_backend, call_ai, run_startup_checks +from config import EXCLUDED_FOLDERS, MAX_FILE_SIZE, VAULT_PATH, MAX_NOTE_CHARS, TEMPERATURES, BRIEFING_TITLE_FORMAT, BRIEFING_DIR_NAME +from core.ai_backend import get_backend, call_ai, run_startup_checks CYAN, GREEN, YELLOW, DIM, BOLD, RESET = "\033[96m", "\033[92m", "\033[93m", "\033[2m", "\033[1m", "\033[0m" @@ -24,7 +24,8 @@ sys.path.insert(0, str(SCRIPT_DIR)) VAULT_PATH = Path(VAULT_PATH).expanduser().resolve() -BRIEFING_FOLDER = VAULT_PATH / "Briefings" +BRIEFING_FOLDER = VAULT_PATH / BRIEFING_DIR_NAME +BRIEFING_FOLDER.mkdir(parents=True, exist_ok=True) # load prompts from prompts.json PROMPTS_PATH = SCRIPT_DIR / "prompts.json" diff --git a/prompts.json b/prompts.json index d57fe24..f70071d 100644 --- a/prompts.json +++ b/prompts.json @@ -71,5 +71,16 @@ "write_prompt": "You are writing a rich, meaningful Obsidian note.\n{instructions_block}\n{grounding}\n\nNote title: \"{title}\"\nNote type: {type}\nWhat this note covers: {summary}\n\nSOURCE TEXT:\n{text}\n\nOther notes in this same batch — link with [[wikilinks]] when relevant:\n{batch_links}\n\nExisting vault notes to link when truly relevant:\n{related_links}\n\nExisting vault tags to reuse:\n{tags_hint}\n\nWrite the full note body in Markdown. Requirements:\n- Start directly with content, no title heading\n- Use # headings to organize sections naturally\n- Be specific and detailed — no vague summaries, include real context\n- Use [[wikilinks]] to link to other notes in this batch or existing vault notes when it genuinely adds value\n- If there are tasks or follow-ups, use - [ ] checkbox format\n- Write in first person if it's a personal note\n- Minimum 200 words, aim for depth over brevity\n- Match the language of the source text\n- Do NOT invent information not present in the source text", "_write_prompt_tip": "Placeholders: {instructions_block}, {grounding}, {title}, {type}, {summary}, {text}, {batch_links}, {related_links}, {tags_hint}." + }, + + "moc_generator": { + "_info": "Used by moc_generator.py. Creates structured Maps of Content from tags, folders, or anchor notes.", + "prompt": "MOC FOCUS: {topic}\nCONTEXT: {context_prompt}\n\nVAULT NOTES (DO NOT ALTER THESE TITLES):\n{notes_data}\n\nTASK: Create a Map of Content (MOC).\n1. Group the notes into logical categories using ## headers.\n2. STRICT FORMATTING: List notes as standard bullet points (e.g., - [[Exact Title]]). DO NOT use headers like ### for the individual note links.\n3. EXACT MATCH RULE: You MUST copy the exact string provided in the 'TITLE:' field wrapped in brackets. Do NOT shorten, guess, or summarize titles. If the title is 'Terraform Introductions', you must write [[Terraform Introductions]], NOT [[Terraform]].\n4. Add a brief 1-sentence description after each bullet point.\n5. Write a brief 'Overview' at the top.", + "_moc_generator_tip": "Placeholders: {topic}, {context_prompt}, {notes_data}. Recommended Temperature: 0.1 for high accuracy with note titles." + }, + + "auto_tagger": { + "prompt": "CONTENT:\n{content}\n\nEXISTING VAULT TAGS:\n{tags_hint}\n\nTASK:\nIdentify the 3-4 best tags for this note based on the content. Use existing tags if they are relevant, otherwise create new ones. \n\nRETURN ONLY the tags separated by commas. Do not write sentences or explanations. Output example: tag1, tag2, tag3", + "_tip": "Placeholders: {content}, {tags_hint}." } } diff --git a/study_recap.py b/study_recap.py index bdd46af..119401b 100644 --- a/study_recap.py +++ b/study_recap.py @@ -15,14 +15,15 @@ import datetime import re -from config import EXCLUDED_FOLDERS, MAX_FILE_SIZE, VAULT_PATH, HOURS_BACK, MAX_NOTE_CHARS, TEMPERATURES, RECAP_TITLE_FORMAT -from ai_backend import get_backend, call_ai, run_startup_checks +from config import EXCLUDED_FOLDERS, MAX_FILE_SIZE, VAULT_PATH, HOURS_BACK, MAX_NOTE_CHARS, TEMPERATURES, RECAP_TITLE_FORMAT, STUDY_DIR_NAME +from core.ai_backend import get_backend, call_ai, run_startup_checks SCRIPT_DIR = Path(__file__).parent sys.path.insert(0, str(SCRIPT_DIR)) VAULT_PATH = Path(VAULT_PATH).expanduser().resolve() -RECAP_FOLDER = VAULT_PATH / "Study Recaps" +RECAP_FOLDER = VAULT_PATH / STUDY_DIR_NAME +RECAP_FOLDER.mkdir(parents=True, exist_ok=True) # --- COLORS --- CYAN, GREEN, YELLOW, DIM, BOLD, RESET = "\033[96m", "\033[92m", "\033[93m", "\033[2m", "\033[1m", "\033[0m" diff --git a/txt_to_notes.py b/txt_to_notes.py index 8c558e3..348319a 100644 --- a/txt_to_notes.py +++ b/txt_to_notes.py @@ -24,15 +24,15 @@ import re import datetime from pathlib import Path - -from config import MAX_FILE_SIZE, VAULT_PATH, TEMPERATURES, TXT_TITLE_FORMAT -from ai_backend import get_backend, call_ai, run_startup_checks +from config import VAULT_PATH, TEMPERATURES, TXT_TITLE_FORMAT, CAPTURES_DIR_NAME +from core.ai_backend import get_backend, call_ai, run_startup_checks +from core.tagger_logic import collect_vault_tags, format_tag SCRIPT_DIR = Path(__file__).parent sys.path.insert(0, str(SCRIPT_DIR)) VAULT_PATH = Path(VAULT_PATH).expanduser().resolve() -OUTPUT_BASE = "Captures" +OUTPUT_BASE = CAPTURES_DIR_NAME PROMPTS_PATH = SCRIPT_DIR / "prompts.json" with PROMPTS_PATH.open(encoding="utf-8") as f: @@ -84,36 +84,6 @@ def parse_input(raw: str) -> tuple[str, str]: return match.group(1).strip(), raw[match.end():].strip() return "", raw.strip() - -def collect_vault_tags() -> list[str]: - """ - Scan all notes and collect existing tags from frontmatter and inline #hashtags. - Passed to the AI so it reuses existing tags instead of inventing new ones. - """ - tags = set() - - for path_obj in VAULT_PATH.rglob("*.md"): - try: - if path_obj.stat().st_size > MAX_FILE_SIZE: - continue - with path_obj.open("r", encoding="utf-8", errors="ignore") as f: - content = f.read() - - fm = re.match(r"^---\n(.*?)\n---", content, re.DOTALL) - if fm: - for line in fm.group(1).splitlines(): - m = re.match(r"\s*-\s*(.+)", line) - if m: - tags.add(m.group(1).strip().lower()) - - for t in re.findall(r"#([a-zA-Z][a-zA-Z0-9_/-]+)", content): - tags.add(t.lower()) - except Exception: - continue - cleaned_tags = {format_obsidian_tag(t) for t in tags if format_obsidian_tag(t)} - return sorted(list(cleaned_tags)) - - def collect_note_titles() -> list[str]: """ Collect all note titles for wikilink generation. @@ -165,7 +135,7 @@ def plan_notes(text, instructions, existing_tags, existing_titles, backend) -> l def write_note_content(text, note_plan, all_titles_in_batch, instructions, existing_tags, backend) -> str: """ - Pass 2: Write the full content for a single note. + Write the full content for a single note. The model knows all other notes in this batch so it can cross-link between them. """ tags_hint = ", ".join(existing_tags[:40]) if existing_tags else "none yet" @@ -191,7 +161,7 @@ def write_note_content(text, note_plan, all_titles_in_batch, instructions, exist def write_file(note_plan, body, folder_path, date_str) -> str: """Write a single note to disk with Obsidian frontmatter.""" title = note_plan.get("title", "Untitled") - tags = [format_obsidian_tag(t) for t in note_plan.get("tags", [])] + tags = [format_tag(t) for t in note_plan.get("tags", [])] safe_title = re.sub(r'[\\/*?:"<>|]', "", title) filename = TXT_TITLE_FORMAT.format(title=safe_title)