Skip to content

INGESTION.md

Latest commit

 

History

History
61 lines (41 loc) · 2.9 KB

INGESTION.md

File metadata and controls

61 lines (41 loc) · 2.9 KB
Raw

📚 Ingestion Guide

← Back to README · See also How It Works · Configuration

For more control than the README Quick Start provides. Power user mode activated 🔋

Resume Formats

PDF (via pypdf), DOCX (via python-docx), Markdown, and plain text. Sonnet extracts name, roles, companies, and skills. If an Engineer node already exists in the graph, re-ingestion reuses it.

Repo Sources

# Explicit repos (GitHub URLs or local paths)
--repos https://github.com/user/repo1 /path/to/local/repo

# All repos for a GitHub user (paginates automatically, skips forks)
--github-user username

For private repos, set GITHUB_TOKEN in .env. Private repos are ingested identically to public ones — the private flag on the Repository node controls whether raw code appears in query responses (governed by SHOW_PRIVATE_CODE).

🗣️ Supported Languages

Language Parser Extracts
🐍 Python tree-sitter-python Functions, classes
🟨 JavaScript tree-sitter-javascript Functions, classes
🔷 TypeScript tree-sitter-typescript Functions, classes
⚛️ TSX tree-sitter-tsx Functions, classes
📓 Jupyter Notebooks JSON + tree-sitter-python Code cells → functions, classes (or per-cell fallback)
🌐 Other (.java, .go, .rs, .rb, .cpp, .c, .h) Fallback Double-newline blocks

🔄 Re-embedding

If you change embedding providers, add repos, or want to regenerate context descriptions:

uv run python scripts/reembed.py                        # auto-detects providers from .env
uv run python scripts/reembed.py --providers voyage      # just Voyage
uv run python scripts/reembed.py --providers nim voyage   # explicit both

This runs in two phases:

  1. Phase 1 — Generate missing Sonnet context descriptions (the context field on CodeSnippet nodes)
  2. Phase 2 — Embed all snippets that have context, in parallel across providers

The pipeline is idempotent — it skips snippets that already have embeddings for the target provider. Safe to re-run anytime. Smash that button 🔘

🏛️ Architecture Summaries

Each repository can have a pre-seeded architecture summary with a mermaid diagram, stored on the Repository node's architecture property. When a user asks "How did you build X?", the agent retrieves this summary verbatim rather than asking Haiku to generate a diagram on the fly (which is unreliable).

To add an architecture summary for a repo:

MATCH (r:Repository {name: "MyRepo"})
SET r.architecture = "## MyRepo\n\nDescription...\n\n```mermaid\nflowchart LR\n    A --> B\n```\n\n### Key Design Decisions\n..."

These are best written by a strong reasoning model (Opus/Sonnet) that can read the actual codebase and produce accurate diagrams. The live demo's summaries were generated by Opus in a single session 🎯