Skip to content

internals.md

Latest commit

 

History

History
82 lines (68 loc) · 3.95 KB

internals.md

File metadata and controls

82 lines (68 loc) · 3.95 KB
Raw

How repolens works

Indexing pipeline

walk → chunk (AST) → contextualize → embed → LanceDB (+ BM25 FTS)
        ↑ manifest      ↑ chat provider   ↑ embeddings provider
  1. Walk (walk.py, pure stdlib): discovers files per the manifest — extension allow-list, directory/glob excludes, per-repo excludes, size cap, binary sniff.
  2. Chunk (chunk_ast.py): tree-sitter isolates top-level definitions (functions, classes, methods…) per language (PHP/Python/TS/JS); comments and decorators are absorbed into their definition; inter-definition "glue" is grouped. Unsupported languages fall back to a character window with overlap (chunk.py). The docs corpus chunks per entry instead: YAML top-level keys, Markdown sections.
  3. Contextualize (context.py, "contextual retrieval"): each chunk gets a one-sentence description of its role in its file, prefixed to the text before embedding. One chat call per file (not per chunk) keeps the cost tiny; a deterministic structural fallback ([repo X | file Y | in <symbol> | lines a-b/n]) covers gaps, failures, and --dry-run. Mode struct gives you the free deterministic variant; off disables prefixes.
  4. Embed (embed.py + a provider): batched under both an input-count and a character budget, throttled globally, retried on 429 with backoff.
  5. Store (store.py, LanceDB): rows carry the raw chunk (text), the contextualized text (embedded + BM25-indexed), line numbers, sha256, the strategy version (embed_ver), and the sidecar metadata (source forge, last commit date, status, permalink).

Incrementality: a file is re-embedded only when its sha256 or the strategy version changed; files gone from disk are removed. The FTS index is rebuilt after any change (deletions included) to stay consistent and mono-fragment.

Query pipeline

question → rewrite (chat) → embed → vector + BM25 → RRF → [LLM rerank] → meta rerank → top-k
  • Rewrite: a chat call expands acronyms (yours, via prompts.domain_hints) and adds plausible symbol names. Failure = fall back to the original question.
  • Hybrid retrieval: vector similarity and BM25 full-text run together, fused by Reciprocal Rank Fusion; oversampled (k×3) when a rerank follows.
  • Meta rerank: a bounded nudge on top of relevance — dead code sinks hard, doubtful/stale (>3 y) code drops a few places, active/recent (<1 y) code rises a little. Relevance stays dominant; the nudge only breaks near-ties.

Call graph

Pure AST, no API (graph.py). Definitions and call sites are extracted per file, then calls are resolved through a confidence cascade:

Resolution Confidence
$this->m() / self:: / parent:: via class hierarchy 0.95
Class::m() / new Class() via use/import/namespace 0.90
free function, exact import match 0.95
free function, same namespace 0.90
unique name in the project 0.75
ambiguous name 0.35

Edges ≥ 0.70 are never pruned; low-confidence name-based edges are capped by fan-out. Each node carries fan-in and PageRank (that's hotspots). Queries bucket results into certain / probable / uncertain tiers so a consumer (or an agent) can qualify its claims. The artifact is versioned gzipped JSON (graph-v2); load_graph rebuilds the reverse adjacency at load time.

Artifacts & compatibility

out_dir/
├── lancedb/         # tables code_chunks, docs_chunks
├── graph.json.gz    # {version, nodes, out, by_fqn}
└── meta.json        # {repo_source, repo_ref, web_base, last_commit, status}

Artifacts are the interface between the build side and the query side: build where your clones and git history live, ship the directory (atomically), query anywhere — including hosts that never see the source code. Schema/versioning constants are frozen and test-guarded; an index built by the ancestor of this library (Infoclimat's code_index) reads unchanged.