Skip to content

architecture.md

Latest commit

 

History

History
125 lines (94 loc) · 6.19 KB

architecture.md

File metadata and controls

125 lines (94 loc) · 6.19 KB
Raw

Architecture

Code Context Engine is a local repository intelligence layer for AI coding agents.

repository
  -> discovery and content hashing
  -> language-provider Tree-sitter chunks
  -> SQLite storage
  -> Tantivy + in-memory BM25 search
  -> symbol graph with reference provider evidence
  -> context expansion and impact analysis
  -> CLI and MCP stdio tools

Core Ideas

Metadata First

Search and context-pack tools return compact metadata:

  • chunk_id
  • file path
  • line span
  • symbol name
  • chunk kind
  • score or expansion reason
  • ranking signals
  • estimated metadata token cost

Full source is returned only by get_chunk.

Stable Internal Model

The important model objects are:

  • CodeChunk
  • SymbolRecord
  • GraphEdge
  • SearchResult
  • ImpactReport

Internal search and graph implementations can change as long as these contracts remain stable.

Reference Providers

Graph edges carry a provider, confidence, and human-readable evidence. The built-in provider is heuristic; it is useful for impact hints and context expansion, but it is not a compiler-grade reference engine.

The indexer exposes registered language-provider descriptors for parser/plugin discovery. The graph layer exposes registered reference-provider descriptors and has a matching language-reference semantics boundary used by the heuristic provider:

  • Rust keeps lightweight use import edges.
  • Rust/Java/TypeScript/Python syntactic calls are stored as calls graph edges when the target can be inferred by the heuristic provider.
  • Java records imports, extends/implements relationships, JUnit-style test methods, and receiver-style calls such as new Worker().work().
  • TypeScript/TSX records named imports, import aliases, class implements, and receiver-style method references/calls such as new Service().run().
  • Python records import / from ... import ... bindings, subclass relationships, and constructor receiver references/calls such as Worker().run().

Local import aliases can resolve to indexed files in the same repository, which lets find_references report a caller using run as execute as a reference to the exported run symbol. This is still syntactic evidence, not full module or type resolution.

context-mcp also supports configured external reference providers. These providers are dynamically loaded as external commands during index_repo: the engine sends repo/chunk/symbol metadata over stdin and expects graph edges on stdout. Returned endpoints are resolved against existing SymbolRecord anchors before storage, so external providers can add compiler/type-checker-grade references and calls edges without owning chunking or persistence. Provider subprocess I/O is handled concurrently so large stdout/stderr output or providers that do not read stdin are bounded by the configured timeout rather than deadlocking the indexer. On Windows, provider commands are attached to an unnamed Job Object when available; timeout cleanup terminates the Job Object so child processes from command shims, Node, or language servers are cleaned up with the provider.

Provider labels remain part of the edge identity. This lets a rust-analyzer/SCIP provider, a TypeScript type-checker provider, and a Python type-checker provider coexist with the heuristic provider for the same symbol pair. Strict providers fail closed on ambiguous or unknown endpoints; non-strict providers skip only the unresolved edges.

context-cli ships first-party protocol adapters for the common compiler-grade path: rust-analyzer/SCIP for Rust, scip-java for Java, the stable TypeScript compiler/language-service API for TypeScript/TSX, and Pyright LSP references for Python. They are still configured as external commands, so deployments can pin tool paths and timeouts without changing the engine. The TypeScript adapter checks the stable compiler API surface before resolving references so toolchain drift fails with explicit missing-capability diagnostics.

Heuristic Impact

Impact analysis is not a sound compiler-grade call graph. It uses indexed symbols, references, imports, and test associations to produce likely impact hints and recommended verification. Impact item reasons include the edge provider label, so a report can distinguish built-in heuristic evidence from external compiler/type-checker evidence.

Crate Layout

context-core       shared types
context-indexer    file discovery and language-provider chunking
context-storage    SQLite persistence
context-search     in-memory BM25, persistent Tantivy, local hash vectors, RRF
context-graph      reference providers and impact hints
context-mcp        MCP stdio server and agent-facing tool contracts
context-cli        CLI wrapper

Index Lifecycle

The SQLite database is the source of truth for indexed files. A normal index run compares discovered file hashes, skips unchanged files, replaces changed files, and removes files that no longer exist. --force clears and rebuilds only the selected repository.

SQLite index updates run inside a BEGIN IMMEDIATE transaction so a failed indexing pass does not leave a partially written logical index. After commit, the persistent Tantivy directory is rebuilt from SQLite using a temporary directory and then published by rename. Search can also rebuild Tantivy lazily if the sidecar index is missing or unreadable.

Repository IDs use the canonical repository path plus a stable hash, so two repositories with the same directory basename can share one SQLite database without colliding. Heuristic graph construction is partitioned by repository, and stored graph reads discard any legacy edge whose endpoints resolve to different repositories.

Search Lifecycle

search_code builds an in-memory BM25 index from SQLite for deterministic correctness and also queries the persistent Tantivy sidecar when the database is file-backed. When requested, it adds the local deterministic hash-vector signal. Results are fused with reciprocal-rank fusion and preserve their signal labels, for example bm25, tantivy_bm25, and local_hash_vector.

Planned Architecture Extensions

  • Additional language parsers and compiler/type-checker-backed reference providers beyond the current Rust, Java, TypeScript/TSX, and Python paths.