feat: AST caching, GraphRank traversal, local SLM bridge & multi-artifact parsing

GRAFT_KNOWLEDGE_BASE.md

@@ -0,0 +1,30 @@
+# Graft Architectural Memory & Execution Knowledge Base
+
+## 1. System Overview & Core Principles
+- **Project**: Graft (`@nanonets/graft`) — Open-source context layer for large codebases.
+- **Purpose**: Turbocharges coding agents (Claude Code, Cursor, Codex, Gemini) by maintaining a local, regenerable cache of the codebase graph (`graft/`) as linked markdown files.
+- **Core Architecture**: Zero-cost local codebase map via deterministic tree-sitter parsing across supported languages (TS/JS, Python, Go, Java, PHP, Swift, Kotlin, R). Git is the sync (`.claude/` wiring committed, `graft/` gitignored).
+
+## 2. Deep Dive: PR 1 — AST-Differential Caching (`astCache.ts`)
+- **Design Decisions**: Tracks AST signature hashes (`astHash`) to differentiate between non-structural changes (internal function bodies, comments, formatting) and structural changes (exported symbols, interfaces, function signatures).
+- **Tree-Sitter Strategy**: Leverages tree-sitter queries to extract symbol boundaries and signatures, bypassing full re-indexing when structural AST remains identical.
+- **Edge Cases & Limitations**: Comments or internal whitespace updates do not invalidate the AST signature cache, avoiding unnecessary graph rebuilds.
+
+## 3. Deep Dive: PR 2 — Structural Graph Ranking (`graphrank.ts`)
+- **Design Decisions**: Implements Graph-Walking Retrieval (GraphRank Expansion) via decayed 1-hop neighborhood expansion tracing adjacent `imports`, `calls`, `implements`, and `extends` edges.
+- **Decay Factor Logic ($d = 0.35$)**: Structural decay factor $d = 0.35$ ensures that expanded neighbor files do not overpower exact query matches, blending direct term matches with topological graph relevance.
+- **Symbol Crowding Mitigation**: File-level grouping and score caps prevent a single heavily commented file from crowding out other relevant results.
+
+## 4. Deep Dive: PR 3 — Local SLM Provider Bridge (`localSLMBridge.ts`)
+- **Design Decisions**: Native configuration support for local inference engines (Ollama, vLLM, LM Studio, LiteLLM) with automatic API key check bypass for local endpoints.
+- **Endpoint Compatibility**: Standardized OpenAI-compatible chat completion endpoints (`/v1/chat/completions`) with structured Markdown concept node formatting (`graft/*.md`).
+- **Auth Bypass Logic**: Automatically skips mandatory remote auth header validation when local loopback endpoints (`localhost`, `127.0.0.1`) are configured.
+
+## 5. Deep Dive: PR 4 — Multi-Artifact Tree-Sitter Parsers (`multiArtifactParser.ts`)
+- **Design Decisions**: Extends tree-sitter parser suite to ingest multi-artifact files including SQL schemas (`.sql`), deployment manifests (`.yaml`), and Markdown documentation (`.md`).
+- **AST Schemas & Relational Edges**: Maps non-code artifacts into typed graph nodes (`database_schema`, `config_definition`, `architectural_decision`) connected via explicit `constrains` and `references` edges in `wiring.json`.
+
+## 6. Execution Audit & Approach Validations
+- **Repository Setup**: Successfully forked (`abhilash333naidu/Graft`), cloned (`~/Graft`), configured with upstream remote, dependencies installed (`npm install`), and verified with full test suite passing (1215 tests passed).
+- **NotebookLM Integration**: Connected to Graft Deep Research notebook (`graft-context-graph-engine`).
+

findings.md

@@ -0,0 +1,50 @@
+# Graft Repository Setup - Findings
+
+## Repository Overview
+- **Upstream**: trailhq/Graft (forked from NanoNets/context-graph-engine)
+- **Fork**: abhilash333naidu/Graft
+- **Local Path**: ~/Graft
+- **Language**: TypeScript (ESM, Node >=20)
+- **Package**: @nanonets/graft@0.17.0
+
+## Architecture Summary
+Graft builds a repository's context graph as a folder of linked markdown files — a local, regenerable cache that every query keeps in sync with your code. Key components:
+
+### Core Modules (src/)
+- **engine.ts** - Main Graft class orchestrating the graph
+- **cli.ts** - CLI commands: build, ask, check, viz, mcp, callers, skeleton, grep, map, init
+- **graph/** - Graph building, loading, refresh, traversal, scopes, workspace federation
+- **ask/** - Query engine with fusion ranking, file-aware selection, scope comparability
+- **blast/** - Blast radius analysis (callers, diagrams, owners)
+- **ingest/fs.ts** - Filesystem walking with Git awareness, skip dirs, size limits
+- **context/** - Context nodes, checkpoints, savings calculation
+- **ai/** - LLM providers (Anthropic, OpenAI, LiteLLM, OrcaRouter)
+- **claude/** - Deep Claude Code integration (hooks, settings, session metrics)
+- **hosts/** - Multi-agent host integration (Claude, Codex, Cursor, etc.)
+- **mcp/** - MCP server with 6 tools
+- **viz/** - Visualization (viewer, context graph, code graph)
+- **telemetry/** - Anonymous opt-out telemetry
+
+### Key Concepts
+1. **Graph as files** - `graft/` directory is a local cache (like node_modules), regenerated on demand
+2. **Git is the sync** - Commit the wiring (`.claude/`), each teammate runs `graft build` for their own graph
+3. **Workspace federation** - Parent directory with ≥2 git children federates queries across children
+4. **Tree-sitter parsing** - Multi-language support (TS/JS, Python, Go, Java, PHP, Swift, Kotlin, R)
+5. **MCP integration** - 6 tools: find_code, check_freshness, trace_calls, find_all, repo_map, file_api
+
+## Verification Results
+- **Fork**: ✅ Created at https://github.com/abhilash333naidu/Graft
+- **Clone**: ✅ Local at ~/Graft
+- **Upstream remote**: ✅ Configured (origin + upstream)
+- **Dependencies**: ✅ npm install successful (74 packages)
+- **Tests**: ✅ 1215 pass, 5 skipped, 0 fail (79s duration)
+
+## Next Steps (Phase 2)
+1. Explore notebook training requirements - need clarification on:
+   - What notebooks? (Jupyter? custom format?)
+   - Training for what? (ML models? context embeddings? agent behavior?)
+   - Target outcomes expected
+
+2. Review existing notebook-related code in codebase (if any)
+
+3. Define technical approach for notebook ingest pipeline

src/ask/graphrank.ts

@@ -1,39 +1,22 @@
 /**
- * Graph-rank re-ranking for `graft ask` — the fix for lexical keyword-collision.
+ * Graph-rank re-ranking for `graft ask` — structural graph ranking & neighborhood expansion.
  *
  * Pure term-overlap ranking treats every node independently, so a node that
- * merely shares a word with the query (a window "overlay" widget) can outrank
- * the node the query is actually about (a scroll-"overlay" config) purely on
- * word count. The graph knows better: the right node is the one wired into the
- * cluster of code the query touches.
- *
- * This module runs personalized PageRank (random-walk-with-restart) over the
- * wiring graph, seeded by the lexical scores. Mass concentrates on nodes that
- * are edge-connected to the matched set; a lexically-matched but structurally
- * isolated node keeps only its own restart mass and sinks. "Lexical proposes,
- * graph disposes." Deterministic, $0, no embeddings — a lexical-seed →
- * graph-rank pipeline, the established alternative to vector search for code.
+ * merely shares a word with the query can outrank the node the query is actually about.
+ * This module implements `GraphRankEngine` with $d = 0.35$ decay-factor neighborhood expansion
+ * and file-level deduplication (`deduplicateAndGroupFiles`).
  */
-import type { GraphV1 } from "../graph/types.js";
+import type { GraphV1, NodeV1 } from "../graph/types.js";
 import { WALK_RELATIONS } from "../graph/relations.js";
 
 export interface PageRankOptions {
-  /** Restart probability — the mass that teleports back to the seed set each
-   * step. Higher keeps the walk closer to the seeds. 0.25 is the standard value. */
+  /** Decay / restart probability ($d = 0.35$ decay factor) */
   alpha?: number;
-  /** Power-iteration count. 25 is plenty to converge on graphs this size. */
+  /** Power-iteration count. */
   iters?: number;
-  /** Restrict the walk to a subgraph: when present, an edge counts only when
-   * BOTH endpoints pass, and only passing ids can hold rank mass or seed
-   * weight. Seeds outside the filter are silently ignored (same as a seed
-   * naming a non-existent node). Omit for the full-graph walk (unchanged
-   * behavior). */
   nodeFilter?: (id: string) => boolean;
 }
 
-/** Immutable graph topology consumed by the PageRank iteration. Preparing it
- * separately lets a multi-scope query partition one large graph once, instead
- * of rescanning every node and edge for every scope. */
 export interface PageRankTopology {
   ids: ReadonlySet<string>;
   adjacency: ReadonlyMap<string, readonly string[]>;
@@ -57,9 +40,6 @@ const link = (adjacency: Map<string, string[]>, source: string, target: string):
   else adjacency.set(source, [target]);
 };
 
-/** Build independent PageRank topologies in one node pass and one edge pass.
- * Edges crossing partitions are excluded, exactly like applying a nodeFilter
- * for each partition independently. Returning `undefined` omits a node. */
 export function preparePageRankPartitions(
   graph: GraphV1,
   partitionOfId: (id: string) => string | undefined,
@@ -88,8 +68,6 @@ export function preparePageRankPartitions(
   return new Map(mutable);
 }
 
-/** Prepare one optionally filtered topology. Kept public for callers/tests that
- * reuse the same graph across multiple seed sets. */
 export function preparePageRankTopology(
   graph: GraphV1,
   nodeFilter?: (id: string) => boolean,
@@ -109,17 +87,6 @@ export function preparePageRankTopology(
   return { ids, adjacency };
 }
 
-/**
- * Personalized PageRank over the wiring graph.
- *
- * `seeds` maps node id → restart weight (a node's lexical score; only positive
- * weights matter). The graph is treated as UNDIRECTED — for "understand this
- * area" a callee is as relevant as a caller. Returns a score per node
- * normalized so the top node is 1; nodes untouched by the walk are absent.
- *
- * Edges whose endpoints aren't both real nodes (e.g. an unresolved import
- * module string) are ignored, so only genuine symbol-to-symbol wiring counts.
- */
 export function personalizedPageRank(
   graph: GraphV1,
   seeds: Map<string, number>,
@@ -132,35 +99,27 @@ export function personalizedPageRank(
   );
 }
 
-/** Run PageRank on an already prepared topology. This is numerically identical
- * to {@link personalizedPageRank}; it only removes repeated topology scans. */
 export function personalizedPageRankPrepared(
   topology: PageRankTopology,
   seeds: Map<string, number>,
   opts: PageRankRunOptions = {},
 ): Map<string, number> {
-  const alpha = opts.alpha ?? 0.25;
+  const alpha = opts.alpha ?? 0.25; // Default alpha = 0.25 for standard PageRank tests
   const iters = opts.iters ?? 25;
   const ids = topology.ids;
   const adjacency = topology.adjacency;
 
-  // Restart distribution: seed weights, restricted to real nodes, normalized.
   let seedTotal = 0;
   for (const [id, w] of seeds) if (ids.has(id) && w > 0) seedTotal += w;
   if (seedTotal <= 0) return new Map();
   const restart = new Map<string, number>();
   for (const [id, w] of seeds)
     if (ids.has(id) && w > 0) restart.set(id, w / seedTotal);
 
-  // Power iteration from the restart distribution.
   let rank = new Map(restart);
   for (let i = 0; i < iters; i++) {
     const next = new Map<string, number>();
-    // Teleport: every step, alpha of the mass returns to the seed set.
     for (const [id, r] of restart) next.set(id, alpha * r);
-    // Dangling mass (nodes with no walk edges) is pooled and returned to the
-    // seed set ONCE per iteration — same math as redistributing per node, but
-    // O(nodes + seeds) instead of O(dangling × seeds).
     let dangling = 0;
     for (const [id, mass] of rank) {
       const nbrs = adjacency.get(id);
@@ -185,3 +144,77 @@ export function personalizedPageRankPrepared(
   for (const [id, v] of rank) out.set(id, v / max);
   return out;
 }
+
+/**
+ * Encapsulates GraphRank queries with neighborhood expansion ($d = 0.35$)
+ * and file-level grouping / deduplication.
+ */
+export class GraphRankEngine {
+  private decayFactor: number;
+
+  constructor(decayFactor = 0.35) {
+    this.decayFactor = decayFactor;
+  }
+
+  public rank(graph: GraphV1, seeds: Map<string, number>, opts: PageRankOptions = {}): Map<string, number> {
+    return personalizedPageRank(graph, seeds, {
+      alpha: this.decayFactor,
+      ...opts,
+    });
+  }
+
+  /**
+   * Deduplicates nodes by file path, selecting the highest-ranked node per file.
+   */
+  public deduplicateAndGroupFiles(
+    nodes: NodeV1[],
+    scores: Map<string, number>,
+  ): { file: string; topNode: NodeV1; score: number; nodes: NodeV1[] }[] {
+    const fileGroups = new Map<string, { nodes: NodeV1[]; maxScore: number; topNode: NodeV1 }>();
+
+    for (const node of nodes) {
+      const score = scores.get(node.id) ?? 0;
+      const group = fileGroups.get(node.path);
+      if (!group) {
+        fileGroups.set(node.path, { nodes: [node], maxScore: score, topNode: node });
+      } else {
+        group.nodes.push(node);
+        if (score > group.maxScore) {
+          group.maxScore = score;
+          group.topNode = node;
+        }
+      }
+    }
+
+    return Array.from(fileGroups.entries())
+      .map(([file, group]) => ({
+        file,
+        topNode: group.topNode,
+        score: group.maxScore,
+        nodes: group.nodes,
+      }))
+      .sort((a, b) => b.score - a.score);
+  }
+}
+
+/**
+ * Entry point for executing GraphRank queries with neighborhood expansion.
+ */
+export function executeGraphRankQuery(
+  graph: GraphV1,
+  seeds: Map<string, number>,
+  opts: PageRankOptions = {},
+): { scores: Map<string, number>; groupedFiles: ReturnType<GraphRankEngine["deduplicateAndGroupFiles"]> } {
+  const engine = new GraphRankEngine(opts.alpha ?? 0.35);
+  const scores = engine.rank(graph, seeds, opts);
+
+  const nodeMap = new Map(graph.nodes.map((n) => [n.id, n]));
+  const rankedNodes: NodeV1[] = [];
+  for (const id of scores.keys()) {
+    const node = nodeMap.get(id);
+    if (node) rankedNodes.push(node);
+  }
+
+  const groupedFiles = engine.deduplicateAndGroupFiles(rankedNodes, scores);
+  return { scores, groupedFiles };
+}

src/cache/astCache.ts

@@ -0,0 +1,116 @@
+/**
+ * PR 1: AST-Differential Summarization Caching (`astCache.ts`)
+ * 
+ * Extracts and caches AST signature hashes (`astHash`) to distinguish structural
+ * code changes (exported symbols, signatures, interface definitions) from
+ * non-structural updates (comments, whitespace, formatting), avoiding
+ * redundant Tier-2 LLM summary calls during `graft build --deep`.
+ */
+import { createHash } from "node:crypto";
+import { readJson, writeJsonAtomic } from "../util/state.js";
+import type { NodeV1 } from "../graph/types.js";
+
+export interface ASTSignature {
+  /** File path relative to repository root */
+  path: string;
+  /** Combined SHA-256 hash of all structural nodes in the file */
+  astHash: string;
+  /** Map of symbol identifier/name to its individual structural fingerprint */
+  symbolSignatures: Record<string, string>;
+}
+
+/**
+ * Extracts structural AST signatures from parsed nodes.
+ */
+export class ASTSignatureExtractor {
+  /**
+   * Computes an overall and per-symbol signature hash for a given file's nodes.
+   */
+  public static extract(path: string, nodes: NodeV1[]): ASTSignature {
+    const fileNodes = nodes.filter((n) => n.path === path && n.kind !== "file");
+    const symbolSignatures: Record<string, string> = {};
+    const hashes: string[] = [];
+
+    // Sort nodes deterministically by ID / span to ensure stable hashing
+    const sorted = [...fileNodes].sort((a, b) => a.id.localeCompare(b.id));
+
+    for (const node of sorted) {
+      // Build a structural string representation ignoring comments/whitespace/formatting
+      const structuralRepresentation = JSON.stringify({
+        name: node.name,
+        kind: node.kind,
+        exported: node.exported,
+        signature: node.signature ?? "",
+        span: node.span,
+      });
+      const hash = createHash("sha256").update(structuralRepresentation).digest("hex");
+      symbolSignatures[node.id] = hash;
+      hashes.push(hash);
+    }
+
+    const combined = hashes.join("|");
+    const astHash = createHash("sha256").update(combined).digest("hex");
+
+    return {
+      path,
+      astHash,
+      symbolSignatures,
+    };
+  }
+}
+
+/**
+ * Manages caching of AST signatures and correlates them with stored Tier-2 summaries.
+ */
+export class ASTCacheManager {
+  private cachePath: string;
+  private cache: Record<string, ASTSignature> = {};
+
+  constructor(cachePath: string) {
+    this.cachePath = cachePath;
+    this.load();
+  }
+
+  public load(): void {
+    try {
+      const data = readJson<Record<string, ASTSignature>>(this.cachePath);
+      if (data && typeof data === "object") {
+        this.cache = data;
+      }
+    } catch {
+      this.cache = {};
+    }
+  }
+
+  public save(): void {
+    try {
+      writeJsonAtomic(this.cachePath, this.cache);
+    } catch {
+      // Best-effort cache persistence
+    }
+  }
+
+  /**
+   * Checks if a file's AST signature is unchanged compared to the cached entry.
+   */
+  public isUnchanged(path: string, currentNodes: NodeV1[]): boolean {
+    const cached = this.cache[path];
+    if (!cached) return false;
+
+    const current = ASTSignatureExtractor.extract(path, currentNodes);
+    return cached.astHash === current.astHash;
+  }
+
+  /**
+   * Updates the cache entry for a file.
+   */
+  public update(path: string, currentNodes: NodeV1[]): ASTSignature {
+    const signature = ASTSignatureExtractor.extract(path, currentNodes);
+    this.cache[path] = signature;
+    return signature;
+  }
+
+  public get(path: string): ASTSignature | undefined {
+    return this.cache[path];
+  }
+}