feat(graph): provision tree-sitter into embed-deps so the auto-build hook works#323
Open
efenocchi wants to merge 1 commit into
Open
feat(graph): provision tree-sitter into embed-deps so the auto-build hook works#323efenocchi wants to merge 1 commit into
efenocchi wants to merge 1 commit into
Conversation
…hook works The code-graph auto-build hook (graph-on-stop, on codex/cursor/hermes) resolves the native tree-sitter module at runtime from the shared ~/.hivemind/embed-deps/node_modules that each agent's plugin symlinks to. But `ensureSharedDeps` only installs @huggingface/transformers there, so on a real install tree-sitter is absent and the graph never auto-builds — the hook just degrades to a graceful skip (the CLI `hivemind graph build` works because it resolves tree-sitter from its own package). This provisions the parsers where the hook actually looks. tree-sitter is a native module: symlinking the package dir alone is not enough (it needs its own node-gyp-build / node-addon-api siblings resolved), so we run a scoped npm install and let npm lay down the full tree. - src/cli/embeddings.ts: new `ensureGraphDeps()` — idempotent (specs-hash marker + per-package presence check catch partial installs and version bumps) and best-effort (any failure is logged and swallowed, never aborts the caller). Two-phase native provisioning: `npm install --ignore-scripts` so a platform without a prebuild (arm64 / Node 24) doesn't fail the download, then scripts/ensure-tree-sitter.mjs ALWAYS runs to validate bindings + compile from source where needed (fast no-op on healthy prebuilds; also repairs an interrupted / ABI-mismatched install). Versions are read from the package's own optionalDependencies (`treeSitterSpecs`) so embed-deps never drifts from what the bundles were built against. Decoupled from the ~600 MB embeddings download — installs only the parsers (tens of MB). - src/cli/embeddings.ts: `ensureSharedDeps` now MERGES the shared package.json instead of overwriting it, so a prior graph-deps declaration survives a first-time embeddings install. - Wired into `installEmbeddings()` and `hivemind graph init` (best-effort). - tests: treeSitterSpecs (version derivation, no drift, excludes transformers) + isGraphDepsInstalled (all-parsers-present / partial-install). Verified e2e: `graph init` provisions tree-sitter into a sandbox embed-deps (12 pkgs), a second run skips the reinstall (idempotent), and the codex graph-on-stop hook then builds the graph (Nodes: 5, Edges: 2). Full suite 5546 passed. Codex review: both prior HIGH findings resolved. Known limitation (pre-existing, unchanged): npm is invoked as bare `npm`, which does not resolve on Windows — same as the existing ensureSharedDeps invocation; a proper cross-platform npm exec is out of scope here.
📝 WalkthroughWalkthroughThe CLI now provisions tree-sitter parser dependencies in shared embedding dependencies, heals native bindings, preserves merged dependency declarations, and invokes provisioning during embeddings installation and graph initialization. New tests cover parser specification extraction and installation detection. ChangesGraph dependency provisioning
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant CLI
participant ensureGraphDeps
participant SharedDependencies
participant npm
participant NativeHealing
CLI->>ensureGraphDeps: provision graph dependencies
ensureGraphDeps->>SharedDependencies: read and update shared package metadata
ensureGraphDeps->>npm: install tree-sitter specs with --ignore-scripts
npm-->>SharedDependencies: write parser packages
ensureGraphDeps->>NativeHealing: heal native bindings
NativeHealing-->>CLI: complete best-effort provisioning
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| cwd: SHARED_DIR, | ||
| stdio: "inherit", | ||
| }); | ||
| writeFileSync(marker, wantKey); |
Contributor
Coverage ReportScope: files changed in this PR. Enforced threshold: 90% per metric (per file via
File Coverage — 2 files changed
Generated for commit f50603a. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/cli/embeddings.ts`:
- Around line 215-225: The shared-dependency installation in the embeddings
install flow must disable package lifecycle scripts. Update the npm install
invocation associated with the package.json written after `ensureGraphDeps` to
include the `--ignore-scripts` option, preserving the existing dependency merge
and installation behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: cfcbd462-e011-4233-b06e-6e5f4421c4ab
📒 Files selected for processing (3)
src/cli/embeddings.tssrc/cli/index.tstests/shared/graph-deps.test.ts
Comment on lines
+215
to
+225
| // Merge, don't clobber: ensureGraphDeps may have already declared the | ||
| // tree-sitter parsers here. Overwriting with transformers-only would drop | ||
| // that declaration and let a subsequent reconcile prune the parsers. | ||
| const pkgPath = join(SHARED_DIR, "package.json"); | ||
| let existing: { dependencies?: Record<string, string> } = {}; | ||
| try { existing = JSON.parse(readFileSync(pkgPath, "utf8")); } catch { /* none yet */ } | ||
| writeJson(pkgPath, { | ||
| name: "hivemind-embed-deps", | ||
| version: "1.0.0", | ||
| private: true, | ||
| dependencies: { [TRANSFORMERS_PKG]: TRANSFORMERS_RANGE }, | ||
| dependencies: { ...(existing.dependencies ?? {}), [TRANSFORMERS_PKG]: TRANSFORMERS_RANGE }, |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
git ls-files src/cli/embeddings.ts
printf '\n--- outline ---\n'
ast-grep outline src/cli/embeddings.ts --view expanded
printf '\n--- relevant search ---\n'
rg -n --no-heading "ensureGraphDeps|ensureSharedDeps|installEmbeddings|ignore-scripts|npm install|writeJson|package.json" src/cli/embeddings.ts
printf '\n--- file slice 1 ---\n'
sed -n '1,320p' src/cli/embeddings.tsRepository: activeloopai/hivemind
Length of output: 18495
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '180,270p' src/cli/embeddings.ts
printf '\n--- search related install steps ---\n'
rg -n --no-heading "ensureGraphDeps|ensureSharedDeps|installEmbeddings|ignore-scripts|tree-sitter|parser" src/cli/embeddings.ts src/cli -g '!**/dist/**'Repository: activeloopai/hivemind
Length of output: 11864
Keep lifecycle scripts disabled for the shared-dependency install.
If shared/package.json already includes tree-sitter parsers, this npm install can run their native lifecycle scripts before ensureGraphDeps() reaches the heal step, so embeddings install can still fail on the platforms this flow is meant to handle. Add --ignore-scripts here.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/cli/embeddings.ts` around lines 215 - 225, The shared-dependency
installation in the embeddings install flow must disable package lifecycle
scripts. Update the npm install invocation associated with the package.json
written after `ensureGraphDeps` to include the `--ignore-scripts` option,
preserving the existing dependency merge and installation behavior.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The code-graph auto-build hook (
graph-on-stop, on codex/cursor/hermes) resolves the nativetree-sittermodule at runtime from the shared~/.hivemind/embed-deps/node_modulesthat each agent's plugin symlinks to. ButensureSharedDepsonly installs@huggingface/transformersthere — tree-sitter is never provisioned into embed-deps, so on a real install the graph never auto-builds. The hook just degrades to a graceful skip (thanks to the crash fix in #321).Note: the CLI
hivemind graph build/graph initdoes work in prod (verified end-to-end via a cleannpm packinstall) because it resolves tree-sitter from its own package. This PR fixes the hook path.Fix
ensureGraphDeps()(src/cli/embeddings.ts) — provisions the tree-sitter parsers into embed-deps where the hook looks. tree-sitter is a native module (needs itsnode-gyp-build/node-addon-apisiblings resolved), so a scopednpm installlays down the full tree rather than symlinking a single dir..graph-depsspecs-hash marker + a per-package presence check skip the re-download and catch partial installs / version bumps.npm install --ignore-scripts(so a platform without a prebuild — arm64 / Node 24 — doesn't fail the download), thenscripts/ensure-tree-sitter.mjsalways runs to validate bindings + compile from source where needed (fast no-op on healthy prebuilds; also repairs an interrupted / ABI-mismatched install).optionalDependencies.ensureSharedDepsnow MERGES the shared package.json instead of overwriting it, so a prior graph-deps declaration survives a first-time embeddings install.installEmbeddings()andhivemind graph init(best-effort).Verification
End-to-end in an isolated sandbox (
HOMEoverride):hivemind graph init→ensureGraphDepsinstalls 12 packages into the sandbox embed-deps; all 10 parsers present + marker written.graph init→already present(idempotent, no reinstall).graph-on-stophook, with embed-deps now populated, builds the graph:Nodes: 5, Edges: 2, snapshot written, exit 0.Known limitation (pre-existing, unchanged)
npm is invoked as bare
npm, which does not resolve on Windows — identical to the existingensureSharedDepsinvocation. A proper cross-platform npm exec is out of scope here (it affects both paths).Session Context
Session transcript
Summary by CodeRabbit
New Features
Bug Fixes