feat(search): fs.search 文件名搜索优先使用 fd/rg,缺失回退 JS 遍历 (fixes #203) - #303
feat(search): fs.search 文件名搜索优先使用 fd/rg,缺失回退 JS 遍历 (fixes #203)#303yanzhaohui1999 wants to merge 17 commits into
Conversation
|
我其实主要担心fd / rg在windows上有什么奇奇怪怪的格式问题)所以当时写的时候直接用的node原生的 |
… test-pin them Review concern (PR omdsh-dev#303): fd/rg output formats on Windows. rg emits backslash-separated paths with dot-backslash prefixes and CRLF line endings; the normalizer only handled the platform sep case, which Linux CI could not exercise with Windows-shaped lines, and the trailing CR was never explicitly stripped (relied on readline behavior). Make the separator injectable (defaults to the platform separator, behavior unchanged) and strip a trailing CR, then pin both Windows shapes in unit tests.
|
Windows 的坑主要在输出格式不稳定:同一个 rg,cmd 下输出 `,Git Bash 下输出 / Windows 真机我会跑一轮验证,完了把结果补回 PR;另外要不要顺便在 CI 加一个 windows runner 跑这两组搜索测试?仓库现在是纯 ubuntu,Windows 格式这类问题只有真机/CI 才兜得住,加的话我可以一起带上。 |
|
加一下? |
|
Windows 真机验证补完了,结果都写在设计文档里。贴几个关键点: 你担心的 Windows 格式问题是真实的:Windows 上 rg 默认输出 Windows 上的 DSH 自带 rg 探测也修了:npm 全局装在 %APPDATA%\npm\node_modules(没有 lib/ 层),旧公式在 Windows 必 miss。实测装 DSH 0.1.1-rc.2 后,新公式 probe 到捆绑 rg.exe;PATH 里没有引擎时它照样兜底。 Bench(Windows 11,10 万+ 文件目录):引擎比 JS 遍历快 3.7~34.7 倍;而且原生遍历会撞 100k visited 预算截断——query= 完整链路(打包→dsh plugin add→dsh web→fs.search 路由)也跑通了,engine=fd ... 25ms 这种日志在真机可见;repro 脚本在 scripts/win-*.cjs。 另外顺手把 fd 截断和 rg 对齐了(fd 之前 --max-results 钉在哨兵位,满集永不标 truncated、多返 1 条;现在 cap+1,两条引擎行为一致,有单测钉死)。 Windows CI 那条我另开 PR 跟进(装 fd/rg + 跑这两组搜索测试 + 吸收这三个脚本)。 |
…-dev#203) The editor side panel's global name search walked the tree in plain JS. On large workspaces a single search can take a second or more. Detect a native engine once per process (lazily, cached) and dispatch to it: - fd (PATH + fixed Homebrew/usr/cargo paths) with fixed-string, case-insensitive, hidden/ignore-aware listing capped by max-results - rg next: DeepSeek Harness' own bundled @vscode/ripgrep binary (resolved from process.execPath's global prefix, the same binary the harness's agent-side search tool runs) before any PATH rg, then fixed paths - plain JS walk as the fallback when no engine is available or fails Engine output is normalized to the plain walk's contract (root-relative, / separators, case-insensitive substring, sorted, truncated); .git is excluded in both engine and plain paths; runtime failures disable the engine in-process and degrade to the next one. Probe/health hooks keep the engine selection unit-testable without real binaries.
Debug instrumentation now lives in src/search-debug.ts and is a no-op unless DSH_SEARCH_DEBUG=1 is set: default is zero console output and zero disk writes. The log file resolves like pty-deps does (DSH_HOME, falling back to ~/.dsh); write failures are swallowed so a missing log dir can never break a search. relRoot uses homedir()/DSH_HOME with a separator boundary instead of process.env.HOME.
… verification in design doc README and AGENTS.md now mention the fd/rg engine dispatch for fs.search (DSH-bundled rg preferred, JS-walk fallback). The design doc records the gated instrumentation file, today real-host acceptance run (workspace timings, literal-semantics checks, .git exclusion) and the node_modules tradeoff decision.
… test-pin them Review concern (PR omdsh-dev#303): fd/rg output formats on Windows. rg emits backslash-separated paths with dot-backslash prefixes and CRLF line endings; the normalizer only handled the platform sep case, which Linux CI could not exercise with Windows-shaped lines, and the trailing CR was never explicitly stripped (relied on readline behavior). Make the separator injectable (defaults to the platform separator, behavior unchanged) and strip a trailing CR, then pin both Windows shapes in unit tests.
…robe The bundled-rg candidate derived only from process.execPath with a POSIX npm-global shape (<prefix>/lib/node_modules/...). On Windows npm global there is no lib/ layer — packages land in %APPDATA%\npm\node_modules — so the probe would resolve to a bogus C:\lib\... path and silently fall back to plain-walk (or system rg, usually absent). Also add the launchd-profile layout (~/.dsh/profiles/node_modules), which a real machine check (2026-08-22) shows is where DSH actually lands here, and the Homebrew/pnpm global roots. Wrong guesses stay cheap: every candidate still goes through the 500ms --version verify; tests pin the Windows shape (%APPDATA%, no lib/ layer) and the profile layout.
rg emits a platform-dependent separator (backslash in cmd/PowerShell on Windows, forward slash in Git Bash — rg#501). fd already pinned '/' via --path-separator; rg supports the same flag since 0.8, so pin it too and let the output match the walk contract before normalization even runs. normalizeEnginePaths stays as a safety net for exotic builds.
Runs rg/fd with and without --path-separator / on a scratch tree and reports backslash/CR/LF presence byte-level, pinning the Windows output shapes this PR normalizes (runs on any OS; Windows is where it earns its keep).
Runs bundledRgCandidates + probeEngines against the live machine and prints which candidate EXISTS plus what the real probe resolves. On a npm-global Windows DSH install this proves the %APPDATA% layout derivation hits the bundled rg.exe (verified live 2026-08-22: C:\Users\y\AppData\Roaming\npm\node_modules\@deepseek-ai\dsh\...).
…fs.search) Replicates scripts/e2e-mount.sh on Windows without bash: scratch profile bootstrap files (package.json / pnpm-workspace.yaml), a helper that checks dsh web state and POSTs /sidebar/api/fs.search with curl. The full chain was verified live on Windows (2026-08-22): plugin mounted into a scratch profile via dsh plugin add, dsh web up on an OS-assigned port, fs.search returned ok with 200 matches, and DSH_SEARCH_DEBUG proved the bundled rg.exe under %APPDATA%\npm\node_modules was probed and picked (fd engine ran the search, 222ms).
… by tests) fd capped at maxMatches+1 exactly, so it stopped AT the stream sentinel: a full result set was never marked truncated and up to maxMatches+1 rows leaked past the 200-row contract. Pin fd one line higher (cap+1) so it overflows into the same sentinel rg triggers; both engines now report truncated identically and the caller slices back to maxMatches. Extract fdArgv/rgArgv as exported pure functions so the argv contract (literal query, / separators, sentinel above cap) is pinned by unit tests. Windows bench script added: plain walk vs engines on the same root (D:\\Project real machine, 2026-08-22: engines 3.7-34.7x faster AND plain walk truncates on the 100k visited budget, dropping matches e.g. query=1234 plain 5 vs engines 40).
Windows 11 + npm-global DSH 0.1.1-rc.2 + scoop fd/rg: byte-level output shapes (backslash + .\ prefix reproduced, --path-separator / verified), %APPDATA% probe hit, full mount->dsh web->fs.search chain, bench (plain truncated on visited budget, engines 3.7-34.7x faster), and the fd truncation-symmetry fix.
…ile parity
Review follow-ups on the fd/rg engine path (verified against real rg 15.0):
- escapeGlob now escapes { }: globset reads {a,b} as alternation, so an
unbalanced '{' made rg fail glob parsing (exit 2 -> engine wrongly
disabled process-wide) and 'a{b}' silently searched 'ab'
- a 15s runtime TIMEOUT no longer adds the engine to the broken set:
a timeout means THIS tree is too large, not that the binary is broken;
the broken set is per-engine, so one huge directory would otherwise
strip every other root of the engine (new EngineTimeoutError)
- worktree '.git' FILE parity: rg gains a second exclusion glob next to
the directory one (it requires a path segment after .git and leaked
the pointer file), and the plain walk skips any entry named .git --
same semantics as fd's --exclude .git
…ew base Rebase follow-ups for the new v0.18.0 main: - Single-source the skip-name list as SKIP_DIR_NAMES (search-engines.ts); the plain walk builds SEARCH_SKIP_DIRS from it. fd (--exclude, one per name) and rg (--iglob directory + entry-only pair, per name) now exclude every name: with --no-ignore active the engines would otherwise re-enter node_modules etc. and regress d0f194b's budget savings. - The plain walk skips a worktree-style .git FILE too (removed the isDirectory gate), keeping parity with the engine excludes; both the node_modules spec and the .git-file spec are kept (walk tests pinned to searchFilesPlain for determinism). - argv symmetry tests: fd excludes == SKIP_DIR_NAMES, rg carries the glob pair per name, query iglob stays last. - escapeGlob keeps escaping '[' (rg globs are gitignore-style: '[d]' is a character class, not a literal) with a reasoned no-useless-escape exemption. - win-* ops scripts: .cjs files are run directly with node on a real host — file-level no-require-imports exemptions; drop an unused os import. - fd/rg engine docs re-anchored to README (v0.12.3 entry) and docs/external-plugin-guide.md (old AGENTS.md is now a slim rules file).
rg --files never emits a directory line, so rg-only machines missed the directory matches fd and the plain walk report. Two fixes, pinned by a real-rg smoke on this machine: - rgArgv gains a second query glob '**/*<q>*/**': the slash-free form follows gitignore semantics and matches BASENAMES only (verified on real rg 15 — 'util/helper.ts' never matched '*util*', so matching directories were invisible to derivation too); the path-level form admits files under matching directories. - deriveRgMatches(paths, query, maxMatches): basename hit stays a file match, a matching directory segment is derived as a directory match; deduped, sorted, capped at maxMatches + 1 with truncated raised when the budget (not the engine) cut the set. Residual lossy: empty directories stay invisible to rg (no file path carries them). - runChild rg branch delegates to deriveRgMatches and ORs the stream truncation flag; unit tests cover derivation, dedupe, case insensitivity, budget capping, and the two-glob argv shape.
8771ccd to
64dafd4
Compare
Rebase 到 v0.18.0 基线(2026-09-06)main 大版本合并(v0.17.1 → v0.18.0,154 commits)后原分支已 DIRTY。已线性化 rebase(丢弃 merge commit,现为 15 commits / 15 files / +1404−48),描述已同步更新。除机械冲突外,rebase 期间追加了三笔实质对齐: 1. 引擎排除与 walk 跳过名单单源化(关键修复)原实现引擎只排除 2. rg 目录名匹配(收窄 documented lossy)
本机(DSH 自带 rg 15)端到端: 3. 冲突解决与文档重锚定
验证
|
…lane Fold PR omdsh-dev#335's standalone `search-windows` lane into the existing ci-windows job instead of adding a second Windows job: real fd/ripgrep land on PATH before the suite (so the whole suite runs with engines present) and the byte-level assertions run against them afterwards. A separate job would pay the install + full-suite cost twice for strictly less coverage; the engine specs inject their own probes, so engine presence cannot flip unit-test outcomes and the ubuntu lane keeps covering the missing-engine fallback. scripts/win-engine-check.cjs gains the --assert mode: self-built scratch tree (no hard-coded C:\Users\y\wintest), skip when an engine is missing, pinned (--path-separator /) output must carry no '\'/CR, and a Unicode filename must survive the UTF-8 pipeline. docs/ci-windows.md carries the red-line list / historical fixes / standard procedure; AGENTS.md §2 points at it. That section's stale `--maxWorkers=2` is corrected to 1 (what package.json and the lane's own comment say).
2026-09-12:同步 main + 把 #335 的引擎 CI 并进来1. 同步 main( 分支此前唯一的红灯 2. #335 的搜索引擎 CI 并进 没有单开 - name: Install fd / ripgrep (choco) # pnpm install 之前,真二进制进 PATH
run: choco install fd ripgrep -y --no-progress
- name: Engine output-shape assertions (real fd/rg) # 套件之后
run: node scripts/win-engine-check.cjs --assert引擎 spec 是插桩式的(
3. 真机证据(ci-windows,run 34684285338,ci-windows 5m01s 绿) 即当年 review 里担心的「Windows 上 rg 的 4. 范围与遗留:#335 已关闭(价值去向记录在其评论里);原先设想随本 PR 携带的 build 跨平台、 |
背景
侧边栏文件窗口的全局文件名搜索(
fs.search)之前是纯 JS 递归遍历:小目录没毛病,一到大目录(十万级)就是秒级,而且 10 万条访问预算一顶,结果会被静默截断——用户感知就是"又慢、结果还不全"。issue #203 里有实测:28G 目录下 JS 遍历 1.56s,fd 只要 0.013s。做了什么
启动后探测本机的 fd / rg(进程内一次、带缓存、500ms 超时验证),探到就用引擎,探不到或运行失败自动回退原来的 JS 遍历,功能只增不减:
--fixed-strings字面量子串匹配,--max-results天然对应 200 条结果上限;文件名 + 目录名都匹配(与遍历一致)@vscode/ripgrep(agent 侧搜索工具同一个二进制),PATH 里没有也能命中;query 双 glob +deriveRgMatches保持"按名字匹配(文件 + 目录)"语义.git/node_modules/ 构建缓存等 18 项)与兜底遍历单源共享(SKIP_DIR_NAMES),两条路径结果形状一致/分隔、大小写不敏感、排序、截断),路由和客户端零改动几个取舍,说明一下
node_modules 不排除(留待产品决策)→ 已排除:main 在 d0f194b 给遍历加了噪声目录跳过,本 PR 跟随并把名单单源化到引擎(SKIP_DIR_NAMES,18 项)——常见词搜索的 200 个名额不再被依赖树占满rg 只报文件、不报目录(documented lossy)→ 已对齐:rg 无斜杠 glob 按 gitignore 语义只匹配 basename(真机 rg 15 验证),补路径级 glob**/*<q>*/**+ 从文件路径倒推目录命中(deriveRgMatches);残留差异仅剩空目录不可见(rg 看不到无文件的目录)2026-09-06 rebase 说明
main 大版本合并(v0.17.1 → v0.18.0,154 commits)后本分支已线性化 rebase(丢弃 merge commit,15 commits / 15 files / +1404−48),上述排除单源化与 rg 目录匹配即为 rebase 期间追加的对齐;期间解决的新 main 语义冲突:
SEARCH_SKIP_DIRS(d0f194b)与本 PR 的 worktree 式.git指针文件跳过合并(两种形态都跳)searchFilesPlain原设想中的 Windows CI lane 曾被 #520(ci-windows)覆盖而暂时撤出;2026-09-12 起改为并进那条 lane(见文末「2026-09-12 更新」),#335 随之关闭。
实测
设计文档 4 组真实目录基准(17 万 / 9 万 / 24 万 / 3.7 万文件):rg 比 JS 遍历快 2.5~20 倍,大目录下 JS 遍历全部预算截断、rg 结果完整。门控调试插桩(
DSH_SEARCH_DEBUG=1才写$DSH_HOME/search-debug.log,默认完全静默)真机 UI 过一遍:9 万文件 workspace 常见词 15~200ms、罕见词全量遍历 ~478ms 且不截断、通配符按字面量处理、无.git内部路径命中。验证
pnpm typecheck+pnpm lint+pnpm build全绿;全量测试与干净 main 基线逐项一致(零回归)tests/search-engines.spec.ts(探测缓存 / 失败禁用 / abort 不误禁 / argv 对称性 / deriveRgMatches)+fs-search.spec.tsdispatch 组,共 60+ 用例util→['util', 'util.ts']、src→['src'])测试计划:hook 注入走单测(引擎路径全部经
setEngineHooks注入,不依赖机器上真有 fd/rg);Windows 由ci-windowslane 承载——该 lane 现在会choco install fd ripgrep装真引擎(整条套件因此都在引擎在场的环境下跑),套件结束后跑node scripts/win-engine-check.cjs --assert的字节级断言。2026-09-12 更新
ci-windows是 main 已修的agent-pty超时 flake,同步后转绿。ci-windowslane(两个步骤:choco install fd ripgrep -y --no-progress、node scripts/win-engine-check.cjs --assert),而不是单开search-windowsjob——单开要把「装依赖 + 跑全量」再付一遍(CI 分钟翻倍)却覆盖更少;并进来则让整条套件都在引擎在场的环境下跑。引擎 spec 是插桩式的,所以引擎在不在 PATH 都不改变单测结论,ubuntu lane 继续覆盖「引擎缺失 → JS 回退」。scripts/win-engine-check.cjs一并升级为--assert版:自建 scratch 目录(不再硬编码C:\Users\y\wintest)、引擎缺失即跳过、钉死--path-separator /的输出不得含\/CR、中文文件名必须命中、失败非零退出。新增docs/ci-windows.md(红线清单 / 历史平台修正 / 标准动作),AGENTS.md §2 埋入口并顺手改正该节过期的--maxWorkers=2(实为 1)。ci(search): add search-windows lane for engine platform regressions #335 已关闭,价值去向记录在其评论里。.\中文文件名.ts\n.\src\a.ts(backslash: true),钉--path-separator /后干净,fd/rg 的中文名都命中,断言全过。fixes #203