[Feature Request] Add a built-in search_corpus tool for recursive code/text search

[dengluozhang-png]

[Feature Request] Add a built-in search_corpus tool for recursive code/text search

Summary

The production.py tools list (10 tools: file_read, file_write,
file_edit, list_dir, quality_check, etc.) is great for operating
on files but doesn't include anything for discovering content across a
codebase. Before writing or reviewing code, agents need to grep the codebase
for patterns (definitions, usages, TODOs). Today the only way is file_read
followed by manual inspection, which is brittle and slow for non-trivial
codebases.

This issue proposes adding a search_corpus tool that performs recursive
literal/regex search across a directory and returns matches with context
lines. It's a small (~110 LOC), zero-dependency tool that fits cleanly into
the existing BaseTool duck-typed interface.

Why

1. Multi-agent pipelines benefit. A code_assist pipeline that searches
   the codebase first ("find all uses of LRUCache") and then plans
   changes against actual usage produces far better results than writing in
   isolation.
2. The data path is already there. PipelineStep accepts tool=...
   and _run_tool (in pipeline.py:317) passes context as params,
   letting search_corpus receive path and pattern keys from the
   pipeline context.
3. No new dependency. Pure Python re module + pathlib.rglob. Works
   on Windows/macOS/Linux identically.

Proposed interface

search_corpus:
  path: str            # required, root directory
  pattern: str         # required, literal or regex
  mode: literal|regex  # default literal
  file_glob: str       # default '*'
  context_lines: int   # default 2
  max_results: int     # default 50
  max_file_size: int   # default 500000 (skip larger)
  case_sensitive: bool # default false

Returns:

{
    "success": True,
    "result": {
        "matches": [
            {"file", "line", "content", "context_before", "context_after"},
            ...
        ],
        "total": int,
        "truncated": bool,
        "files_searched": int,
        "files_skipped": int,
        "text_view": "<formatted human-readable dump>",
    }
}

Defaults that skip noisy dirs (hardcoded): .git, .hg, .svn,
__pycache__, .pytest_cache, .mypy_cache, .ruff_cache,
node_modules, venv, .venv, env, .env, dist, build, target,
out, .next, .nuxt, .idea, .vscode, .DS_Store.

Why not just ripgrep?

• ripgrep is fast but adds an external binary dependency. Most OpenSymphony
  deployments run on consumer hardware where pulling another binary is
  friction.
• Pure Python is ~10× slower on huge repos but plenty fast for the
  realistic LLM-call volume (an agent may search a repo a handful of times
  per turn).
• For users who want ripgrep speed, a SearchCorpusTool(rg_path="rg")
  variant is straightforward to add later as a subclass — but a single
  zero-dep implementation covers the 95% case today.

Reference implementation

I've been using this locally for ~1 month across 3 multi-agent pipelines.
The implementation is ~110 lines of pure Python:

class SearchCorpusTool:
    """Recursively search a directory for a pattern, return matches with context."""
    name = "search_corpus"
    description = (
        "Recursively search a directory for a literal string or regex pattern, "
        "returning matches with surrounding context. Useful for code search "
        "before generating or reviewing code. Params: {path, pattern, mode?, "
        "file_glob?, context_lines?, max_results?, max_file_size?, case_sensitive?}"
    )

    def execute(self, params: dict) -> dict:
        path_str = params.get("path") or params.get("corpus_path") or ""
        pattern  = params.get("pattern") or params.get("query") or params.get("search") or ""
        # ... validation, compilation, walk, collect, format text_view ...
        return {"success": True, "result": {...}}

Full source available on my fork (PR ready to land):
https://github.com/dengluozhang-png/opensymphony/blob/feat/search-corpus-tool/opensymphony/tools/production.py

Suggested tests

• test_search_corpus_literal_match: matches "class Agent" in known file
• test_search_corpus_regex_mode: pattern def .*\\(self\\) finds methods
• test_search_corpus_skips_dot_git: file count assertion in a fixture
  repo with .git/
• test_search_corpus_max_results_truncation: 100 matches with
  max_results=10 returns 10 + truncated=True
• test_search_corpus_max_file_size_skips_huge_files
• test_search_corpus_case_sensitivity: same query, two flag values, two counts
• test_search_corpus_field_aliases: corpus_path and query accepted
  alongside path and pattern

Related

• PR docs+test: gateway quickstart section + regression tests for issue #3 #7 just landed for issue [Bug] _parse_tool_call silently drops tool calls whose JSON contains an invalid escape sequence #3 docs + tests; this is a separate,
  additive feature.
• Pipeline system: opensymphony/pipeline.py already supports
  tool= steps and passes context as params — no engine changes
  needed.

Open questions

1. Naming: search_corpus vs grep vs code_search vs find_in_files?
   My local fork uses search_corpus because "corpus" reflects the use case
   (searching a body of code), but grep is most familiar.
2. Tool Workshop integration: should SearchCorpusTool be added to
   ToolWorkshop too, or kept production-only? I lean production-only
   (it's a common utility, not domain-specific).
3. Default max_file_size: 500KB matches file_read. Should this be a
   shared constant?