feat(depth): add graphify depth iterative sliding-window build mode

CHANGELOG.md

@@ -4,6 +4,7 @@ Full release notes with details on each version: [GitHub Releases](https://githu
 
 ## 0.9.43 (unreleased)
 
+- Feature: `graphify depth <root>` runs the full extract pipeline per sub-bucket of a large corpus and merges the per-bucket graphs into a single cross-bucket graph. Designed for the >500-file / >500K-word case where `graphify <root>` would warn. Auto-detects top-level subdirs as buckets, or accept explicit `--focus <path>`. Supports `--parallel N` (capped at 4 to respect LLM rate limits), `--resume` (skip buckets whose graph.json is fresh), `--retries N` (transient-failure retry with exponential backoff), `--skip-on-error` (default) / `--no-skip-on-error`, `--dry-run` (preview auto-detected buckets without running extract), `--global` (fold the merged graph into the user's cross-repo global graph; `--global-tag NAME` overrides the default tag), and `-- <extract args>` to forward flags to every per-bucket `graphify extract`. Output: a per-bucket `<root>/graphify-out/depth/buckets/<name>/graph.json`, the cross-bucket `<root>/graphify-out/graph.json`, and a `DEPTH_REPORT.md` that surfaces cross-bucket signal labels (entity names minted under multiple bucket prefixes).
 - Feature: OCaml `.ml`/`.mli` extraction via tree-sitter-ocaml (optional `[ocaml]` extra). Extracts modules, top-level and module-level values/functions, types and their variant constructors, `open` imports, and function calls; qualified calls (`Geo.area`) resolve to the value, and cross-file `open`/call targets collapse onto the unique real definition via the corpus stub rewire.
 - Fix: a cross-file INFERRED `uses` edge now binds to the symbol whose body actually references the imported name (a module-level function is a valid source; a co-located class that never touches the import gets no edge), instead of fanning out from the import line to every class in the importing file (#2652, thanks @ousamabenyounes). A reference at module top level, with no enclosing symbol, emits no edge.
 - Fix: a named `function` declaration nested inside another function now gets its own node, a `contains` edge from the enclosing function, and its own call scope, so calls made from inside it are no longer dropped as dangling (#2653, thanks @himanshupatro-334). Coverage was extended to the arrow idioms too: a function declared inside an arrow-defined component (`const Panel = () => { function handleClick(){} }`) or inside an arrow callback (`useEffect(() => { function h(){} })`) is captured and attributed to the nearest enclosing named scope.

NEWS.md

@@ -0,0 +1,45 @@
+# News
+
+This file lists significant new capabilities contributed to graphify
+upstream, in chronological order. Each entry names the method, the
+contributor, the merged commit, and a one-paragraph description so a
+reader can decide whether to read the full PR / commit history.
+
+---
+
+## Iterative sliding-window depth-graph method (0.9.43)
+
+**Contributor:** JFWaskin
+**Merged in:** `569cf56` (production polish), `dcdef67` (robustness), `feb7581` (pilot)
+**Shipped as:** the `graphify depth <root>` command + the `DEPTH_REPORT.md` output
+
+The `graphify depth` command introduces the *iterative sliding-window
+depth-graph method*: a multi-pass build for the >500-file / >500K-word
+case where a single-pass `graphify <root>` warns and asks the user to
+narrow manually. The method auto-detects a corpus into N sub-buckets
+(top-level subdirs with at least M files / W words) or accepts an
+explicit `--focus <path>` set, runs the full extract pipeline per
+bucket via subprocess, then merges the per-bucket graphs into a single
+cross-bucket graph using the same prefix-and-compose path the existing
+`graphify merge-graphs` already uses. The new output is `DEPTH_REPORT.md`,
+which surfaces "cross-bucket signals" — entity LABELS (not ids) that
+appear under multiple bucket prefixes in the merged graph and are
+therefore the most actionable cross-system hint a reviewer can get
+from a build.
+
+The command is a thin orchestration layer over the existing
+`graphify extract` and `graphify merge-graphs` code paths; no existing
+command or its behaviour is changed. The 8 production scenarios
+covered are: monorepo (>500 files), selective focus, resume after
+interruption, CI / flaky network (transient-failure retry with
+exponential backoff), CI parallel (capped at 4 workers to respect
+LLM API rate limits), cross-repo integration (--global), sandbox /
+read-only (--dry-run preview), and partial-failure containment
+(--skip-on-error vs --no-skip-on-error).
+
+The contribution ships with 21 tests (14 unit + 7 integration,
+including a real-extract smoke test that invokes the installed
+`graphify extract` subprocess on a fixture corpus with `--code-only
+--no-cluster` and no LLM API key required), and was validated
+end-to-end against the real graphify source itself (~70 packages,
+4 000+ source files).

graphify/__main__.py

@@ -528,6 +528,22 @@ def _run_cli() -> None:
         print("  merge-driver <base> <current> <other>  git merge driver: union-merge two graph.json files (set up via hook install)")
         print("  merge-graphs <g1> <g2>  merge two or more graph.json files into one cross-repo graph")
         print("    --out <path>            output path (default: graphify-out/merged-graph.json)")
+        print("  depth <root>            iterative sliding-window build: extract per sub-bucket, then merge")
+        print("    --focus PATH            explicit bucket path (repeatable); default: auto-detect top-level dirs")
+        print("    --out DIR               depth output dir (default: <root>/graphify-out)")
+        print("    --merged PATH           cross-bucket graph.json path (default: <out>/graph.json)")
+        print("    --report PATH           DEPTH_REPORT.md path (default: <out>/DEPTH_REPORT.md)")
+        print("    --max-buckets N         cap auto-detected buckets (default 20)")
+        print("    --min-files N           per-bucket file-count floor for auto-detect (default 20)")
+        print("    --min-words N           per-bucket word-count floor for auto-detect (default 5000)")
+        print("    --parallel N            run N buckets concurrently (capped at 4; default 1)")
+        print("    --timeout S             per-bucket extract timeout in seconds")
+        print("    --skip-on-error         default; do not abort on a single bucket failure")
+        print("    --no-skip-on-error      abort the whole run on the first bucket failure")
+        print("    --resume                skip buckets whose graph.json is fresher than their source")
+        print("    --dry-run               auto-detect buckets and report, do not run extract")
+        print("    --graphify-bin PATH     override the graphify executable used for per-bucket extract")
+        print("    -- <extract args>...    args forwarded to every per-bucket `graphify extract` (e.g. --backend X)")
         print("    --branch <branch>       checkout a specific branch (default: repo default)")
         print("    --out <dir>             clone to a custom directory (default: ~/.graphify/repos/<owner>/<repo>)")
         print("  add <url>               fetch a URL and save it to ./raw, then update the graph")

graphify/cli.py

@@ -2398,6 +2398,141 @@ def _to_simple(g: "_nx.Graph") -> "_nx.Graph":
         print(f"Merged {len(graphs)} graphs -> {merged.number_of_nodes()} nodes, {merged.number_of_edges()} edges")
         print(f"Written to: {out_path}")
 
+    elif cmd == "depth":
+        # `graphify depth <root>`: iterative sliding-window build.
+        # Runs the full extract pipeline per sub-bucket (auto-detected or
+        # supplied via --focus) and merges the per-bucket graphs into a
+        # single cross-bucket graph. See graphify/depth.py for the full
+        # design. Output is written to <root>/graphify-out/graph.json
+        # by default, plus a per-bucket sub-dir and a DEPTH_REPORT.md.
+        if len(sys.argv) < 3:
+            print(
+                "Usage: graphify depth <root> "
+                "[--focus PATH]... [--out DIR] [--merged PATH] "
+                "[--report PATH] [--max-buckets N] [--min-files N] "
+                "[--min-words N] [--parallel N] [--timeout S] "
+                "[--retries N] [--retry-backoff S] "
+                "[--skip-on-error|--no-skip-on-error] [--resume] [--dry-run] "
+                "[--global] [--global-tag NAME] "
+                "[--graphify-bin PATH] [-- <extract args>...]",
+                file=sys.stderr,
+            )
+            sys.exit(1)
+        if sys.argv[2].startswith("-"):
+            target = Path(".").resolve()
+        else:
+            target = Path(sys.argv[2]).resolve()
+        if not target.exists():
+            print(f"error: path not found: {target}", file=sys.stderr)
+            sys.exit(1)
+
+        focuses: list[Path] = []
+        out_dir: Path | None = None
+        merged_path: Path | None = None
+        depth_report_path: Path | None = None
+        max_buckets = 20
+        min_files = 20
+        min_words = 5_000
+        parallel = 1
+        timeout_s: int | None = None
+        retries = 0
+        retry_backoff_s = 2.0
+        skip_on_error = True
+        resume = False
+        dry_run = False
+        add_to_global = False
+        global_tag: str | None = None
+        graphify_bin: str | None = None
+        # Anything after a literal `--` is forwarded to every per-bucket
+        # `graphify extract` invocation (e.g. `--backend`, `--model`,
+        # `--mode deep`, `--no-cluster`, `--code-only`).
+        extract_args: list[str] = []
+        i = 3
+        args = sys.argv[3:]
+        if "--" in args:
+            sep = args.index("--")
+            forward = args[sep + 1 :]
+            args = args[:sep]
+            extract_args = forward
+        while i - 3 < len(args):
+            a = args[i - 3]
+            if a == "--focus" and i - 3 + 1 < len(args):
+                focuses.append(Path(args[i - 3 + 1]).resolve())
+                i += 2
+            elif a == "--out" and i - 3 + 1 < len(args):
+                out_dir = Path(args[i - 3 + 1]).resolve()
+                i += 2
+            elif a == "--merged" and i - 3 + 1 < len(args):
+                merged_path = Path(args[i - 3 + 1]).resolve()
+                i += 2
+            elif a == "--report" and i - 3 + 1 < len(args):
+                depth_report_path = Path(args[i - 3 + 1]).resolve()
+                i += 2
+            elif a == "--max-buckets" and i - 3 + 1 < len(args):
+                max_buckets = int(args[i - 3 + 1]); i += 2
+            elif a == "--min-files" and i - 3 + 1 < len(args):
+                min_files = int(args[i - 3 + 1]); i += 2
+            elif a == "--min-words" and i - 3 + 1 < len(args):
+                min_words = int(args[i - 3 + 1]); i += 2
+            elif a == "--parallel" and i - 3 + 1 < len(args):
+                parallel = int(args[i - 3 + 1]); i += 2
+            elif a == "--timeout" and i - 3 + 1 < len(args):
+                timeout_s = int(args[i - 3 + 1]); i += 2
+            elif a == "--retries" and i - 3 + 1 < len(args):
+                retries = int(args[i - 3 + 1]); i += 2
+            elif a == "--retry-backoff" and i - 3 + 1 < len(args):
+                retry_backoff_s = float(args[i - 3 + 1]); i += 2
+            elif a == "--no-skip-on-error":
+                skip_on_error = False; i += 1
+            elif a == "--skip-on-error":
+                skip_on_error = True; i += 1
+            elif a == "--resume":
+                resume = True; i += 1
+            elif a == "--dry-run":
+                dry_run = True; i += 1
+            elif a == "--global":
+                add_to_global = True; i += 1
+            elif a == "--global-tag" and i - 3 + 1 < len(args):
+                global_tag = args[i - 3 + 1]; i += 2
+            elif a == "--graphify-bin" and i - 3 + 1 < len(args):
+                graphify_bin = args[i - 3 + 1]; i += 2
+            else:
+                print(f"error: unknown flag: {a}", file=sys.stderr)
+                sys.exit(2)
+
+        from graphify.depth import depth_command
+        result = depth_command(
+            root=target,
+            focuses=focuses or None,
+            out_dir=out_dir,
+            merged_path=merged_path,
+            depth_report_path=depth_report_path,
+            max_buckets=max_buckets,
+            min_files=min_files,
+            min_words=min_words,
+            extract_args=tuple(extract_args),
+            graphify_bin=graphify_bin,
+            timeout_s=timeout_s,
+            parallel=parallel,
+            skip_on_error=skip_on_error,
+            resume=resume,
+            dry_run=dry_run,
+            retries=retries,
+            retry_backoff_s=retry_backoff_s,
+            add_to_global=add_to_global,
+            global_tag=global_tag,
+        )
+        print(
+            f"[graphify depth] status={result.status} "
+            f"buckets={len(result.buckets)} "
+            f"merged={result.merged_graph_path} "
+            f"elapsed={result.total_elapsed_s:.1f}s"
+        )
+        if result.status == "failed":
+            sys.exit(1)
+        if result.status == "partial" and not skip_on_error:
+            sys.exit(2)
+
     elif cmd == "clone":
         if len(sys.argv) < 3:
             print(