[Superseded by #46/#159] Benchmark text-only vs structure-aware coding-agent strategies

[dgenio]

Context

The debate around Claude Code, parsing, ASTs, tool loops, and symbolic structure should be testable. Instead of arguing whether coding agents “really understand” code, this repo can benchmark concrete routing/context strategies.

agent-routing-eval-lab is a good place to compare coding-agent strategies before production use.

Proposal

Add an evaluation scenario for coding-agent routing/context strategies.

Strategies to compare:

• Plain file/grep retrieval.
• Embedding-based file retrieval.
• AST/symbol-based retrieval.
• LSP-based symbol retrieval.
• Repo knowledge graph retrieval.
• contextweaver generated context packs.
• Hybrid strategies with budget-aware routing.

Metrics

• Correctness / task success.
• Tokens used.
• Tool calls used.
• Latency / wall time.
• Number of failed edits.
• Number of test runs.
• Unnecessary files touched.
• Public API changes.
• Security/policy violations.
• Human-review burden.

Acceptance criteria

• Define a small benchmark dataset of coding tasks over fixture repos.
• Add strategy abstraction for text-only vs structure-aware retrieval/context.
• Capture tool-call, token, cost, and failure metrics.
• Include at least one structure-aware baseline.
• Add reporting that shows trade-offs by strategy.
• Document how this can validate contextweaver, ChainWeaver, vibeguard, and agent-kernel design choices.

Notes

This should turn the symbolic-vs-neural discussion into measurable engineering trade-offs.