feat(autoresearch): expand Claude RTK rewriting for chains, guarded find, and rg

[MattPua]

Purpose

This PR reduces Claude agent context usage by routing more eligible Bash commands through RTK. It does not change RTK's compression algorithm. It fixes which commands safely reach it:

• Rewrite eligible segments inside && and ; chains instead of skipping the whole chain
• Route safe rg commands through native RTK
• Keep unsupported find predicates and unsafe shell shapes raw
• Preserve shell behavior, exit codes, and the facts agents need

Codex receives matching developer guidance, but has no deterministic command-rewrite hook. The measured savings below apply only to Claude sessions.

Important

Production-weighted upper estimate: 13.6% less Bash output than main.

Across the measurable 30-day production sample, that is approximately 1.58 MB less output, or 396k estimated command-output tokens.

This weights observed production output bytes by command shape. Actual savings may be lower because a chain can contain both eligible and ineligible segments.

What changes

• Rewrite eligible top-level segments inside && and ; chains
• Route rg through native RTK only when a real executable exists
• Keep output modes that RTK cannot preserve raw
• Allowlist the find predicates RTK supports
• Fail safe for multiline input, substitutions, unbalanced syntax, pipes, and redirects
• Mirror the policy in Codex guidance as a best-effort hint
• Add a benchmark against the real policy on main and a live context-fidelity E2E

Before and after

1. Large rg searches: 86.4% less output

rg -n '^import' packages/core/src

	main	This PR	Difference
Output lines	1,474	204	1,270 fewer
Output bytes	142,172	19,400	122,772 fewer
Exact totals available	No summary	1,474 matches in 511 files	Preserved

On main: all 1,474 repetitive matches enter the conversation

packages/core/src/agent-chat/agentChat.module.ts:1:import { ContainerModule } from "inversify";
packages/core/src/agent-chat/agentChat.module.ts:2:import { AgentChatService } from "./agentChatService";
packages/core/src/agent-chat/agentChat.module.ts:3:import { AGENT_CHAT_SERVICE } from "./identifiers";
... 1,471 more lines sent to the model

With this PR: one exact summary plus 200 representative matches enter the conversation

1474 matches in 511 files:

packages/core/src/agent-chat/agentChat.module.ts:1:import { ContainerModule } from "inversify";
packages/core/src/agent-chat/agentChat.module.ts:2:import { AgentChatService } from "./agentChatService";
... remaining repetitive matches omitted

Important

The agent keeps the exact answer, 1,474 matches across 511 files, while 86.4% of the output is removed from context.

2. Batched commands: compression now works inside chains

git status --short && git branch --show-current && git log --oneline --decorate -20 && git diff --stat && git diff

On main: the presence of && disables RTK for the entire line

git status --short && git branch --show-current && git log ... && git diff --stat && git diff
└──────────────────────── all five commands return raw output ────────────────────────────────┘

Result: 9,975 bytes / 188 lines

With this PR: every eligible segment is compressed independently

rtk git status --short && rtk git branch --show-current && rtk git log ... && rtk git diff --stat && rtk git diff
└──────────────────────── all five commands reach RTK ────────────────────────────────────────────┘

Result: 9,128 bytes / 168 lines

Important

This small worktree saves 8.5%. The same change unlocks compression for the 28.9% of production Bash calls containing &&, with larger savings when their output is larger.

Command order, output facts, exit codes, and && short-circuit behavior remain unchanged.

3. Unsupported find: error becomes complete results

find packages/agent -name '*.test.ts' -not -path '*/node_modules/*'

On main: RTK receives syntax it does not support

$ rtk find packages/agent -name '*.test.ts' -not -path '*/node_modules/*'
rtk: rtk find does not support compound predicates or actions (e.g. -not, -exec). Use `find` directly.
exit 1 · 0 of 102 files returned

With this PR: the unsupported shape stays raw

$ find packages/agent -name '*.test.ts' -not -path '*/node_modules/*'
packages/agent/src/acp-helpers.test.ts
packages/agent/src/acp-limits.test.ts
... all remaining files returned
exit 0 · 102 of 102 files returned

Important

The result changes from an error with no data to all 102 requested files, avoiding a failed turn and retry.

Results

Production-calibrated estimate

The measurable 30-day Bash-span sample contains 7,745 calls and 11.7 MB of actual output. Unlike the larger command-frequency analysis below, this sample includes output payloads and can be weighted by bytes.

Candidate behavior	Share of production Bash-output bytes	Fixture reduction	Estimated total reduction
Native rg	0.71%	86.4%	0.61 points
Likely eligible single-line chains	23.65%	54.8%	12.95 points
Combined directional estimate			up to 13.6%

Applied to the sample's 11.7 MB of Bash output:

• Native rg: approximately 72 KB removed
• Eligible chains: approximately 1.51 MB removed
• Combined: approximately 1.58 MB, or 396k estimated tokens using bytes divided by four

Warning

This is not an observed production saving. It applies fidelity-validated fixture compression rates to production output-byte shares. It is best treated as an upper estimate until a production shadow evaluation or post-rollout measurement can compare actual output before and after rewriting.

Safety

• Preserve chain ordering and && short-circuit behavior
• Leave multiline input, substitutions, unbalanced syntax, pipes, and redirects untouched
• Keep unsafe rg output modes raw
• Run rg through RTK only when RTK can execute a real rg binary
• Keep unsupported find predicates raw instead of replacing results with an RTK error
• Fidelity-gate benchmark rows on exit status, output presence, and exact expected facts

How did you test this?

• 117/117 scoped RTK unit tests passed
• Deterministic benchmark:
  • Candidate fidelity failures: 0
  • Main-policy fidelity failure: unsupported find -not
• Five authenticated, four-turn live Claude gateway runs:
  • RTK engagement: 5/5
  • Compressed grep file and line: 5/5
  • Compressed 4,800-match total: 5/5
  • Compressed chained Git state: 5/5
  • Full suite: 4/5
• The single live failure was the intentionally raw find case. Claude counted 0 instead of 9 once; RTK did not rewrite that command.

Automatic notifications

• Publish to changelog?
• Alert Sales and Marketing teams?

---

Created with PostHog Code

[trunk-io]

Merging to main in this repository is managed by Trunk.

• To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

[github-actions]

React Doctor found no issues in the changed files. 🎉

_{Reviewed by React Doctor for commit 19b645d.}

[greptile-apps]

_{Reviews (1): Last reviewed commit: "chore(agent): clarify RTK benchmark scop..." | Re-trigger Greptile}

[github-actions]

This PR has had no activity for 7 days and has been marked stale. We are moving to the monorepo and tightening PR staleness in preparation, so it will be closed in 7 days if no further activity occurs.

[charlesvien]

Hey @MattPua, we've migrated this repo into the PostHog/posthog monorepo, so this PR needs to be recreated there.

Easiest path: check out the monorepo and run the /porting-code-prs skill with your agent pointed at this PR. It has the migration guide and path remappings baked in so the agent doesn't have to figure them out. You don't strictly need it though, just asking your agent to port the PR works fine too.

Shout if you get stuck.