Investigate commands on PATH before answering / autofixing (#287)#306
Open
vanzue wants to merge 6 commits into
Open
Investigate commands on PATH before answering / autofixing (#287)#306vanzue wants to merge 6 commits into
vanzue wants to merge 6 commits into
Conversation
The terminal agent routed "how do I use X" / "what is X" questions to Chat mode, which answers from general knowledge with "No tool calls" — so it gave the same "use help / Get-Command" boilerplate whether or not X actually exists on the user's machine, ignoring local PowerShell scripts and programs on PATH (issue #287). Reframe these as a chat answer that requires read-only investigation first, taught via a single worked sample rather than a rigid per-shell rule table: look at the PATH (Get-Command), try the command's own help to learn its usage, then answer from evidence. When nothing is found, say so plainly and offer real near-matches from the PATH via Get-Command -UseFuzzyMatching (reusing PowerShell's own resolver/fuzzy engine instead of inventing spelling correction). Prompt-only change to the embedded terminal-agent.md; seed_prompt_files migrates it to users on next start. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the embedded Terminal Agent system prompt to ensure that “how do I use X” / “what is X” questions trigger a quick read-only investigation of whether X exists on the user’s machine, rather than defaulting to generic guidance.
Changes:
- Refines Chat mode guidance to allow an exception where command-specific questions require investigation before answering.
- Adds a new “Chat answers that need investigation” section with a worked example showing how to resolve a command, query its help/usage, and answer from evidence.
- Updates the response-format section to reiterate that local-command chat questions should be investigated first.
Just want to say I really appreciate the work you put in! Since I need Windws, Linux and macOS I need a nice terminal and conhost, even if it's a legend, lacks agent feature! God Bless! 👌👑👑👑 |
The autofix prompt could correct typos it knew from training data (`dotent`->`dotnet`) but never suggested the *local* PowerShell scripts and programs on the user's PATH that they most likely mistyped (`deploit`->`deploy-it`), because the LLM has no knowledge of the user's environment. Add `command_recall`, which grounds "did you mean" suggestions in the user's real PATH. In the autofix prompt-assembly path (PowerShell only in v1): - Extract the failing command token from the captured [command + output] buffer (ControlCore::ReadLastPrompt starts at the FTCS command mark, so there is no prompt prefix to strip). - Gate locale-independently on existence, not on the (localized) error text: a cheap in-process `which` pre-gate short-circuits the common case (failed build/test/git where the program exists) with no subprocess; only a genuine not-found pays one `Get-Command` enumerate. - Rank the real command list by Damerau-Levenshtein (strsim), with an anagram tie-break so a transposition (`gti`->`git`) outranks an equidistant substitution (`gti`->`gci`). Inject the top matches as a `### Near Matches` section. `auto-fix.md` is updated to consume `### Near Matches` and to state plainly when a command isn't on PATH instead of giving generic "check spelling / use help" advice. strsim is promoted to a direct dependency; it was already in the lockfile via clap, so the resolved graph (and cgmanifest/NOTICE) is unchanged. Known blind spot (accepted for v1): the enumerate runs `-NoProfile`, so it sees PATH executables/scripts (the issue's concern) but not profile-only interactive functions/aliases. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
vanzue
changed the title
…hing The investigation sample told the agent to spelling-correct an unknown command with `Get-Command -UseFuzzyMatching`, but on PS 7.6.2 that ranking buries PATH applications and external scripts (git ranked ~300th for `gti`; `deploy-it` not returned for `deploit`) — exactly the local scripts/programs issue #287 is about. Replace it with a PATH-grounded approach that needs no fixed matcher: the agent searches the real command list with a wildcard stem (`Get-Command -Name "*depl*"` / `compgen -c | grep`) and picks the closest itself — the LLM is a better judge of a likely typo than the built-in fuzzy ranker. Existence is still confirmed with `Get-Command`, so the issue's minimum ("say there is no such command") holds in any locale. (The strsim-backed near-match engine stays in the autofix path, where wta assembles the prompt in-process and doesn't depend on any agent-callable tool. There is no MCP server and `wta` isn't on PATH, so exposing a CLI/MCP tool to the agent pane isn't viable without disproportionate C++/packaging work.) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…, wording Address Copilot review feedback on PR #306: - strip_exe_ext: guard the slice with str::get so a non-ASCII command name (Unicode function/alias) whose byte boundary lands mid-char no longer panics and crashes autofix prompt assembly. - command_exists / rank_near_matches: strip the user-typed token's own executable extension too, so an explicit 'deploy-it.ps1' matches the stripped candidate instead of being misreported as not-found. - enumerate_powershell_commands: include Cmdlet in the Get-Command existence gate so a failing cmdlet invocation (e.g. Get-Item with a bad path) isn't misclassified as a not-found command. - Reword the injected '### Near Matches' section and auto-fix.md to say 'not found as a command in this shell' instead of 'on PATH', since the list comes from Get-Command (aliases/functions/cmdlets), not just PATH. Adds 3 regression unit tests (non-ASCII no-panic, token-with-extension existence, token-extension ranking). cargo test: 917 passed / 0 failed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… help Address remaining Copilot review feedback on PR #306: - Chat-mode investigation: make explicit that the agent runs the read-only probe via its OWN tools (execute_command / view / read_text_file) and must NOT emit a recommendation card asking the user to run it. Resolves the concern that Chat mode (No JSON) had no stated path to actually execute the investigation. - Usage discovery: prefer Get-Help and reading the script's param block (non-executing) as the source of truth; demote the command's own help flag (-? / --help) to a side-effect-aware last resort, since a plain script may run its body before any help check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Two unresolved Copilot review threads: - client.rs: wrap each \### Near Matches\ candidate in inline-code so command names containing Markdown-significant chars (_, *) are read as literals, not formatting. - terminal-agent.md: Get-Command resolves more than PATH (cmdlets/functions/aliases), so reword the sample to describe the command by its actual type and avoid claiming it isn't 'on PATH' when it may be a function/alias. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #287 — asking the agent "How do I use X" gave the same generic "use
help/Get-Command" advice whether or not X exists, ignoring local PowerShell scripts and programs on PATH and never flagging a non-existent command.The reporter's scenario is a question to the agent ("How do I use X"), which routes to the agent-pane planner — so the fix is primarily there (Part 1). A related autofix improvement rides along (Part 2).
Part 1 — Agent pane: investigate before answering "how do I use X"
terminal-agent.mdrouted these questions to Chat mode ("No tool calls"), so the agent answered from general knowledge without checking the live environment. Now, a question about a specific local command requires a quick read-only investigation first, taught via one worked sample:Get-Command X(meets the issue's minimum — "say there is no such command" — in any locale).Get-Help/-?/--help, or read the script).For "did you mean", the agent searches the real command list with a wildcard stem (
Get-Command -Name "*depl*"/compgen -c | grep) and picks the closest itself — the LLM is a better judge of a likely typo thanGet-Command -UseFuzzyMatching, which (measured on PS 7.6.2) buries PATH applications/scripts (gitranked ~300th forgti;deploy-itnot returned fordeploit). No fixed matcher, no new tooling — works for every agent and shell.Part 2 — Autofix: local PATH near-matches for not-found commands
When a run command fails as not-found,
command_recallinjects a### Near Matchessection so the agent can suggest the local script/program the user mistyped. This runs in-process inside wta (which assembles the autofix prompt), so it can use a real ranking engine:whichpre-gate skips the common case with no subprocess.strsim), with an anagram tie-break so a transposition (gti→git) outranks an equidistant substitution (gti→gci).auto-fix.mdconsumes### Near Matchesand states plainly when a command isn't on PATH.Why the split (delivery reality)
The agent pane can't call a wta "tool": there is no MCP server in the codebase (the
wta mcpmode in AGENTS.md is stale — integration is ACP-only), andwta.exeis not on PATH (onlywtcliandwtai=Windows Terminal have AppExecutionAliases). Exposing a CLI/MCP resolver to the agent would need disproportionate C++/packaging work, so Part 1 stays prompt-only (LLM over real shell output) and the strsim engine lives in Part 2, where wta runs it itself.strsimis promoted to a direct dep; it was already in the lockfile via clap, so the resolved graph (andcgmanifest.json/NOTICE.md) is unchanged.Known blind spot (v1): the autofix enumerate runs
-NoProfile, so it sees PATH executables/scripts (the issue's concern) but not profile-only interactive functions/aliases.Test
cargo build✅ (only pre-existing warnings)cargo test✅ 914 passed / 0 failed, including 11command_recallunit tests (token extraction, extension stripping, existence gate, transposition/anagram ranking, threshold rejection, dedup/cap).