Skip to content
This repository was archived by the owner on Jun 5, 2025. It is now read-only.
This repository was archived by the owner on Jun 5, 2025. It is now read-only.

Suspicious Command Detection #34

Open
4 of 7 issues completed
Open
Story 🗺️
4 of 7 issues completed
@jhrozek

Description

@jhrozek

Suspicious Command Detection

Summary
Introduce a mechanism to detect and flag potentially suspicious commands generated by AI assistants / agents. This feature will prompt the user to double-check such commands before they are executed or accepted, particularly for fully agentic workflows (where commands might be auto-run).


Background & Motivation

  • Certain shell commands, while not even outright malicious, can pose security risks or unintended consequences if run blindly (e.g., curl | bash, nc -l, sudo).
  • CodeGate should help developers identify and confirm these commands before they’re executed, reducing the risk of accidentally introducing vulnerabilities or making catastrophic changes.
  • This is especially relevant in “agentic” scenarios, where an AI assistant or workflow might automatically execute commands without explicit user approval.

NOTE: As always, start small, simple and validate, the following acts as a guideline of where this could lead.

Requirements

  1. Suspicious Command Detection
    • Maintain a list of known suspicious patterns (e.g., curl | bash, nc -l, sudo).
    • Provide a mechanism (e.g., regex checks, pattern scanning) to identify these commands in AI-generated outputs.
    • Examples of suspicious commands/patterns:
      • curl | bash
      • nc -l
      • sudo
      • Changing environment variables (PATH, LD_LIBRARY_PATH, etc.)
      • File ownership/permission changes (chown, chmod)
      • Package installs like npm install, unless a positive vetting mechanism is in place
      • Destructive Patterns (rm -rf *, fork-bomb :(){ :|:& };:
  2. User Prompt or Flagging
    • For chat-based interactions, automatically flag suspicious commands in the conversation with a warning or prompt:
      “Are you sure you want to run this command? It may have system-wide effects.”
    • For fully agentic workflows, prevent automatic execution of flagged commands until the user explicitly confirms.
  3. Configuration & Customization
    • Allow users to customize or extend the suspicious commands list (add, remove, or override default patterns).
    • Provide a toggle to enable/disable suspicious command blocking entirely (for advanced users).
  4. Logging & Auditing
    • Log all instances where suspicious commands are detected, along with whether the user approved or declined execution.
    • Store logs locally for auditing and compliance purposes.
  5. Seamless Integration
    • Integrate with existing interception Pipeline logic so flagged commands can be audited in the DB and dashboard.
    • Ensure minimal latency or disruption to the typical developer workflow.

Implementation Ideas

  1. Regex-Based Detection
    • A local rule set of suspicious patterns (e.g., YAML or JSON config) keyed to relevant commands.
    • Simple “contains string” or regex scans on the generated command text.
  2. User Confirmation Dialog
    • For chat usage, display a warning or highlight the suspicious command snippet. perhaps like we do for secrets, but we will need to block instead as its the reverse path
    • For agentic flows, pause execution and present a modal or CLI prompt:
      “Command flagged: curl | bash. Confirm to proceed or skip.”
  3. Integration with Policy Enforcement
    • If the user’s policies forbid certain commands, auto-block or require multiple steps of confirmation (e.g., privileged operations).
  4. Future Enhancements
    • Expand detection to suspicious script constructs (e.g., “rm -rf /”), or advanced heuristics that leverage AI to detect anomalies.
    • Add a “whitelist mode” to automatically approve certain commands in controlled environments.

Acceptance Criteria

  1. Suspicious Patterns Defined
    • A default set of suspicious commands is included.
    • Users can add their own via a config or UI.
  2. Flagging & Approval Flow
    • Chat-based usage: Suspicious commands are highlighted with a caution or warning.
    • Agentic usage: Execution is blocked until the user explicitly confirms.
  3. Logging
    • All flagged commands are recorded in a local log with timestamps and user actions (approve/deny).
  4. Performance
    • The detection/flagging process does not introduce significant lag to normal AI interactions.
  5. Documentation
    • Clear instructions on how to manage suspicious command lists, enable/disable the feature, and interpret the logs.

Additional Notes

  • Security Considerations: This feature aims to reduce the chance of accidentally running high-risk commands, but it does not replace general best practices (e.g., running commands in a sandbox or dev environment first) and may not always capture all commands.

Sub-issues

Metadata

Metadata

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions