[BUG] Vulnerability reports contain hallucinated code analysis in black-box scans

[mason5052]

Description

When running Strix in black-box mode (no source code provided), the vulnerability reports include a Code Analysis section with fabricated file paths, line numbers, and code snippets that do not exist in the target application.

To Reproduce

1. Deploy OWASP Juice Shop: docker run -d -p 3000:3000 bkimminich/juice-shop:latest
2. Run Strix scan:

strix -n -t http://localhost:3000 -m standard --instruction "Focus on OWASP Top 10: SQL injection, XSS, broken auth, IDOR, SSRF."

3. Check generated vulnerability reports in strix_runs/<run-id>/vulnerabilities/

Observed Behavior

The vulnerability reports contain fabricated code analysis:

vuln-0001 (SQL Injection):

• References src/controllers/auth.js lines 15-18 -- this file does NOT exist in Juice Shop
• References endpoint /api/v1/auth/login -- Juice Shop uses /rest/user/login
• Shows fabricated code: const query = SELECT * FROM users WHERE...

vuln-0002 (XSS):

• References src/controllers/input.js lines 10-12 -- does not exist

vuln-0004 (IDOR):

• References src/controllers/user.js lines 30-34 -- does not exist
• References /api/v1/users/ -- actual endpoint is /api/Users/

vuln-0005 (SSRF):

• References src/controllers/fetch.js lines 15-20 -- does not exist

Juice Shop actual structure uses routes/, models/, data/ directories -- none of the cited files exist.

Expected Behavior

In black-box scans (no source code provided):

1. The Code Analysis section should either be omitted or clearly marked as inferred/hypothetical
2. Endpoints referenced in PoC scripts should be verified to actually exist (return non-404)
3. File paths and line numbers should NOT be fabricated

Impact

• Users attempting remediation will look for non-existent files
• PoC scripts reference wrong endpoints and would fail if executed
• Undermines trust in report accuracy
• Particularly problematic in professional/compliance contexts

Proposed Fix

Option A: In the vulnerability reporting prompt/template (finish_actions.py), add a condition: if no source code target was provided, suppress the Code Analysis section or replace it with "Source code not available for analysis."

Option B: Before including code analysis, validate that referenced endpoints actually returned non-404 responses during the scan.

System Information

• OS: Amazon Linux 2023 (x86_64)
• Strix Version: 0.8.1
• Python Version: 3.12.9
• LLM Used: openai/gpt-4o-mini
• Scan Mode: standard
• Docker: 25.0.6