Harden JetBrains inspection skill against false-green route and session states

[cbusillo]

Summary

The jetbrains-inspection skill and helper guidance should be hardened so agents cannot confidently treat ambiguous JetBrains inspection runs as clean readiness evidence.

A quick red-lane test on 2026-06-20 reproduced two relevant problems:

1. The maintained WebStorm red-lane smoke in cbusillo/jetbrains-inspection-api failed to confirm RED for an intentionally broken fixture.
2. Manual helper invocation can still produce GREEN/0 findings in a setup where the fixture is intentionally broken, depending on route/profile/scope selection.

Implementation issue filed in the plugin/helper repo:

• WebStorm red-lane proof fails to confirm RED and exposes false-green route risk jetbrains-inspection-api#113

Why This Matters

The skill is used during readiness/closeout flows. If route, worktree, session, profile, or scope ambiguity can resolve to GREEN, agents may report false confidence while inspecting the wrong project, stale project, parent git worktree, or a default profile that does not exercise expected findings.

The user concern was specifically that worktrees and session setup allow false greens. The quick test supports that concern.

Evidence

WebStorm red-lane smoke failed to confirm RED

From /Users/cbusillo/Developer/jetbrains-inspection-api:

./scripts/dogfood-red-lane-smoke.sh --product webstorm --json-out tmp/dogfood-red-lane-webstorm-quick.json

Observed:

status=failed bucket=red_not_confirmed verdict=UNKNOWN total=0 cleanup=closed

Key fields:

{
  "product": "webstorm",
  "verdict": "UNKNOWN",
  "verdict_reason": "inspection_trigger_empty_model",
  "total_problems": 0,
  "profile_resolved_name": "RedLane",
  "suspicious_empty_model": true
}

IntelliJ red-lane smoke confirmed RED

./scripts/dogfood-red-lane-smoke.sh --product intellij --json-out tmp/dogfood-red-lane-intellij-quick.json

Observed:

status=ok bucket=red_confirmed verdict=RED total=1 cleanup=closed

So RED is possible in the helper ecosystem; the issue is not just that all inspections are clean.

Manual false-green hazard

Manual invocation against the WebStorm fixture without --profile RedLane returned GREEN:

uv run /Users/cbusillo/Developer/codex-skills/jetbrains-inspection/scripts/jb-inspect.py \
  inspect-closeout \
  --repo /Users/cbusillo/Developer/jetbrains-inspection-api/test-fixtures/inspection-red-lane-webstorm \
  --ide WebStorm \
  --project-path /Users/cbusillo/Developer/jetbrains-inspection-api/test-fixtures/inspection-red-lane-webstorm \
  --worktree-path /Users/cbusillo/Developer/jetbrains-inspection-api/test-fixtures/inspection-red-lane-webstorm \
  --no-worktree-check \
  --scope all \
  --timeout-ms 120000 \
  --prepare-timeout-ms 60000

Observed:

STATUS: clean
VERDICT: GREEN reason=no_matching_findings
SUMMARY: clean=True total_problems=0 problems_shown=0

The fixture contains an intentional duplicate JSON property, so this is at minimum a warning that default profile/scope invocations can be misleading for proof-lane tests.

Route/worktree inference hazard

When targeting the nested fixture by repo path, the helper normalized the target to the parent git worktree:

target_worktree=/Users/cbusillo/Developer/jetbrains-inspection-api

while the actual open WebStorm project was:

base_path=/Users/cbusillo/Developer/jetbrains-inspection-api/test-fixtures/inspection-red-lane-webstorm

This required explicit --project-path, --worktree-path, and --no-worktree-check to target the fixture. The skill should warn agents away from trusting inferred route results in nested worktree/project situations.

Desired Skill / Helper Guidance Changes

• Strengthen the skill text around GREEN evidence:
  • GREEN is valid only when the exact worktree/project route, scope, and profile are known and fresh.
  • If helper route diagnostics mention parent worktree normalization, unmatched open projects, stale session, missing profile, or inspection_trigger_empty_model, the result is UNKNOWN and not readiness evidence.
• Make red-lane smoke a recommended health check before relying on JetBrains inspection for high-confidence readiness, especially after helper/plugin changes.
• Add explicit guidance for nested projects and linked worktrees:
  • pass --project-path / --worktree-path when inspecting a nested project or non-root fixture;
  • do not use --no-worktree-check for normal repo readiness unless intentionally proving a nested fixture or documented exception;
  • do not report GREEN if the helper had to fall back to a parent worktree or another open project.
• Consider adding a skill command or documented recipe for red-lane verification across products:
  • IntelliJ: ./scripts/dogfood-red-lane-smoke.sh --product intellij
  • WebStorm: ./scripts/dogfood-red-lane-smoke.sh --product webstorm
  • PyCharm if relevant.
• Consider requiring agents to include inspection route evidence in closeout summaries when JB inspection is used:
  • IDE/product/version
  • project key/base path
  • scope/profile
  • verdict reason
  • stale/session status

Acceptance Criteria

• The skill documentation makes it much harder for agents to convert UNKNOWN or ambiguous route/session state into GREEN.
• The skill includes a concise red-lane verification workflow or points to the maintained jetbrains-inspection-api smoke script.
• Guidance explicitly addresses worktrees, nested projects, and session setup as false-green risks.
• The skill links to the implementation issue WebStorm red-lane proof fails to confirm RED and exposes false-green route risk jetbrains-inspection-api#113 as the current known WebStorm red-lane failure.

[cbusillo]

Update from the false-green proof pass:

This issue now has a concrete helper-side proof/fix branch: fix/jb-inspection-false-green-proof.

Focused tests first failed for the exact false-green risks, then passed after the helper fix:

uv run jetbrains-inspection/tests/test_jb_inspect.py \
  BuildContextTest.test_nested_idea_project_remains_requested_project_path \
  HumanOutputTest.test_proof_failure_overrides_plugin_green_verdict

Before the fix:

• nested .idea project paths inside a git repo collapsed to the parent git root, so exact-route verification could compare against the wrong target;
• plugin/helper GREEN won even when structured proof failures showed the result was not trustworthy.

After the fix:

• nested .idea project paths remain the requested project_path / exact route target;
• proof_failures override plugin-provided GREEN and classify the helper result as UNKNOWN.

Full helper validation currently passes:

uv run jetbrains-inspection/tests/test_jb_inspect.py
# Ran 113 tests ... OK

The agent-facing side is now covered by a Code exec harness scenario in code-prealign-new-skills:

python3 tools/code-exec-harness/harness.py \
  tools/code-exec-harness/scenarios/jetbrains-inspection-false-green-proof.json \
  --inherit-auth

Passing evidence:

failures: []
run_dir: .tmp/code-exec-harness/20260620-153934-jetbrains-inspection-false-green-proof

Important token-bloat note: the helper should compute compact verdict evidence (verdict, verdict_reason, proof_failures, small proof summary). Bulky IDE/capture diagnostics should stay in artifacts/logs unless the user or a debugging flow asks for them.

[cbusillo]

Cross-link update:

• Define compact inspection proof evidence contract for agent-facing verdicts jetbrains-inspection-api#114 owns the compact proof evidence contract this helper should consume.
• Add exec-harness false-green fixture for contradictory GREEN/0 inspection proof code#431 owns the Code exec-harness fixture that proves agents classify contradictory GREEN/0 evidence as UNKNOWN.
• WebStorm red-lane proof fails to confirm RED and exposes false-green route risk jetbrains-inspection-api#113 remains the product/red-lane parent risk.

This issue owns the helper/skill behavior: route/session/profile/scope/capture contradictions must stay non-clean and compact proof must override false green states.

[cbusillo]

Destination correction for the exec-harness fixture:

• cbusillo/code is retired, so Add exec-harness false-green fixture for contradictory GREEN/0 inspection proof code#431 should be treated as historical/staging context only.
• cbusillo/codex-lab#155 is the durable destination issue for importing the false-green harness fixture from local staging at ../code-prealign-new-skills once codex-lab is PR-ready.

[cbusillo]

Review follow-up update:

The helper branch fix/jb-inspection-false-green-proof was reviewed and tightened after review findings.

Additional fixes now included:

• proof_failures is copied through the real helper summary path, not only handled when already top-level.
• route_sort_key() now prefers exact_route_path for nested .idea project contexts instead of ranking the parent git root as exact.
• route diagnostics via flat_project_matches_context() now match exact_route_path/project_path before worktree_root.
• regression coverage was added for summary propagation, exact nested route ranking, and nested project diagnostic matching.

Current validation:

uv run jetbrains-inspection/tests/test_jb_inspect.py
# Ran 116 tests ... OK

The Code exec harness fixture was also strengthened: the fake helper no longer prints a stderr spoiler telling the agent to choose UNKNOWN, and the scenario still passes from compact proof fields and policy alone.

Latest passing harness evidence:

python3 tools/code-exec-harness/harness.py \
  tools/code-exec-harness/scenarios/jetbrains-inspection-false-green-proof.json \
  --inherit-auth
# failures: []
# run_dir: .tmp/code-exec-harness/20260620-163457-jetbrains-inspection-false-green-proof

[cbusillo]

Final verification update:

A final read-only review agent checked the current fix/jb-inspection-false-green-proof helper branch after the propagation/routing fixes and reported:

no blocking findings

Current validations:

uv run jetbrains-inspection/tests/test_jb_inspect.py
# Ran 116 tests ... OK

Agent-facing harness proof in staging also passes after removing the fake helper's stderr spoiler:

python3 tools/code-exec-harness/harness.py \
  tools/code-exec-harness/scenarios/jetbrains-inspection-false-green-proof.json \
  --inherit-auth
# failures: []
# run_dir: .tmp/code-exec-harness/20260620-163809-jetbrains-inspection-false-green-proof

Plain-English state: the helper now treats contradictory proof as UNKNOWN, not GREEN, and nested .idea projects inside a git repo remain the exact route target instead of silently collapsing to the parent checkout.

[cbusillo]

Final cross-repo status: this helper/skill hardening workstream is complete.

What landed on the helper side:

• PR Harden JetBrains inspection proof verdicts #390 hardened jb-inspect.py so contradictory proof cannot be accepted as GREEN.
• Nested .idea project paths inside a git repo remain exact route targets instead of silently collapsing to the parent checkout.
• The staged Code exec-harness false-green fixture continued to pass without relying on long diagnostic dumps or prompt spoilers.

Product/API follow-through is also complete:

• Define compact inspection proof evidence contract for agent-facing verdicts jetbrains-inspection-api#114 / PR Add rollout friction skill #115 implemented the compact proof contract that the helper consumes.
• WebStorm red-lane proof fails to confirm RED and exposes false-green route risk jetbrains-inspection-api#113 / PR Detect rollout friction workflow signals #116 fixed the remaining WebStorm red-lane empty-model path.
• Post-merge validation from main proved WebStorm red-lane now returns actionable RED evidence:

plugin_build_fingerprint=91ace601dbf7-clean
status=ok
bucket=red_confirmed
verdict=RED
total=2
cleanup=closed

The remaining harness import is tracked separately in cbusillo/codex-lab#155 because cbusillo/code is retired and codex-lab is the durable destination once it is PR-ready.

[cbusillo]

Closing as complete: helper PR #390 is merged, product proof contract/fix landed via cbusillo/jetbrains-inspection-api#115/#116, and merged-main WebStorm red-lane proof now returns RED total=2 cleanup=closed.