risk_assessment can say LOW/SAFE while issues[] contains a HIGH finding, with nothing in the report flagging the disagreement

[Mark2Mac]

Summary

risk_assessment.severity / .recommendation can read LOW / SAFE on a report whose
issues[] contains a HIGH finding, and nothing in the report says the two disagree. A
consumer that reads the summary — which is what a summary is for — gets the opposite of what the
findings say.

Reproduction

mkdir -p /tmp/verdict/skill
cat > /tmp/verdict/skill/SKILL.md <<'EOF'
---
name: skill
description: minimal case.
---
keyring
EOF

uvx --from 'git+https://github.com/NVIDIA/skillspector' skillspector scan /tmp/verdict/skill --no-llm --format json -o /tmp/verdict/out.json
jq '{severity: .risk_assessment.severity, recommendation: .risk_assessment.recommendation, issues: [.issues[] | "\(.id)/\(.severity)"]}' /tmp/verdict/out.json

v2.9.6 (git HEAD):

{
  "severity": "LOW",
  "recommendation": "SAFE",
  "issues": ["PE3/HIGH"]
}

(The PE3 finding itself is #396 — a separate problem. This issue is about the summary, and it
reproduces with any single HIGH finding that normalisation smooths away.)

Why this is worse than a noisy rule

A false positive costs the reader a minute. A summary that contradicts the findings costs the
reader their attention permanently, and in a predictable direction:

• Someone who reads only the headline installs a skill that has a HIGH finding.
• Someone who reads the findings learns that the headline is unreliable, and stops reading it.

Both outcomes are worse than either signal alone. I hit this building a pre-install gate: the gate
keyed on risk_assessment.severity and returned "install" on skills whose issues[] contained
HIGH — measured on a corpus of deliberately malicious fixtures, 0 of 5 were stopped. Keying on the
worst finding instead took it to 2 of 5 (static only). The remaining gap is detection, which is
fair; the first gap was the tool telling me SAFE.

What I am not asking for

Not "always report the max". Normalisation clearly exists for a reason — with rules that fire on
documentation prose, a max-severity summary would read HIGH on almost everything, and that is its
own kind of useless. The problem is not the smoothing, it is that the smoothing is silent.

Suggested direction

Expose the disagreement instead of resolving it invisibly. Something like:

"risk_assessment": {
  "severity": "LOW",
  "recommendation": "SAFE",
  "max_issue_severity": "HIGH",
  "normalisation_applied": true
}

That is enough for a consumer to make its own policy — a CI gate can key on
max_issue_severity, a dashboard can show the normalised value, and neither has to re-derive it
by walking issues[] and re-implementing the severity ranking. Today every consumer that wants
to be safe has to do exactly that, which means every consumer implements it slightly differently.

A one-line note in the human-readable output would help too: today the terminal report prints
SAFE and then lists a [HIGH] line below it, with nothing connecting the two.

Environment: v2.9.6 (git HEAD) and v2.5.1 (PyPI), Python 3.13, Linux, --no-llm.