Skip to content

osv_client silently drops 90% of vulnerabilities on packages with many advisories #386

Description

@Mark2Mac

Title: osv_client silently drops 90% of vulnerabilities on packages with many advisories

Summary

_fetch_vuln_details() hard-caps at 10 vulnerability IDs (vuln_ids[:10]) and reports the
truncation only through logger.warning. The dropped advisories leave no trace in the report:
analysis_completeness has no ledger event for them, so a consumer reading the JSON cannot
tell that 90% of the input was never examined.

https://github.com/NVIDIA/skillspector/blob/main/src/skillspector/nodes/analyzers/osv_client.py#L200-L206

def _fetch_vuln_details(vuln_ids: list[str]) -> list[VulnResult]:
    if len(vuln_ids) > 10:
        logger.warning("Processing 10 of %d vulnerabilities, truncating the rest", len(vuln_ids))
    results: list[VulnResult] = []
    with httpx.Client(timeout=_REQUEST_TIMEOUT) as client:
        for vid in vuln_ids[:10]:

Still present on main as of 2.9.5.

Measured impact

Across 91 scan reports produced on one workstation (Claude Code skill marketplaces):

truncation events 16
advisories examined 160
advisories silently discarded 323

That is 67% of the advisories for the affected units, on a scan that reports
execution_successful: true and no ledger exception.

Worst single case observed: Processing 10 of 153 vulnerabilities, truncating the rest
— 143 advisories dropped for one unit.

Why this matters beyond the cap itself

The cap is a defensible engineering choice (rate limits, latency). The problem is that it is
invisible in the artifact. SkillSpector already has exactly the right mechanism for this —
InspectionLedger with LedgerReason — and it is used for SIZE_LIMIT, BINARY_CONTENT,
READ_ERROR and others. A tool that carefully accounts for unread files and then silently
drops two thirds of the advisories is inconsistent with its own design.

Suggested fix

Emit a ledger event when the cap trips, so the truncation surfaces in
analysis_completeness.ledger_exceptions and is visible to any report consumer:

if len(vuln_ids) > _MAX_VULN_DETAILS:
    ledger_event(
        outcome=LedgerOutcome.SKIPPED,
        phase="static",
        analyzer_id="supply_chain",
        reason=LedgerReason.SIZE_LIMIT,   # or a new TRUNCATED reason
        observed_count=len(vuln_ids),
        limit_count=_MAX_VULN_DETAILS,
    )

Making the cap configurable (SKILLSPECTOR_OSV_MAX_VULNS) would be welcome too, but the
reporting gap is the part that changes conclusions: right now a scan can be 67% blind on
vulnerability detail and still look complete.

Environment

SkillSpector 2.5.1 (pinned) and main @ 2.9.5, Linux, Python 3.13.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions