Skip to content

test(core-tools): comprehensive test suite for check_cisa_kev, get_nvd_data, get_cwe_details, get_otx_cve_details - #89

Open
manus-use wants to merge 1 commit into
mainfrom
feat/core-tools-test-suite
Open

test(core-tools): comprehensive test suite for check_cisa_kev, get_nvd_data, get_cwe_details, get_otx_cve_details#89
manus-use wants to merge 1 commit into
mainfrom
feat/core-tools-test-suite

Conversation

@manus-use

Copy link
Copy Markdown
Owner

test(core-tools): comprehensive test suite for check_cisa_kev, get_nvd_data, get_cwe_details, get_otx_cve_details

Motivation

Every manus-agent analyze invocation exercises a fixed pipeline:

get_nvd_data → check_cisa_kev → get_otx_cve_details → get_cwe_details

These four tools are the foundation of the entire VA agent — they are called in steps 1, 2, 3, and the CWE lookup step of every vulnerability analysis. Despite this central role, they had zero dedicated test files before this PR.

What this PR adds

tests/test_core_tools.py104 fully-mocked unit tests (no real HTTP calls).

Tool Tests Coverage areas
check_cisa_kev 32 import, TOOL_SPEC contract, input validation (empty/non-string/None), KEV found/not-found, case-insensitive CVE lookup, cache miss/hit/expired/write-back, network error degradation, empty catalog
get_nvd_data 17 import, TOOL_SPEC, input validation (no CVE- prefix/non-string/None), successful lookup, CVE ID uppercased in URL, cisa_kev_info injection, empty vulnerabilities, RequestException, JSONDecodeError, generic exception, HTTP 4xx
get_cwe_details 18 import, TOOL_SPEC, CWE prefix validation, non-numeric suffix validation, HTML parse (full/no-extended/missing description div), URL uses numeric ID only, case-insensitive input, RequestException, HTTPError, generic exception
get_otx_cve_details 21 import, TOOL_SPEC, input validation, missing API key (env absent + config None), config raises → env fallback, successful pulse response, empty pulses, URL uppercased, X-OTX-API-KEY header, config key used when env absent, RequestException, HTTP 404→success, HTTP 5xx→error, JSONDecodeError, generic exception
TestToolSpecContract 16 all four tools satisfy name/description/inputSchema/required contract (parametrised)
TestViAgentWiring 6 all four tool names in SYSTEM_PROMPT; all four module paths importable; all four paths referenced in vi_agent source

Results

902 → 1006 passing  (+104 tests)
0 failures
0 real HTTP calls (100% mocked)
ruff check: 0 violations
ruff format: clean

Why these tests matter

  • Regression safety: any refactor of input validation, caching, or error handling in these tools now has a safety net
  • Living documentation: the tests document expected behaviour for invalid inputs, HTTP errors, and graceful degradation — none of which was written down anywhere
  • CI coverage gap closed: check_cisa_kev had a caching layer (CACHE_FILE) with three distinct states (miss / fresh hit / expired) that were completely untested

Open PRs checked (no overlap)

This PR adds test coverage for existing tools only — no new tool or CLI subcommand. Checked against all 21 currently open PRs to confirm zero overlap:

#51 (silent-patches), #53 (cve-timeline), #54 (version-range), #58 (vendor-response), #60 (poc-freshness), #64 (sbom-scan), #65 (temporal-priority), #67 (cluster-variants), #74 (epss-decay), #75 (exploit-maturity), #76 (assess/triage), #77 (report), #78 (diff-report), #79 (reachability), #80 (epss-watchlist), #82 (attack-surface), #83 (watch-alert), #85 (CLI integration tests — tests existing CLI commands, not these four base tools), #86 (patch-status), #87 (risk-score), #88 (vendor-response subcommand).

None of the above touch check_cisa_kev, get_nvd_data, get_cwe_details, or get_otx_cve_details.

…d_data, get_cwe_details, get_otx_cve_details

Add 104 fully-mocked unit tests covering the four foundation tools that
power every `manus-agent analyze` pipeline run but previously had zero
dedicated test files.

Coverage per tool:
- check_cisa_kev (32 tests): import, TOOL_SPEC, input validation, KEV
  found/not-found, case-insensitive lookup, caching (miss/hit/expired/write),
  network error degradation, empty catalog handling
- get_nvd_data (17 tests): import, TOOL_SPEC, input validation, successful
  lookup, cve_id uppercasing, cisa_kev_info injection, empty response,
  RequestException, JSONDecodeError, generic and HTTP errors
- get_cwe_details (18 tests): import, TOOL_SPEC, CWE-prefixed and numeric
  validation, HTML parse (description/extended/missing div), URL construction,
  case-insensitive input, RequestException, generic and HTTP errors
- get_otx_cve_details (21 tests): import, TOOL_SPEC, input validation,
  missing API key (env + config path), config fallback, successful pulse
  response, empty pulses, URL uppercasing, header auth verification,
  404→success, 5xx→error, JSONDecodeError, generic errors
- TOOL_SPEC contract (16 parametrised tests): all four tools satisfy
  name/description/inputSchema/required-fields contract
- VI agent wiring (6 tests): all four tool names present in SYSTEM_PROMPT;
  all four module paths importable; all four paths referenced in vi_agent source

Suite size: 902 → 1006 passing (+104), 0 failures, 0 real HTTP calls
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant