feat(fetch/openssl/secjson): add OpenSSL security advisory JSON fetcher

[MaineK00n]

What

Add a new fetch data source openssl-secjson that collects OpenSSL's security advisory JSON files from https://openssl-library.org/news/secjson/.

• pkg/fetch/openssl/secjson/ — fetcher + types + golden tests
• pkg/cmd/fetch/fetch.go — wire newCmdOpenSSLSecJSON()

Output layout:

<dir>/statements.json
<dir>/<year>/CVE-yyyy-nnnn.json

Why

OpenSSL publishes its own advisories as per-CVE records in the MITRE CVE Record Format (dataType: CVE_RECORD, dataVersion 5.0 / 5.1), i.e. the same schema as fetch/mitre/cve/v5, and as the CNA for its own CVEs it is the authoritative source for affected version ranges.

statements.json is fetched as well because its disputed entries (CVE-2002-20001, CVE-2007-6755, CVE-2010-0928) do not exist as per-CVE files — this is the only place in the dataset where "OpenSSL considers this not a vulnerability / not affected" is recorded, which maps naturally onto fixstatus.ClassNotAffected for a future extractor. Its statements entries are OpenSSL's own out-of-support declarations per version series, which map onto types/eol.

How

Types are copied from pkg/fetch/mitre/cve/v5/types.go (the same duplication fetch/paloalto/json already does for this schema), plus a small Statements / Statement type for statements.json.

Upstream offers no aggregate feed or index API — only the HTML index — so the fetcher scrapes it with goquery (main a), resolves hrefs via url.ResolveReference, and then:

• statements.json → <dir>/statements.json
• cve-*.json → PipelineGet → <dir>/<year>/CVE-yyyy-nnnn.json, year taken from cveMetadata.cveId exactly as mitre/cve/v5 does

Since the data set depends on statements.json for the disputed records, the index scan is strict rather than lenient: a missing statements.json link and an unrecognized JSON file name in the index are both errors. A quietly shrinking data set is worse than a failed run, and a new upstream file should be a deliberate decision, not a silent drop. Non-JSON links (e.g. the secadv/*.txt advisories) are ignored.

Defaults: retry=3, concurrency=5, wait=1s.

Testing

• GOEXPERIMENT=jsonv2 go build ./...
• GOEXPERIMENT=jsonv2 go test -race ./pkg/fetch/openssl/... — golden test with four cases, each error case verified to fail for its intended reason:
  • happy — a legacy x_generator.importer record, one with lessThanOrEqual + credits[].user + supportingMedia, one full Vulnogram record, statements.json, plus a non-JSON link
  • notfound — a listed record that 404s; also asserts the statements.json written before the failure is correct
  • no-statements — index that no longer links statements.json
  • unexpected — index carrying an unrecognized advisories.json
• GOEXPERIMENT=jsonv2 go vet, gofmt -l, golangci-lint run — clean
• Live run against the real site: vuls-data-update fetch openssl-secjson → 272 CVE records + statements.json
• Re-checked the live index after making the scan strict: it links exactly statements.json + 272 cve-*.json and nothing else, so the strict path does not trip on the current upstream page
• Lossless check: every one of the 273 written files was compared semantically (parsed-JSON equality) against the upstream file it came from — 0 mismatches, so no field of the OpenSSL data is dropped by the types
• The live run and a run against a local mirror of the same data produced byte-identical trees

Checklist

• Tests pass
• Golden files updated
• Deterministic output verified — all output goes through util.Write; two independent runs produced byte-identical trees
• Backward compatibility maintained — no changes under pkg/extract/types

Notes for reviewers

• No extractor in this PR; vuls-data-db keeps the extract-* matrix entries commented out (see feat(ci): add openssl-secjson vulsio/vuls-data-db#206).
• If an extractor is written later, note that statements[].base takes non-version sentinels (none, fips) in addition to version series like 0.9.8.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds a new fetch data source (openssl-secjson) to download OpenSSL’s per-CVE CVE JSON 5.x records plus statements.json, writing them into a deterministic on-disk layout under the fetch cache directory and wiring it into the Cobra command tree.

Changes:

• Implement pkg/fetch/openssl/secjson fetcher: scrape the HTML index, require statements.json, and fetch per-CVE JSON concurrently.
• Add OpenSSL secjson golden/fixture test data and a table-driven httptest-based fetcher test.
• Wire the new openssl-secjson subcommand into pkg/cmd/fetch/fetch.go.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
pkg/fetch/openssl/secjson/secjson.go	New fetcher implementation (index scrape + strict link handling + concurrent CVE downloads).
pkg/fetch/openssl/secjson/types.go	CVE JSON 5.x types (plus Statements types) used for lossless JSON decoding/encoding.
pkg/fetch/openssl/secjson/secjson_test.go	Golden-style fetcher test using httptest fixtures.
pkg/cmd/fetch/fetch.go	Registers openssl-secjson command and hooks it into vuls-data-update fetch.
pkg/fetch/openssl/secjson/testdata/fixtures/happy/indexof.html	HTML fixture for a successful index scrape.
pkg/fetch/openssl/secjson/testdata/fixtures/happy/statements.json	JSON fixture for statements.json.
pkg/fetch/openssl/secjson/testdata/fixtures/happy/cve-2002-0655.json	JSON fixture for a legacy CVE record.
pkg/fetch/openssl/secjson/testdata/fixtures/happy/cve-2022-3996.json	JSON fixture for a CVE record with lessThanOrEqual and credits[].user.
pkg/fetch/openssl/secjson/testdata/fixtures/happy/cve-2024-6119.json	JSON fixture for a newer CVE record with supportingMedia, etc.
pkg/fetch/openssl/secjson/testdata/fixtures/notfound/indexof.html	HTML fixture where a listed CVE link 404s.
pkg/fetch/openssl/secjson/testdata/fixtures/notfound/statements.json	JSON fixture used for the “notfound” test case.
pkg/fetch/openssl/secjson/testdata/fixtures/no-statements/indexof.html	HTML fixture missing statements.json (expected error).
pkg/fetch/openssl/secjson/testdata/fixtures/unexpected/indexof.html	HTML fixture with an unexpected JSON filename (expected error).
pkg/fetch/openssl/secjson/testdata/golden/happy/statements.json	Golden output for statements.json in the happy path.
pkg/fetch/openssl/secjson/testdata/golden/happy/2002/CVE-2002-0655.json	Golden output for CVE-2002-0655.
pkg/fetch/openssl/secjson/testdata/golden/happy/2022/CVE-2022-3996.json	Golden output for CVE-2022-3996.
pkg/fetch/openssl/secjson/testdata/golden/happy/2024/CVE-2024-6119.json	Golden output for CVE-2024-6119.
pkg/fetch/openssl/secjson/testdata/golden/notfound/statements.json	Golden output asserting statements.json is written before the 404 failure.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated no new comments.

Suppressed comments (2)

pkg/fetch/openssl/secjson/secjson_test.go:70

• These subtests exercise multiple distinct failure modes (404, missing statements link, unexpected JSON filename, invalid CVE ID), but the test currently only asserts that an error occurred. Adding a small check that the error contains an expected substring per case would ensure each fixture fails for the intended reason (and prevents false positives where an unrelated error happens to occur).

			switch {
			case err != nil && !tt.hasError:
				t.Error("unexpected error:", err)
			case err == nil && tt.hasError:
				t.Error("expected error has not occurred")
			default:

pkg/fetch/openssl/secjson/secjson.go:210

• The expected filename pattern reported in this error message doesn’t match the actual acceptance check (strings.HasPrefix(b, "cve-")). As written, the message implies only cve-<year>-<digits>.json is allowed, but the code will accept any cve-*.json. Aligning the message with the predicate will make failures easier to diagnose.

				return "", nil, errors.Errorf("unexpected json file name. expected: %q, actual: %q", []string{statementsFilename, "cve-yyyy-\\d{4,}.json"}, b)

[shino]

🍻