feat(extract/fortinet/cvrf): supplement detection for historical advisories without product_statuses

[shino]

What

Adds an embedded supplement table to extract fortinet-cvrf so the historical CVRF advisories (2012–2022) whose product_statuses/product_tree are empty upstream still produce detection. 425 of the 465 statuses-empty advisories gain detection (680 product rows); with this, 1,068 of the 1,119 corpus files carry detection (previously 643).

Why

Fortinet only started populating product_statuses consistently in 2022, and CSAF only exists for recent advisories (effectively 2024+), so the 2012–2022 corpus extracted as content-only — a detection gap for exactly the advisories the retired handmade dataset used to cover.

How

• pkg/extract/fortinet/cvrf/supplement_data.go — a frozen, generated Go-literal table (same pattern as microsoft/bulletin's compiled-in archive amendments) keyed by advisory ID: per product either exact versions, ranges, or neither (= whole product). Generated once from two Fortinet-authored sources and then hand-reviewed:
  • the curated version ranges of the legacy vuls-data-raw-fortinet (handmade) dataset — precise for 2012–2018, but its 2019+ entries are mostly constraint-less and sometimes list not-impacted products, so only version-constrained nodes are used;
  • the affected-version data of Fortinet's own CNA records in cvelistV5 (containers.cna.affected, assignerShortName: fortinet only) — covers 2016–2022, exactly where handmade degrades.
  • Per advisory the sources merge at product level (handmade wins per product, CNA fills missing products); disagreements and the 40 advisories left content-only (third-party CVEs, hardware-model-only listings, informational advisories) were reviewed individually.
• supplement.go — builds criterions from the table in the same shapes the existing paths emit (exact versions → CPEMatches like the Known Affected path; ranges → per-product range type like the CSAF extractor), with the same whitelist + hard-error policy. cvrf.go falls back to it only when product_statuses is empty.
• CPEs come exclusively from the existing internal/product whitelist (no CPEs are taken from cvelistV5), so the CPE vocabulary is identical to the CSAF/CVRF paths. 14 historical products are added to the table (CPE slugs from the legacy handmade curation) with per-product range types.
• Normalizes vulnerability cve[] entries to bare CVE IDs — FG-IR-14-010 suffixes free text and embeds a stray HTML tag (CVE-<br />2014-2722 key issue), which previously became the vulnerability ID verbatim.
• Adds FortiSIEMWindowsAgent (product tree branch of the new FG-IR-26-155) to the whitelist, which the current raw corpus requires to extract at all.

The gap is historical and closed going forward (2023+ statuses-empty files are third-party/informational only), so the table is a frozen asset, not a maintained feed.

Testing

• GOEXPERIMENT=jsonv2 go test ./... passes.
• New golden fixtures cover the three row shapes: range (FG-IR-012-001), CNA enumeration (FG-IR-21-085), whole-product (FG-IR-14-010); existing fixtures FG-IR-012-003 / FG-IR-13-008 now gain detection.
• TestSupplementCriterions walks every embedded table entry and asserts it builds cleanly against the product table and numeric version scheme.
• Full-corpus end-to-end: extract fortinet-cvrf over the current vuls-data-raw-fortinet-cvrf (1,119 files) completes without error.

The table generator (inputs fetch + generation + cross-source arbitration log) is published as a gist: https://gist.github.com/shino/5cefdef6d36283f50795e32ffcf1a51d

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR enhances the extract fortinet-cvrf pipeline to restore CPE detection for historical Fortinet CVRF advisories (2012–2022) where upstream product_statuses / product_tree are empty, by falling back to an embedded, frozen supplement table. It also normalizes messy cve[] entries into proper CVE IDs and expands the Fortinet product whitelist/range-type vocabulary to support additional historical products.

Changes:

• Add embedded supplement.json + builder (supplement.go) and wire CVRF extraction to use it only when product_statuses.status.type is empty.
• Normalize CVRF vulnerability.cve[] entries to bare CVE IDs via regex extraction (handling the known <br /> artifact).
• Extend Fortinet product whitelist and Fortinet per-product range types; update/add golden + fixture coverage and add a whole-table validation test.

Reviewed changes

Copilot reviewed 14 out of 15 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
pkg/extract/types/data/detection/condition/criteria/criterion/cpecriterion/range/range.go	Adds new Fortinet per-product RangeType constants + JSON/string mappings + Compare dispatch entries.
pkg/extract/fortinet/internal/product/table.go	Extends the Fortinet product-name → (CPE, RangeType) whitelist to include additional historical products.
pkg/extract/fortinet/cvrf/supplement.go	Implements embedded supplement-table loading and conversion into detection criterions (exact versions / ranges / whole-product).
pkg/extract/fortinet/cvrf/supplement.json	Frozen generated advisory→affected-product table used to supplement historical statuses-empty CVRF advisories.
pkg/extract/fortinet/cvrf/cvrf.go	Falls back to supplement detection when product_statuses type is empty; normalizes CVE IDs using cvePattern.
pkg/extract/fortinet/cvrf/export_test.go	Exposes supplement helpers to _test package for table-wide validation.
pkg/extract/fortinet/cvrf/cvrf_test.go	Adds TestSupplementCriterions to ensure every supplement entry builds cleanly and unknown IDs return nil.
pkg/extract/fortinet/cvrf/testdata/golden/data/2012/FG-IR-012-001.json	New/updated golden to cover supplemented range-shaped detection output.
pkg/extract/fortinet/cvrf/testdata/golden/data/2012/FG-IR-012-003.json	Updated golden to reflect newly supplemented detection output.
pkg/extract/fortinet/cvrf/testdata/golden/data/2013/FG-IR-13-008.json	Updated golden to reflect newly supplemented detection output (mixed exact/range/whole-product cases).
pkg/extract/fortinet/cvrf/testdata/golden/data/2014/FG-IR-14-010.json	New golden validating whole-product detection + CVE ID normalization from messy cve[] strings.
pkg/extract/fortinet/cvrf/testdata/golden/data/2021/FG-IR-21-085.json	New golden validating supplemented enumeration-shaped detection output.
pkg/extract/fortinet/cvrf/testdata/fixtures/2012/FG-IR-012-001.json	Adds fixture exercising statuses-empty advisory structure.
pkg/extract/fortinet/cvrf/testdata/fixtures/2014/FG-IR-14-010.json	Adds fixture containing the known malformed CVE strings (free text + <br />).
pkg/extract/fortinet/cvrf/testdata/fixtures/2021/FG-IR-21-085.json	Adds fixture for a statuses-empty advisory that gains supplemented detection.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

pkg/extract/fortinet/cvrf/supplement.go:109

• supplementCriterions currently validates that bounds are parseable, but it does not validate that a constructed range is non-contradictory (e.g. GreaterEqual > LessEqual, or equal bounds with an exclusive end). This allows mistakes in the frozen supplement table to silently produce empty ranges and therefore silently drop detection.

		for _, sr := range row.Ranges {
			r := ccRangeTypes.Range{
				Type:         rt,
				GreaterEqual: sr.GreaterEqual,
				GreaterThan:  sr.GreaterThan,
				LessEqual:    sr.LessEqual,
				LessThan:     sr.LessThan,
			}
			for _, b := range []string{r.GreaterEqual, r.GreaterThan, r.LessEqual, r.LessThan} {
				if b == "" {
					continue
				}
				if _, err := numericVersion.NewVersion(b); err != nil {
					return nil, errors.Wrapf(err, "unexpected range bound %q for %q in supplement entry %q", b, row.Product, id)
				}
			}

[Copilot]

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

pkg/extract/fortinet/cvrf/supplement.go:106

• supplementCriterions validates that range bounds parse as numeric, but it doesn’t validate that the bounds form a sensible range (e.g., lower <= upper) or that mutually-exclusive fields (ge+gt, le+lt) aren’t both set. Since the frozen supplement table already contains inverted ranges, adding these checks would make bad table edits fail loudly and prevent silent false negatives.

			for _, b := range []string{r.GreaterEqual, r.GreaterThan, r.LessEqual, r.LessThan} {
				if b == "" {
					continue
				}
				if _, err := numericVersion.NewVersion(b); err != nil {

[Copilot]

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

pkg/extract/fortinet/cvrf/supplement.go:106

• supplementCriterions validates that bounds parse as numeric versions, but it doesn’t validate that each range is logically satisfiable (lower bound <= upper bound). The current supplement table already contains inverted bounds, which will silently produce criterions that never match. Consider adding an explicit lower/upper ordering check here and failing extraction when a table entry is inconsistent.

				LessThan:     sr.LessThan,
			}
			for _, b := range []string{r.GreaterEqual, r.GreaterThan, r.LessEqual, r.LessThan} {
				if b == "" {
					continue

[Copilot]

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated no new comments.