feat(fetch/nvd/api/cvehistory): add CVE change history fetcher by MaineK00n · Pull Request #899 · MaineK00n/vuls-data-update

MaineK00n · 2026-07-30T06:00:26Z

What

Add vuls-data-update fetch nvd-api-cvehistory, a fetcher for the NVD CVE Change History API 2.0.

Each change event is stored in its own file:

<dir>/<year>/<CVE ID>/<cveChangeId>.json

Why

The NVD CVE API exposes only the current state of a CVE. The change history tells when and by whom each field (CVSS, CWE, references, CPE configuration, affected, tags, ...) was added, changed or removed, which the CVE API cannot answer.

How

Follows pkg/fetch/nvd/api/cve:

Preliminary API call with resultsPerPage=1 to get totalResults, then page through with startIndex. resultsPerPage max is 5,000 for this API (the CVE API is 2,000).
Reuses nvdutil.CheckRetry / nvdutil.Backoff (408, Retry-After, truncated body) and the same per-page result count sanity check, including the "last page grew during execution" allowance.
--change-start-date / --change-end-date are validated as a pair and rejected when the range exceeds 120 days, like lastMod* in the CVE fetcher.

One point where the response deviates from the sibling fetchers: details[].oldValue / newValue are usually strings, but the Affected type carries CVE v5 affected structures as an array of objects (the schema allows string, array and object). They are typed as jsontext.Value so the response is stored as is, without coercing numbers through float64 or reordering object members.

Storing one file per change event (rather than one array per CVE) keeps the fetch fully streaming: a CVE's change events are scattered across pages, so grouping them would mean either holding all ~2.2M events in memory or spilling them to disk first.

Also included

While reviewing the retry wiring of this fetcher, the same defect turned up in the sibling packages, so it is fixed here too (fcbd113):

pkg/fetch/nvd/api/{cve,cpe,cpematch} multiply retryWaitMin/retryWaitMax by time.Second even though #608 changed those options to time.Duration. That turns the 6s/30s defaults into ~190/~950 years, and since nvdutil.Backoff derives its sleep from min (clamped to max), the first retry hangs the fetch. All four NVD API fetchers now pass the durations as is.

Testing

GOEXPERIMENT=jsonv2 go test ./... — pass
GOEXPERIMENT=jsonv2 go vet ./..., gofmt -l — clean
New fetch tests cover: empty result, single item, precisely paged, multiple pages, totalResults growing mid-execution, API key header, change date range, 120 day range violation, and non-string oldValue/newValue (fixture built from a real Affected change event).
Manual run against the real API with a 2 hour change window: 8 change events written to the expected paths.

Note: a full run is ~2.2M change events over ~445 requests, so it is slow (deep startIndex responses take up to ~80s each). Narrowing with changeStartDate/changeEndDate was measured to be slower per request, so plain startIndex paging is used.

Checklist

Tests pass
Golden files updated (if applicable)
Deterministic output verified (if types changed)
Backward compatibility maintained (if types/data changed) — new package only, no existing types touched

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Adds a new NVD data source fetcher for the CVE Change History API 2.0, integrated into the Cobra fetch command tree, with fixtures/golden tests to validate paging behavior and preservation of non-string oldValue/newValue JSON.

Changes:

Introduce pkg/fetch/nvd/api/cvehistory fetcher that pages through the CVE History API and stores one change-event per file under <dir>/<year>/<CVE ID>/<cveChangeId>.json.
Wire the new fetcher into vuls-data-update fetch nvd-api-cvehistory with optional change-date range and API key support.
Add table-driven fetch tests plus fixture and golden datasets (including non-string detail values).

Reviewed changes

Copilot reviewed 45 out of 45 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
pkg/fetch/nvd/api/cvehistory/types.go	Defines API response and persisted `Change`/`Detail` types (including raw JSON values).
pkg/fetch/nvd/api/cvehistory/cvehistory.go	Implements the CVE History API paging fetcher and file-per-event storage layout.
pkg/fetch/nvd/api/cvehistory/cvehistory_test.go	Adds fetcher tests using httptest + fixture/golden comparisons.
pkg/cmd/fetch/fetch.go	Registers the new `nvd-api-cvehistory` subcommand and flags.
pkg/fetch/nvd/api/cvehistory/testdata/golden/nonstring_values/2025/CVE-2025-5914/5A88AA91-2A39-400B-8CEA-3B7871636C68.json	Golden output for non-string `oldValue`/`newValue` (array/object).
pkg/fetch/nvd/api/cvehistory/testdata/golden/increase/2024/CVE-2024-6488/F76168A4-D7A4-4C62-9555-758EAAC5650D.json	Golden output for “totalResults increases” scenario.
pkg/fetch/nvd/api/cvehistory/testdata/golden/increase/2024/CVE-2024-6488/F5BA0B1F-86AB-4B18-A698-3AE52EDDBD09.json	Golden output for “totalResults increases” scenario.
pkg/fetch/nvd/api/cvehistory/testdata/golden/increase/2024/CVE-2024-6488/CC438820-317D-46E7-8D4F-9857F634EB6B.json	Golden output for “totalResults increases” scenario.
pkg/fetch/nvd/api/cvehistory/testdata/golden/increase/2024/CVE-2024-45342/9F98E250-EEF7-4FC7-A58E-4057DC798FEA.json	Golden output for “totalResults increases” scenario.
pkg/fetch/nvd/api/cvehistory/testdata/golden/increase/2024/CVE-2024-45342/0147E440-12A8-405E-BF5A-C937B2FF8D8F.json	Golden output for “totalResults increases” scenario.
pkg/fetch/nvd/api/cvehistory/testdata/golden/increase/2024/CVE-2024-2567/386BB84A-837B-4FB2-ADCC-A3EBCD964566.json	Golden output for “totalResults increases” scenario.
pkg/fetch/nvd/api/cvehistory/testdata/golden/increase/2022/CVE-2022-23815/91FE9AB8-2F77-49A0-B634-98495FCB844A.json	Golden output for “totalResults increases” scenario.
pkg/fetch/nvd/api/cvehistory/testdata/golden/increase/2017/CVE-2017-16532/BE9AAB08-C7AF-4C04-BB34-ECA3D2368C54.json	Golden output for “totalResults increases” scenario.
pkg/fetch/nvd/api/cvehistory/testdata/golden/changedate/2024/CVE-2024-6488/F76168A4-D7A4-4C62-9555-758EAAC5650D.json	Golden output for change-date filtering behavior.
pkg/fetch/nvd/api/cvehistory/testdata/golden/changedate/2024/CVE-2024-6488/F5BA0B1F-86AB-4B18-A698-3AE52EDDBD09.json	Golden output for change-date filtering behavior.
pkg/fetch/nvd/api/cvehistory/testdata/golden/changedate/2024/CVE-2024-6488/CC438820-317D-46E7-8D4F-9857F634EB6B.json	Golden output for change-date filtering behavior.
pkg/fetch/nvd/api/cvehistory/testdata/golden/3_pages/2024/CVE-2024-6488/F76168A4-D7A4-4C62-9555-758EAAC5650D.json	Golden output for multi-page paging behavior.
pkg/fetch/nvd/api/cvehistory/testdata/golden/3_pages/2024/CVE-2024-6488/F5BA0B1F-86AB-4B18-A698-3AE52EDDBD09.json	Golden output for multi-page paging behavior.
pkg/fetch/nvd/api/cvehistory/testdata/golden/3_pages/2024/CVE-2024-6488/CC438820-317D-46E7-8D4F-9857F634EB6B.json	Golden output for multi-page paging behavior.
pkg/fetch/nvd/api/cvehistory/testdata/golden/3_pages/2024/CVE-2024-45342/9F98E250-EEF7-4FC7-A58E-4057DC798FEA.json	Golden output for multi-page paging behavior.
pkg/fetch/nvd/api/cvehistory/testdata/golden/3_pages/2024/CVE-2024-45342/0147E440-12A8-405E-BF5A-C937B2FF8D8F.json	Golden output for multi-page paging behavior.
pkg/fetch/nvd/api/cvehistory/testdata/golden/3_pages/2024/CVE-2024-2567/386BB84A-837B-4FB2-ADCC-A3EBCD964566.json	Golden output for multi-page paging behavior.
pkg/fetch/nvd/api/cvehistory/testdata/golden/3_pages/2022/CVE-2022-23815/91FE9AB8-2F77-49A0-B634-98495FCB844A.json	Golden output for multi-page paging behavior.
pkg/fetch/nvd/api/cvehistory/testdata/golden/3_pages/2017/CVE-2017-16532/BE9AAB08-C7AF-4C04-BB34-ECA3D2368C54.json	Golden output for multi-page paging behavior.
pkg/fetch/nvd/api/cvehistory/testdata/golden/3_items/2024/CVE-2024-6488/F76168A4-D7A4-4C62-9555-758EAAC5650D.json	Golden output for “precisely single page” scenario.
pkg/fetch/nvd/api/cvehistory/testdata/golden/3_items/2024/CVE-2024-45342/0147E440-12A8-405E-BF5A-C937B2FF8D8F.json	Golden output for “precisely single page” scenario.
pkg/fetch/nvd/api/cvehistory/testdata/golden/3_items/2022/CVE-2022-23815/91FE9AB8-2F77-49A0-B634-98495FCB844A.json	Golden output for “precisely single page” scenario.
pkg/fetch/nvd/api/cvehistory/testdata/golden/1_item/2022/CVE-2022-23815/91FE9AB8-2F77-49A0-B634-98495FCB844A.json	Golden output for single-item scenario.
pkg/fetch/nvd/api/cvehistory/testdata/fixtures/nonstring_values/0-3.json	Fixture response containing array/object detail values.
pkg/fetch/nvd/api/cvehistory/testdata/fixtures/nonstring_values/0-1.json	Fixture response for preliminary (resultsPerPage=1) call.
pkg/fetch/nvd/api/cvehistory/testdata/fixtures/increase/6-3.json	Fixture response for “totalResults increases” scenario (later page).
pkg/fetch/nvd/api/cvehistory/testdata/fixtures/increase/3-3.json	Fixture response for “totalResults increases” scenario (mid page).
pkg/fetch/nvd/api/cvehistory/testdata/fixtures/increase/0-3.json	Fixture response for “totalResults increases” scenario (first page).
pkg/fetch/nvd/api/cvehistory/testdata/fixtures/increase/0-1.json	Fixture response for “totalResults increases” scenario preliminary call.
pkg/fetch/nvd/api/cvehistory/testdata/fixtures/empty/0-1.json	Fixture response for empty result set.
pkg/fetch/nvd/api/cvehistory/testdata/fixtures/changedate/0-3.json	Fixture response for change-date filter testing (paged).
pkg/fetch/nvd/api/cvehistory/testdata/fixtures/changedate/0-1.json	Fixture response for change-date filter testing (preliminary).
pkg/fetch/nvd/api/cvehistory/testdata/fixtures/3_pages/6-3.json	Fixture response for multi-page testing (later page).
pkg/fetch/nvd/api/cvehistory/testdata/fixtures/3_pages/3-3.json	Fixture response for multi-page testing (mid page).
pkg/fetch/nvd/api/cvehistory/testdata/fixtures/3_pages/0-3.json	Fixture response for multi-page testing (first page).
pkg/fetch/nvd/api/cvehistory/testdata/fixtures/3_pages/0-1.json	Fixture response for multi-page testing preliminary call.
pkg/fetch/nvd/api/cvehistory/testdata/fixtures/3_items/0-3.json	Fixture response for precisely single page testing.
pkg/fetch/nvd/api/cvehistory/testdata/fixtures/3_items/0-1.json	Fixture response for precisely single page preliminary call.
pkg/fetch/nvd/api/cvehistory/testdata/fixtures/1_item/0-3.json	Fixture response for single item testing.
pkg/fetch/nvd/api/cvehistory/testdata/fixtures/1_item/0-1.json	Fixture response for single item preliminary call.

Comments suppressed due to low confidence (1)

pkg/fetch/nvd/api/cvehistory/cvehistory_test.go:83

Same issue here: new(time.Date(...)) is not valid Go and prevents tests from compiling.

			name: "date range exceeds 120 days",
			args: []cvehistory.Option{
				cvehistory.WithChangeStartDate(new(time.Date(2024, time.July, 1, 0, 0, 0, 0, time.UTC))),
				cvehistory.WithChangeEndDate(new(time.Date(2025, time.January, 1, 0, 0, 0, 0, time.UTC))),
			},

- retryWaitMin/retryWaitMax are already time.Duration, so multiplying them by time.Second turned the 6s/30s defaults into ~190/~950 years, hanging the fetch on the first retry - validate cveId/cveChangeId against the schema patterns before using them as path elements, so a malformed response cannot write outside the output dir - fix the stale Change comment: one change event is stored per file, not []Change Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 47 out of 47 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

pkg/fetch/nvd/api/cvehistory/cvehistory_test.go:107

The "With API Key" test case doesn't currently assert that the fetcher actually sends the apiKey request header, so it can pass even if header propagation breaks. Add an explicit header check in the httptest handler for that case (and return after writing the error).

				resultsPerPage := "5000"
				if value := r.URL.Query().Get("resultsPerPage"); value != "" {
					resultsPerPage = value
				}

pkg/fetch/nvd/api/cvehistory/cvehistory_test.go:113

In the change-date handler branch, if os.Open fails you call http.Error but then still defer f.Close(). Since f will be nil, this can panic and obscure the real failure. Return immediately after sending the error response.

					f, err := os.Open(filepath.Join("testdata", "fixtures", tt.fixturePrefix, fmt.Sprintf("%s-%s.json", startIndex, resultsPerPage)))
					if err != nil {
						http.Error(w, fmt.Sprintf("Internal Server Error. err: %s", err), http.StatusInternalServerError)
					}
					defer f.Close()

retryWaitMin/retryWaitMax became time.Duration in #608, but the client setup kept multiplying them by time.Second, turning the 6s/30s defaults into ~190/~950 years. nvdutil.Backoff derives its sleep from min (clamped to max), so the first retry would hang the fetch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 50 out of 50 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

pkg/fetch/nvd/api/cvehistory/cvehistory_test.go:105

The "With API Key" test case sets WithAPIKey("foobar"), but the httptest handler never asserts that the request actually includes the apiKey header. As written, this test will still pass even if the header wiring breaks.

			ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
				startIndex := "0"
				if value := r.URL.Query().Get("startIndex"); value != "" {
					startIndex = value
				}

pkg/fetch/nvd/api/cvehistory/cvehistory_test.go:113

In the httptest handler, several http.Error(...) branches don’t return. For example, if os.Open fails, the handler will still defer f.Close() on a nil file and continue processing, which can panic and produce confusing test failures. Return immediately after writing an error response (and consider doing the same for the other error branches below).

					f, err := os.Open(filepath.Join("testdata", "fixtures", tt.fixturePrefix, fmt.Sprintf("%s-%s.json", startIndex, resultsPerPage)))
					if err != nil {
						http.Error(w, fmt.Sprintf("Internal Server Error. err: %s", err), http.StatusInternalServerError)
					}
					defer f.Close()

Copilot

Pull request overview

Copilot reviewed 50 out of 50 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

pkg/fetch/nvd/api/cvehistory/cvehistory_test.go:106

The "With API Key" test case sets WithAPIKey("foobar"), but the httptest server never asserts that the request includes the expected apiKey header. As a result, this test would still pass even if the fetcher stopped sending the header, so it doesn't actually cover the behavior it claims to cover.

			ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
				startIndex := "0"
				if value := r.URL.Query().Get("startIndex"); value != "" {
					startIndex = value
				}
				resultsPerPage := "5000"
				if value := r.URL.Query().Get("resultsPerPage"); value != "" {
					resultsPerPage = value
				}

shino

🖊️

feat(fetch/nvd/api/cvehistory): add CVE change history fetcher

eca08f5

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

MaineK00n self-assigned this Jul 30, 2026

MaineK00n marked this pull request as ready for review July 30, 2026 06:11

Copilot AI review requested due to automatic review settings July 30, 2026 06:11

Copilot started reviewing on behalf of MaineK00n July 30, 2026 06:11 View session

Copilot AI reviewed Jul 30, 2026

View reviewed changes

Comment thread pkg/fetch/nvd/api/cvehistory/cvehistory_test.go

Comment thread pkg/fetch/nvd/api/cvehistory/cvehistory.go Outdated

Comment thread pkg/fetch/nvd/api/cvehistory/types.go

Comment thread pkg/fetch/nvd/api/cvehistory/cvehistory.go

Copilot AI review requested due to automatic review settings July 30, 2026 06:29

Copilot started reviewing on behalf of MaineK00n July 30, 2026 06:30 View session

Copilot AI reviewed Jul 30, 2026

View reviewed changes

MaineK00n requested a review from shino July 30, 2026 07:33

Copilot AI review requested due to automatic review settings July 30, 2026 07:36

Copilot started reviewing on behalf of MaineK00n July 30, 2026 07:36 View session

Copilot AI reviewed Jul 30, 2026

View reviewed changes

Comment thread pkg/fetch/nvd/api/cvehistory/cvehistory.go

MaineK00n requested a review from Copilot July 30, 2026 08:34

Copilot started reviewing on behalf of MaineK00n July 30, 2026 08:34 View session

Copilot AI reviewed Jul 30, 2026

View reviewed changes

shino approved these changes Jul 31, 2026

View reviewed changes

MaineK00n merged commit 3b82436 into nightly Jul 31, 2026
3 checks passed

MaineK00n deleted the MaineK00n/fetch-nvd-api-cvehistory branch July 31, 2026 01:55

MaineK00n mentioned this pull request Jul 31, 2026

feat(ci): add nvd-api-cvehistory vulsio/vuls-data-db#207

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(fetch/nvd/api/cvehistory): add CVE change history fetcher - #899

feat(fetch/nvd/api/cvehistory): add CVE change history fetcher#899
MaineK00n merged 3 commits into
nightlyfrom
MaineK00n/fetch-nvd-api-cvehistory

MaineK00n commented Jul 30, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

shino left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

MaineK00n commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

How

Also included

Testing

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

shino left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

MaineK00n commented Jul 30, 2026 •

edited

Loading