Skip to content

test(models): validate hot and offline catalogs independently - #669

Open
HaningZS wants to merge 1 commit into
HarnessMD:mainfrom
HaningZS:test/model-catalog-contract-652
Open

HaningZS wants to merge 1 commit into
HarnessMD:mainfrom
HaningZS:test/model-catalog-contract-652

Conversation

@HaningZS

@HaningZS HaningZS commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Addresses the catalog-test part of #652. CI suite execution already has an open contribution (#423); this PR deliberately does not duplicate it or close the whole issue.

The module contract explicitly describes the baked catalog as an offline fallback and the remote catalog as hot updates without shipping a release. Requiring them to be identical (including their provider keys) prevents that intended workflow. Instead:

  • Both shipped files must survive the production parser without silent entry loss, sanitizing, truncation or deduplication.
  • The shipped remote payload must update the actual picker overlay; clearing it must restore the offline lists.
  • Existing model-id safety, malformed-payload, version-filtering and baked-model expectations remain in place. No model data or production code changes.

Before

Upstream f44b986a: the existing remote-catalog suite reports 17 pass / 1 fail. This is actual node:test output rendered as an image, not a GUI screenshot.

Existing contract failure

After

The corrected contract suite reports 19 pass / 0 fail. This is a test-only change: the suite now independently validates both files and adds the real shipped-payload overlay/offline round trip, rather than keeping the contradictory equality assertion.

Independent catalog contract passes

Validation

macOS / Node 24.13.0:

  • Catalog suites: 27 passed.
  • Full npm run test:focused: 966 tests, 965 passed, 1 platform skip, 0 failed.
  • npm run typecheck and npm run build: passed.
  • Final integration with the independent worktree-origin (fix(worktree): persist isolation origins for agent restore #670) and reflection (fix(memory): respect providers and quota cooldowns in reflection #671) fixes: 1004 tests / 1003 passed / 1 platform skip / 0 failed, typecheck/build passed, no merge conflicts. The three real feature commits are combined in a temporary checkout; its full suite uses no test substitution.
  • Negative-control experiment: injected a newline into the real remote file's first model label at the require boundary. The new shipped-remote test failed as intended (18 passed / 1 failed). This checks that relaxing equality did not remove the malformed-entry guard.

Self-review

No new dependencies, OS-specific implementation, network test dependency, production changes or permission changes. The original payload parser remains unchanged. Existing version-filter tests continue to exercise remote entries. The before/after use the same upstream payload data; the assertion contract intentionally differs and is described above.

Why:
- Exact remote/baked equality contradicts the catalog's hot-update contract.

What:
- Validate both shipped payloads without silent row loss or rewriting.
- Exercise the real remote picker overlay and offline restoration.
- Include rendered before/after test evidence.

Risk:
- CI suite execution remains in the existing PR HarnessMD#423; this addresses only the catalog part of HarnessMD#652.

Tests:
- Catalog suites: 27 passed; malformed-label mutation: expected failure.
- npm run test:focused: 965 passed, 1 skipped, 0 failed.
- npm run typecheck and npm run build passed on macOS.

Live Docs:
- Contract: src/shared/modelCatalogPayload.ts.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant