Clarivate / Web of Science API integration — evidence layer for architecture artefacts

[tractorjuice]

Summary

Wire Clarivate research-intelligence APIs (Web of Science, Pivot-RP funding data, Derwent patents) into ArcKit as evidence sources for architecture and strategy artefacts.

Full plan: docs/plans/2026-07-16-clarivate-api-integration.md

Problem / use case

The weak point in any AI-generated strategy artefact is provenance. A Wardley map or options appraisal is only as defensible as the evidence under it. Clarivate supplies exactly the class of evidence that is expensive to gather by hand and hard to dispute once cited: peer-reviewed publication volume, citation weight, funding flows, patent activity, and organisation-level research footprints.

This lets ArcKit answer questions it currently can only assert:

• Where does a capability sit on the evolution axis? (Publication/patent growth curves as a defensible genesis → commodity proxy.)
• Who are the credible players in a technology area, by research output rather than marketing presence?
• Which funders are backing a field, at what scale, and is the money rising or falling?
• What is an institution's real research strength, for due-diligence artefacts?

The evolution-signal case is the headline one: it strengthens the Wardley skill ArcKit already ships, and it is the hardest thing to do credibly by hand.

Proposed solution

A clarivate plugin following the existing plugin/skill shape, with a deliberately source-agnostic transforms/ layer so an OpenAlex/Crossref client can be swapped in behind the same interface (plan §7).

plugins/clarivate/
  SKILL.md
  client/     wos_starter.py  wos_expanded.py  derwent.py  cache.py
  transforms/ evolution_signal.py  funder_flows.py  org_footprint.py
  commands/   /arckit:evidence-scan  /arckit:funder-map

Design principles: keys out-of-band (env / git-ignored secrets, never committed or logged); graceful degradation by plan (volume-only signals on free tier); hard caching keyed by query under .arckit/cache/clarivate/ with snapshot dates, reusing arckit-build --refresh semantics; every datum carries its WoS UID / grant reference / DOI into the artefact.

Phased delivery

• Phase 0 — spike. Done. Endpoints, auth, plan differences, GRANTS database, swagger specs confirmed; wos_test.sh harness exists (Starter + Expanded).
• Phase 1 — free-tier evidence-scan. wos_starter.py + volume-only evolution_signal.py + /arckit:evidence-scan. Self-serve Free Trial key, zero licence cost. Proves the value.
• Phase 2 — citation-weighted signals. Institutional Member plan or Expanded; citation-weighted evolution curves; org_footprint.
• Phase 2b (parallel) — open-source source. OpenAlex/Crossref client behind the same transform interface for a zero-licence path.
• Phase 3 — funder value-chain. Expanded GRANTS support + funder_flows.py + /arckit:funder-map. Novel artefact type; closes the loop to Pivot-RP.
• Phase 4 — patents (optional). Derwent client, licence permitting; adds the patent axis to evolution scoring.

Acceptance criteria (Phase 1)

• Free Trial key read from env / git-ignored secrets, never committed or echoed into artefacts or logs
• /arckit:evidence-scan <capability> returns a publication-volume time series with WoS UIDs cited
• Responses cached with snapshot date; --refresh forces re-pull
• Client throttles to 1 req/s and respects the 50/day quota
• Missing times-cited field detected and degraded gracefully, not fatal
• Evolution-signal note attaches to a Wardley map component without auto-overwriting manual placement

Key reconnaissance findings (from live probing)

• The Free Trial plan is self-serve and instant, but deliberately withholds times-cited counts — the citation graph is the monetised asset. Volume signals work on free tier; citation-weighted signals need Institutional Member or Expanded.
• The Expanded databaseId enum includes GRANTS as a first-class searchable database — the same funding data behind Pivot-RP's awarded-grants module, reachable programmatically through one endpoint.
• optionView=FR is the single switch returning funding + full affiliation blocks vs. the trimmed default.
• Derwent's GraphQL endpoint leaks its schema to unauthenticated introspection but gates all data behind the key — useful for building a client offline, useless for pulling patents without a licence.

Risks / open questions

• Licence terms. WoS API Product/Service Terms restrict redistribution of retrieved data. Pulled records must be treated as evidence within an artefact for the licensed user, not a redistributable dataset. Decides whether cached data can travel with a shared artefact (likely not — artefact cites UIDs, licensed user re-pulls). Read before Phase 2.
• Plan availability. Citation-weighted signals depend on paid Expanded or an institutional WoS subscription — confirm affiliation availability. Free-tier-only users get volume signals only; set expectations in the skill doc.
• Entity resolution. Organisation and funder name variants need a normalisation pass (same weakness as the govbuy work) — reuse that approach.
• Rate limits vs. build size. Free Trial is 1 req/s, 50/day. A large parallel arckit-build could exhaust quota fast — hard caching mitigates; consider a build-level budget guard.

Immediate next step

Register a Free Trial application on developer.clarivate.com, generate a Starter key, run wos_test.sh starter to confirm real-record retrieval, then build Phase 1 against it. Everything above Phase 1 waits on a licence or institutional-plan decision.

AI Tool

None

[mheilimo]

The plan already has the right starting points: query-keyed caching, snapshot dates, stable record identifiers, plan-aware degradation and human-owned Wardley placement. I would add one licence-aware promotion boundary between the local response cache and any shareable artefact.

An EvidenceSnapshot could record:

id, provider, plan, query_hash, retrieved_at,
coverage_start, coverage_end, raw_response_hash,
local_cache_locator, transform_id, transform_version,
entity_resolution_version, record_ids,
redistribution_class, availability_state,
derived_values, quality_notes, gaps

Then bind each artefact statement to a separate ArtifactClaim:

claim_id, artifact_id, snapshot_id, metric_name,
value, unit, observation_window, citation_ids,
transform_version, interpretation, human_owner

That gives the implementation a few testable rules:

1. Licensed raw responses stay in the local cache. A shared artefact exposes only identifiers, citations and derived values permitted by redistribution_class.
2. Every derived value remains tied to the exact query, source snapshot, transform and entity-resolution version that produced it.
3. A missing times-cited field is unavailable_by_plan or not_returned, never numeric zero.
4. --refresh appends a new snapshot and a delta. It does not mutate the evidence behind an existing artefact.
5. Clarivate and OpenAlex/Crossref adapters run the same conformance fixtures for units, windows and required provenance. Coverage or query-semantics differences remain visible instead of being normalised away.
6. The evidence note can inform an evolution hint, but the final map placement and its rationale remain human-owned.

The strongest acceptance fixture may be a shared artefact built from a paid snapshot. It should rebuild from the frozen local snapshot, omit non-redistributable raw fields, retain allowed UIDs or DOIs, and show an explicit gap when a second user lacks the same plan.

Disclosure: I work on DDScore at Playful Pixels Oy. In an adjacent private-company evaluation workflow, DDScore analyses submitted materials across 12 dimensions, cross-checks relevant claims against current public information, reports confidence and named gaps, and leaves judgement to the reviewer. This is not an ArcKit or Clarivate integration. DDScore does not validate bibliographic data, licence terms or architecture decisions, provide investment advice, or replace full diligence.