Skip to content

feat: add Webarchiv Österreich provider #63

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
oritwoen merged 1 commit into from
Sep 1, 2026
Merged

feat: add Webarchiv Österreich provider #63

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
# PROJECT KNOWLEDGE BASE

**Last reviewed:** 2026-08-26
**Last reviewed:** 2026-09-01
**Branch:** main

> Verify against current HEAD: `git rev-parse HEAD`. Code map line numbers reflect the snapshot above; rerun `grep -n` if they look stale.

## OVERVIEW

Unified TypeScript interface for querying web archive providers (Wayback Machine, Arquivo.pt, Archive.today, Memento/MemGator, Common Crawl, Perma.cc, WebCite). Built on the unjs ecosystem: ofetch, unstorage, c12, consola, ufo, obuild, changelogen.
Unified TypeScript interface for querying web archive providers (Wayback Machine, Arquivo.pt, Webarchiv Österreich, Archive.today, Memento/MemGator, Common Crawl, Perma.cc, WebCite). Built on the unjs ecosystem: ofetch, unstorage, c12, consola, ufo, obuild, changelogen.

## STRUCTURE

Expand Down Expand Up @@ -67,6 +67,7 @@ archives/
| `UnsupportedOperationError` | class | archive.ts:18 | Thrown by `getPages()` when every queried provider is unsupported. Carries `providers` list. |
| `providers` | object | providers/index.ts:14 | Lazy-loading factory. Each method returns `Promise<ArchiveProvider>`. |
| `ArquivoProvider` | class | providers/arquivo.ts | Public Arquivo.pt CDX index and raw `noFrame/replay` capture reads. |
| `WebarchivProvider` | class | providers/webarchiv.ts | Austrian National Library public CDXJ index and raw `id_` replay for exact URLs. |
| `MementoProvider` | class | providers/memento.ts | JSON TimeMap from several archives via ODU MemGator; reads exact Memento URI, then proxy fallback. |
| `ArchiveInterface` | interface | types.ts:127 | Public API: `snapshots()`, `getPages()`, `use()`, `useAll()`. |
| `ArchiveProvider` | interface | types.ts:117 | Provider contract: `name`, `slug?`, `snapshots()`. |
Expand Down Expand Up @@ -112,7 +113,7 @@ archives/
- **OMP loader imports stay literal**: `existsSync(src)` chooses between `import("../../../src/tool-operations.ts")` and `import("../../../dist/tool-operations.mjs")`. Never `import(url.href)`. `tsc` resolves that dist specifier, so `test:types` builds before it type-checks.
- **MCP result is text only**: `details` never reaches an MCP client, so anything a caller needs for the next call belongs in `content[].text`.
- **Listing fans out, reading falls back**: `snapshots()` queries providers in parallel and merges; `content()` walks them in order and stops at the first body, because there is one page to read rather than a set to merge. Providers that failed or cannot read are reported beside the body in `_meta`.
- **A capture is read raw or not at all**: bodies come from `id_` playback (Wayback, Arquivo.pt, Archive-It) or a WARC byte range (Common Crawl). An archive that only serves its own rendition of a page returns `createUnsupportedContentResponse` with the reason instead.
- **A capture is read raw or not at all**: bodies come from `id_` playback (Wayback, Arquivo.pt, Webarchiv Österreich, Archive-It) or a WARC byte range (Common Crawl). An archive that only serves its own rendition of a page returns `createUnsupportedContentResponse` with the reason instead.
- **A stored capture is the response as it travelled**: a WARC record keeps the chunked framing and the `Content-Encoding` the server used, so reading its text means undoing both before the charset is applied. Playback endpoints do it for you, which is why only the Common Crawl path carries this.
- **The library decodes, a surface renders**: charset decoding, WARC unwrapping and transfer/content encodings are library work, and the body it returns is text; `htmlToText`, clipping to `maxChars` and the untrusted-data fence are applied in `tool-operations.ts`, so a library consumer keeps the whole document rather than a reader's view of it. Text is the contract, not the raw bytes: a capture that is not text decodes lossily and its bytes stay behind `_meta.rawSnapshot` or the WARC coordinates.
- **The MCP process does not trust its own cwd**: `src/commands/mcp.ts` calls `setConfigCwd(homedir())` because a client spawns the server in an arbitrary checkout, and c12 executes the `archives.config.ts` it finds. `consola.level` is pinned there too — stdout carries the JSON-RPC frames.
Expand Down Expand Up @@ -154,6 +155,7 @@ pnpm release # test + changelogen + publish
- **Defaults**: concurrency=3, batchSize=20, timeout=10000ms, retries=1, cache TTL=7 days. README and code must match.
- **Memento Time Travel is gone**: `mementoweb.org` remains a static documentation site after LANL discontinued the aggregator in 2025. `providers.memento()` defaults to the live public ODU MemGator endpoint and may be pointed at another compatible instance with `baseUrl`.
- **Arquivo.pt is a direct provider**: query `https://arquivo.pt/wayback/cdx` as newline-delimited JSON and read raw bodies from `noFrame/replay/<timestamp>id_/<url>`. It belongs in `providers.all()` even though MemGator may also return Arquivo.pt captures, because Memento stays outside that fan-out.
- **Webarchiv Österreich uses CDXJ for one URL at a time**: query `https://webarchiv.onb.ac.at/web/cdx` with the URL written as HTTP, because the index canonicalizes schemes but the HTTPS version can fail upstream. `from`, `to`, `limit` and `reverse=true` are supported; wildcard and `sort` queries are not. Read raw bodies from `/web/<timestamp>id_/<url>`. It requires no credentials and belongs in `providers.all()`.
- **WebCite has no list-by-domain API**: `webcite.snapshots(domain)` returns `unsupported: true` with a `unsupportedReason`. Direct snapshot retrieval (`webcitation.org/<id>`) is planned via a future `getById` API. New archives have not been accepted since ~2019.
- **Archive.today uses Memento API**: parses timemap link headers with regex. Fragile if format changes.
- **Playground targets Cloudflare**: `nitro.preset = 'cloudflare_module'` with `nodeCompat: true`.
Expand Down
44 changes: 27 additions & 17 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Unified TypeScript interface for querying web archive providers. One API, multip

## Features

- 🔍 **Multiple providers** - Wayback Machine, Arquivo.pt, Archive-It, Conifer, Archive.today, Memento/MemGator, Common Crawl, Perma.cc, WebCite
- 🔍 **Multiple providers** - Wayback Machine, Arquivo.pt, Webarchiv Österreich, Archive-It, Conifer, Archive.today, Memento/MemGator, Common Crawl, Perma.cc, WebCite
- 📄 **Reads captures, not just lists them** - `content()` returns what an archived page said, decoded from the original response
- 🌳 **Tree-shakable** - providers are lazy-loaded via dynamic imports, bundle only what you use
- 📦 **Caching built in** - pluggable storage layer via [unstorage](https://github.com/unjs/unstorage) with configurable TTL
Expand Down Expand Up @@ -62,7 +62,7 @@ const archive = createArchive(
const response = await archive.snapshots("example.com", { from: "2019", to: "2019-06" });
```

Providers whose index takes a window (Wayback, Arquivo.pt, Archive-It) narrow the query itself; for the rest the listing is filtered after it returns, so captures outside the window never mix into a fan-out. While a window is active, `limit` applies after the filter rather than at the provider, so a tight limit cannot eat the window. What the window cannot reach past is the single batch a windowed fetch asks for: up to 1000 index rows from Arquivo.pt, Common Crawl and Conifer, 100 from Perma.cc, and the library does not paginate beyond that.
Providers whose index takes a window (Wayback, Arquivo.pt, Webarchiv Österreich, Archive-It) narrow the query itself; for the rest the listing is filtered after it returns, so captures outside the window never mix into combined results. While a window is active, `limit` applies after the filter rather than at the provider, so a tight limit cannot eat the window. What the window cannot reach past is the single batch a windowed fetch asks for: up to 1000 index rows from Arquivo.pt, Webarchiv Österreich, Common Crawl and Conifer, 100 from Perma.cc, and the library does not paginate beyond that.

### Arquivo.pt

Expand All @@ -73,6 +73,15 @@ const archive = createArchive(providers.arquivo());
const response = await archive.snapshots("example.com");
```

### Webarchiv Österreich

Webarchiv Österreich searches the Austrian National Library's public CDXJ index and serves captures through raw `id_` replay. It needs no API key, is included in `providers.all()`, and searches one exact URL rather than every path on a domain:

```ts
const archive = createArchive(providers.webarchiv());
const response = await archive.snapshots("https://www.onb.ac.at/");
```

### Perma.cc

Perma.cc requires an API key and searches archives accessible to that account by exact submitted URL:
Expand Down Expand Up @@ -168,7 +177,7 @@ const older = await archive.content("https://example.com/page", { timestamp: "20
await archive.content("https://web.archive.org/web/20190301120000/https://example.com/");
```

Bodies are read through each archive's raw capture endpoint where one exists: Wayback, Arquivo.pt and Archive-It replay the original response under the `id_` modifier, Memento reads the TimeMap's exact Memento URI with a raw replay modifier where supported and falls back to MemGator's proxy when direct playback fails, and Common Crawl serves the byte range of the WARC record the index points at. Archive.today has no raw endpoint at all, so its `content()` returns the page as the site renders it, wrapper markup and all, rather than the bytes the original server sent; a rate limit or CAPTCHA answer becomes an error instead of posing as the capture.
Bodies are read through each archive's raw capture endpoint where one exists: Wayback, Arquivo.pt, Webarchiv Österreich and Archive-It replay the original response under the `id_` modifier, Memento reads the TimeMap's exact Memento URI with a raw replay modifier where supported and falls back to MemGator's proxy when direct playback fails, and Common Crawl serves the byte range of the WARC record the index points at. Archive.today has no raw endpoint at all, so its `content()` returns the page as the site renders it, wrapper markup and all, rather than the bytes the original server sent; a rate limit or CAPTCHA answer becomes an error instead of posing as the capture.

Providers are tried in order and the first body wins, because there is one page to read rather than a set to merge. The ones that could not answer are reported next to the body:

Expand All @@ -182,18 +191,19 @@ response._meta?.unsupportedProviders; // [{ provider: "webcite", reason: "..." }

## Providers

| Provider | Factory | `content()` | Notes |
| --------------- | -------------------------- | ----------- | ----------------------------------------------------------------------------------------------------------- |
| Wayback Machine | `providers.wayback()` | yes | web.archive.org CDX API; captures replayed under `id_` |
| Arquivo.pt | `providers.arquivo()` | yes | Public CDX API; raw captures replayed through `noFrame/replay` |
| Archive-It | `providers.archiveIt()` | yes | Requires a numeric `collection`; collection-specific CDX/C API |
| Conifer | `providers.conifer()` | no | Requires `user` and `collection`; searches an existing public collection |
| Archive.today | `providers.archiveToday()` | yes | archive.ph via Memento timemap; bodies are the rendered wrapper page, not the original bytes |
| Memento | `providers.memento()` | yes | Public ODU MemGator JSON TimeMap; queries several archives; excluded from `all` to avoid duplicate requests |
| Common Crawl | `providers.commoncrawl()` | yes | Defaults to latest collection; bodies read from the WARC byte range |
| Perma.cc | `providers.permacc()` | no | Requires `apiKey`; exact URL lookup only; API returns metadata only |
| WebCite | `providers.webcite()` | no | No list-by-domain API; `snapshots()` returns unsupported. New archives no longer accepted (~2019). |
| All | `providers.all()` | n/a | Wayback, Arquivo.pt, Archive.today, Common Crawl, and WebCite |
| Provider | Factory | `content()` | Notes |
| -------------------- | -------------------------- | ----------- | ----------------------------------------------------------------------------------------------------------- |
| Wayback Machine | `providers.wayback()` | yes | web.archive.org CDX API; captures replayed under `id_` |
| Arquivo.pt | `providers.arquivo()` | yes | Public CDX API; raw captures replayed through `noFrame/replay` |
| Webarchiv Österreich | `providers.webarchiv()` | yes | Public CDXJ index; exact URL lookup; raw captures replayed under `id_` |
| Archive-It | `providers.archiveIt()` | yes | Requires a numeric `collection`; CDX/C API specific to that collection |
| Conifer | `providers.conifer()` | no | Requires `user` and `collection`; searches an existing public collection |
| Archive.today | `providers.archiveToday()` | yes | archive.ph via Memento timemap; bodies are the rendered wrapper page, not the original bytes |
| Memento | `providers.memento()` | yes | Public ODU MemGator JSON TimeMap; queries several archives; excluded from `all` to avoid duplicate requests |
| Common Crawl | `providers.commoncrawl()` | yes | Defaults to latest collection; bodies read from the WARC byte range |
| Perma.cc | `providers.permacc()` | no | Requires `apiKey`; exact URL lookup only; API returns metadata only |
| WebCite | `providers.webcite()` | no | No API for listing a domain; `snapshots()` returns unsupported. New archives no longer accepted (~2019). |
| All | `providers.all()` | n/a | Wayback, Arquivo.pt, Webarchiv Österreich, Archive.today, Common Crawl, and WebCite |

A provider that cannot serve bodies answers `content()` as unsupported with the reason, exactly as it does for a listing it has no endpoint for.

Expand Down Expand Up @@ -293,7 +303,7 @@ interface ArchivedContent {
}
```

The `_meta` object on each page carries fields specific to each provider. Wayback includes `status` and `timestamp` in its raw format. Arquivo.pt adds `digest`, `mime` and `length`. Memento adds the upstream `archive` hostname and raw `datetime`. Common Crawl adds `digest`, `mime`, `collection`. Perma.cc has `guid`, `title`, `created_by`. Archive.today provides `hash` and `raw_date`.
The `_meta` object on each page carries fields specific to each provider. Wayback includes `status` and `timestamp` in its raw format. Arquivo.pt and Webarchiv Österreich add `digest`, `mime` and `length`. Memento adds the upstream `archive` hostname and raw `datetime`. Common Crawl adds `digest`, `mime`, `collection`. Perma.cc has `guid`, `title`, `created_by`. Archive.today provides `hash` and `raw_date`.

### Unsupported operations

Expand All @@ -313,7 +323,7 @@ Example:
const archive = createArchive(providers.all());
const response = await archive.snapshots("example.com");

response.pages; // results from Wayback, Arquivo.pt, Archive.today, Common Crawl
response.pages; // results from Wayback, Arquivo.pt, Webarchiv Österreich, Archive.today, Common Crawl
response._meta?.unsupportedProviders;
// [{ provider: "webcite", reason: "WebCite has no list-by-domain API. ..." }]
```
Expand Down
Loading
Loading