Skip to content

feat(cloudflare/ai-search): AiSearch, AiSearchInstance and AiSearchNamespace#639

Merged
sam-goodwin merged 11 commits into
mainfrom
sam/ai-search
Jun 18, 2026
Merged

feat(cloudflare/ai-search): AiSearch, AiSearchInstance and AiSearchNamespace#639
sam-goodwin merged 11 commits into
mainfrom
sam/ai-search

Conversation

@sam-goodwin

@sam-goodwin sam-goodwin commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Full Cloudflare AI Search (AutoRAG) coverage: a convenience construct with a discriminated data source, Worker bindings in both Effect-native and async styles, and the underlying namespace-scoped resources.

AiSearch construct — discriminated by source

The fast path for the common case. It picks the data source from what you pass as source: an R2Bucket for R2, or a URL string for a web crawl.

// R2 — pass the bucket; the construct mints the service token for you
const bucket = yield* Cloudflare.R2Bucket("docs");
const search = yield* Cloudflare.AiSearch("docs-search", {
  source: bucket,
});

// Web crawler — pass a URL; no service token needed
const crawlStore = yield* Cloudflare.R2Bucket("crawl-store");
yield* Cloudflare.AiSearch("site-search", {
  source: "https://example.com",
  parse: { type: "crawl" },
  crawl: { depth: 3, includeSubdomains: true },
  store: { bucket: crawlStore },
});

For R2, the construct mints a least-privilege AccountApiToken (AI Search Index Engine) + AiSearchToken and wires it into the instance — Cloudflare only provisions this token via dashboard/Wrangler, never on a programmatic create. Pass your own tokenId to skip minting.

Source-specific options are flattened onto the props:

  • R2: prefix, include / exclude (micromatch globs, max 10 each, exclude wins), jurisdiction.
  • Web crawler: parse, crawl, store (an R2 bucket to hold crawl output).

Group pipelines under a namespace by passing the AiSearchNamespace resource itself (not its name) — the engine then orders the pipeline after the namespace on deploy and before it on destroy:

const ns = yield* Cloudflare.AiSearchNamespace("docs", {});
const search = yield* Cloudflare.AiSearch("docs-search", {
  source: bucket,
  namespace: ns,
});

Worker bindings (Effect-native + async)

Bindings resolve to the current AiSearchInstance / AiSearchNamespace runtime shapes (the deprecated AutoRAG shape is gone). Bind the AiSearch result during a Worker's init phase with Cloudflare.AiSearchInstance.bind(search) for an Effect-native client:

const search = yield* Cloudflare.AiSearch("docs-search", { source: bucket });
const client = yield* Cloudflare.AiSearchInstance.bind(search);

const answer = yield* client.chatCompletions({
  messages: [{ role: "user", content: query }],
});
const hits = yield* client.search({ query }); // retrieval only

The client surfaces chatCompletions (retrieval + generation), search (retrieval only), info, stats, and raw. The namespace client adds .get(name), namespace-wide list / search, and raw.

Or pass it via env for a vanilla async Worker. InferEnv maps both flavors to the runtime types (AiSearchInstance, AiSearchNamespace):

const worker = yield* Cloudflare.Worker("api", {
  main: "./worker.ts",
  env: { SEARCH: search, NS: namespace },
});

Underlying resources

The construct is just these resources composed; drop down when you need to share a token across instances or adopt an existing one.

  • AiSearchInstance uses the namespace-scoped API and exposes a namespace attribute (its identity is instanceId). The namespace, source, type, and embeddingModel are immutable — changing any replaces the instance.
  • AiSearchNamespace treats the account-provided default namespace as reserved: it's bindable and adoptable but never updated or deleted on teardown (Cloudflare disallows deleting it).
  • AiSearchToken wraps an AccountApiToken into the service-token shape the R2 indexer needs.

Docs

  • New tutorial tutorial/cloudflare/ai-search favoring the AiSearch helper, explaining what it provisions and binding it into an Effect / async Worker.
  • Richer resource-level JSDoc on AiSearch, AiSearchInstance, and AiSearchNamespace (R2-first ordering, source filters, web-crawler options, namespace grouping via the AiSearch construct, Effect/async binding), regenerated API reference.

distilled

Depends on the namespace-scoped instance operations (create/read/update/list/delete) with resource-qualified typed errors (AiSearchInstanceNotFound, NamespaceNotFound, WebCrawlerDomainNotOwned, …), pushed to distilled main and bumped here.

Tests

Live-tested against Cloudflare: Namespace, Instance (incl. custom-namespace + replacement), Token, the AiSearch construct (R2 + web-crawler), and Bindings (async + Effect workers). Web-crawler lifecycle uses an account-owned workers.dev target; an ungated probe asserts the WebCrawlerDomainNotOwned typed error for unowned domains.

…nstruct, worker bindings

- AiSearchInstance: namespace-scoped CRUD + `namespace` attribute (replacement on change)
- AiSearchNamespace: reserved `default` namespace is bindable but never updated/deleted
- AiSearch: convenience construct that auto-mints the API + service token for R2 sources
- Worker bindings: `ai_search` (instance) + `ai_search_namespace`, async (`env`) and Effect-native (`.bind`)
- Bump distilled for namespace-scoped instance operations with typed errors

Co-authored-by: Cursor <cursoragent@cursor.com>
@alchemy-version-bot

alchemy-version-bot Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Install the packages built from this commit:

alchemy

bun add alchemy@https://pkg.ing/alchemy/230210f

@alchemy.run/better-auth

bun add @alchemy.run/better-auth@https://pkg.ing/@alchemy.run/better-auth/230210f

@alchemy.run/pr-package

bun add @alchemy.run/pr-package@https://pkg.ing/@alchemy.run/pr-package/230210f

sam-goodwin and others added 10 commits June 17, 2026 14:52
Rewrite AiSearch worker-binding docs to show full Effect Worker first,
then async, and wire AiSearchInstance.bind static for the examples.

Co-authored-by: Cursor <cursoragent@cursor.com>
- Add an AI Search tutorial favoring the AiSearch helper (Effect + async worker)
- Enrich AiSearchInstance/AiSearchNamespace JSDoc with binding + namespace examples
- Rename AiSearchInstance attribute `id` -> `instanceId`
- Rename Instance/Namespace/Token source files to AiSearch* and fix imports

Co-authored-by: Cursor <cursoragent@cursor.com>
Web-crawler instances index a seed URL and need no service token. Cloudflare
only crawls a domain the account owns, so the tests seed the crawl at a
deployed Worker's workers.dev URL with `parseType: "crawl"`.

- AiSearch construct: web-crawler source returns `token: undefined`
- AiSearchInstance: create/read/delete a web-crawler instance with no token
- `AiSearch` `url` now accepts `Input<string>` so a Worker URL output works
- bump distilled for the typed `WebCrawlerDomainNotOwned` error

Co-authored-by: Cursor <cursoragent@cursor.com>
…emap gotcha

The crawl target now serves a linked multi-page site so `parseType: "crawl"`
has more to walk. Emulating a sitemap was attempted but Cloudflare validates
it synchronously at create (discovered via robots.txt) and a fresh workers.dev
URL returns `missing_sitemap`, so crawl remains the reliable parse mode.

Co-authored-by: Cursor <cursoragent@cursor.com>
…minated `source`

The construct now discriminates on what you pass as `source`:

```ts
// R2 — pass the bucket; filters at top level
AiSearch("docs", { source: bucket, prefix: "docs/", include: ["published/"] });

// Web crawler — pass a URL; parse/crawl/store groups
AiSearch("site", {
  source: url,
  parse: { type: "crawl", contentSelector: [{ path: "/docs", selector: "main" }] },
  crawl: { depth: 3, includeSubdomains: true },
  store: { bucket: storeBucket },
});
```

`parse`/`crawl`/`store` map onto distilled's
`sourceParams.webCrawler.{parseType+parseOptions,crawlOptions,storeOptions}`;
`store.bucket` takes an R2Bucket. The low-level AiSearchInstance keeps the
distilled shape and now documents every web-crawler option.

Co-authored-by: Cursor <cursoragent@cursor.com>
…rce filters

- Keep R2 examples ahead of web-crawler in the AiSearch + AiSearchInstance
  reference (move the basic crawler example into the web-crawler section).
- Tutorial: document `prefix` / `include` / `exclude` for an R2 `source`,
  and add a note linking out to the reference for the web-crawler options.

Co-authored-by: Cursor <cursoragent@cursor.com>
…ect DX

- Switch Worker bindings off the deprecated `AutoRAG` shape onto the
  `AiSearchInstance` / `AiSearchNamespace` runtime types: `chatCompletions`
  (retrieval + generation), `search`, `info`, `stats`, plus namespace-level
  `list` / `search` and `raw` accessors.
- `AiSearch` construct accepts `namespace: AiSearchNamespace` (the resource,
  not its name) so the engine orders pipeline-after-namespace.
- Tune R2 service-token propagation retry (exponential 1.5x, 10 attempts).

Co-authored-by: Cursor <cursoragent@cursor.com>
@sam-goodwin sam-goodwin merged commit 71b4e9a into main Jun 18, 2026
4 checks passed
@sam-goodwin sam-goodwin deleted the sam/ai-search branch June 18, 2026 00:05
@sam-goodwin sam-goodwin changed the title feat(cloudflare/ai-search): namespace-scoped instances, auto-token construct, worker bindings feat(cloudflare/ai-search): AiSearch, AiSearchInstance and AiSearchNamespace Jun 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant