fix(import): chunked, resumable WordPress plugin import (fixes Worker resource limits)

[swissky]

What does this PR do?

Fixes #475 for the plugin import path. A single /execute invocation that imports a whole site — taxonomy setup, every content page, menus, all comments, site settings — exceeds Cloudflare Worker resource limits (CPU time, subrequests) on sites with a few hundred posts, and there is no way to resume after the failure.

This PR makes the import chunked and resumable, as announced in discussion #1831 (comment):

• Client-driven chunking. /execute gains optional phase + cursor params. The admin wizard drives the import as a loop of bounded requests: one WP content page (50 posts) per call, then paginated comments (500/page), then a small finalize step (menus + site identity). Each invocation stays far below CPU and subrequest limits on both free and paid plans.
• Stateless server. Cross-chunk state — the WP-post-ID → EmDash-ID map that menus/comments need, translation groups, comment threading roots — is accumulated by the client and sent back with each request. No server-side session state, no new tables, no cleanup.
• Resume for free. An aborted import (closed tab, network drop) simply re-runs: skipExisting skips already-imported content while rebuilding the ID maps, so the import fast-forwards to where it stopped. Comments and media are idempotent the same way (timestamp/body match, content-hash dedupe).
• Media in batches. The media step now sends bounded batches of 25 attachments per request instead of one giant streaming request (the single point where large media libraries died before).
• Real progress. The wizard shows "312 of 507" with a progress bar instead of an indefinite spinner, plus per-phase status for comments and finalize.
• Backward compatible. A request without phase behaves exactly as before, so older admin clients keep working.

Later content chunks reload the taxonomy lookup maps with two SELECTs (loadTaxonomyPlanFromDb) instead of re-walking every term through the creation path; term/def creation runs once in the first chunk.

Update: #1830 has merged — this branch is now rebased onto main and contains only the chunked-import commit.

Integration tests cover the state that must survive chunk boundaries: translation-group linking across chunks, ID-map rebuild on a resumed run, taxonomy plan reload, and comment threading across pages.

Type of change

• Bug fix (non-breaking change which fixes an issue)

Checklist

• I have read the contributing guidelines
• I have added tests that prove my fix is effective or that my feature works
• I have added a changeset (or this change doesn't need one)
• New and existing unit tests pass locally with my changes
• Any user-facing strings in the admin UI are localized

AI-generated code disclosure

• Parts of this PR were AI-generated (tool: Cursor, model: Fable 5). All code was reviewed and tested by the submitter.

[github-actions]

Scope check

This PR changes 3,301 lines across 21 files. Large PRs are harder to review and more likely to be closed without review.

If this scope is intentional, no action needed. A maintainer will review it. If not, please consider splitting this into smaller PRs.

See CONTRIBUTING.md for contribution guidelines.

[changeset-bot]

🦋 Changeset detected

Latest commit: ad7b420

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages

Name	Type
emdash	Minor
@emdash-cms/admin	Minor
@emdash-cms/cloudflare	Minor
@emdash-cms/sandbox-workerd	Patch
@emdash-cms/fixture-perf-site	Patch
@emdash-cms/perf-demo-site	Patch
@emdash-cms/cache-demo-site	Patch
@emdash-cms/do-demo-site	Patch
@emdash-cms/do-solo-demo-site	Patch
@emdash-cms/auth	Minor
@emdash-cms/blocks	Minor
@emdash-cms/gutenberg-to-portable-text	Minor
@emdash-cms/x402	Minor
create-emdash	Minor
@emdash-cms/auth-atproto	Patch
@emdash-cms/plugin-embeds	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[emdashbot]

The PR’s approach is sound: moving the WordPress plugin import to client-driven, stateless chunks directly addresses issue #475 and fits EmDash’s Worker/serverless model. The route split keeps the legacy single-shot path intact, and the client carries the cross-chunk ID maps, translation groups, and comment roots as designed. Supporting changes in importContent, importCommentsFromPlugin, and loadTaxonomyPlanFromDb are covered by targeted tests.

I reviewed the diff statically, reading the new route handler, schema, admin API client, the wizard component, and supporting source/taxonomy/comments code. No SQL-injection, authorization, or data-loss regressions stood out, and the content lookups remain locale-aware. The items in findings are all fixable before merge: dead UI toggles, unlocalized or hard-coded English strings, and dark: Tailwind prefixes that violate the repo’s Kumo-based styling rules.

---

Findings

• [needs fixing] packages/admin/src/components/WordPressImport.tsx:546

  The new importMenus, importSiteTitle, importLogo, and importSeo toggles introduced around line 161 are rendered in ReviewStep, but this config object never includes them, and the server-side execute.ts never reads them. As a result the toggles are non-functional: menus, site settings, and SEO fields are imported regardless of the user’s choices. Either wire them through ImportConfig/the Zod schema and conditionally skip those steps on the server, or remove the toggles until they are implemented.

• [needs fixing] packages/admin/src/components/WordPressImport.tsx:1040

  These placeholders are hard-coded English strings shown to users. AGENTS.md requires every user-facing string to go through Lingui. The same applies to the URL placeholder on line 1085.

  								placeholder={t`em1.…`}
  

• [needs fixing] packages/admin/src/components/WordPressImport.tsx:1476

  Hard-coded English placeholders in the manual plugin-auth form. The application-password placeholder on line 1490 has the same problem.

  							placeholder={t`admin`}
  

• [needs fixing] packages/admin/src/components/WordPressImport.tsx:208

  Hard-coded error messages surfaced to the UI. t should wrap these so they are extracted for localization. Line 494 has the same issue ("OAuth authorization requires HTTPS. Please use manual credentials.").

  			setImportError(t`WordPress authorization was rejected`);
  

• [needs fixing] packages/admin/src/components/WordPressImport.tsx:1140-1162

  The feature names and notes in FEATURE_COMPARISON are rendered as user-visible text (item.feature and note) but are hard-coded English constants. They should be defined as Lingui msg descriptors and resolved with t() inside the component so they can be translated.

• [needs fixing] packages/admin/src/components/WordPressImport.tsx:1074

  The component repeatedly uses dark: Tailwind prefixes (e.g., here, line 1196, 1213, 1435, and elsewhere). AGENTS.md forbids dark: prefixes in the admin UI in favor of Kumo semantic tokens (bg-kumo-tint, text-kumo-subtle, etc.) that use CSS light-dark() internally.

  					<div className="p-3 rounded-full bg-kumo-tint">
  

[pkg-pr-new]

Open in StackBlitz

@emdash-cms/admin

npm i https://pkg.pr.new/@emdash-cms/admin@1849

@emdash-cms/auth

npm i https://pkg.pr.new/@emdash-cms/auth@1849

@emdash-cms/auth-atproto

npm i https://pkg.pr.new/@emdash-cms/auth-atproto@1849

@emdash-cms/blocks

npm i https://pkg.pr.new/@emdash-cms/blocks@1849

@emdash-cms/cloudflare

npm i https://pkg.pr.new/@emdash-cms/cloudflare@1849

@emdash-cms/contentful-to-portable-text

npm i https://pkg.pr.new/@emdash-cms/contentful-to-portable-text@1849

emdash

npm i https://pkg.pr.new/emdash@1849

create-emdash

npm i https://pkg.pr.new/create-emdash@1849

@emdash-cms/gutenberg-to-portable-text

npm i https://pkg.pr.new/@emdash-cms/gutenberg-to-portable-text@1849

@emdash-cms/plugin-cli

npm i https://pkg.pr.new/@emdash-cms/plugin-cli@1849

@emdash-cms/plugin-types

npm i https://pkg.pr.new/@emdash-cms/plugin-types@1849

@emdash-cms/registry-client

npm i https://pkg.pr.new/@emdash-cms/registry-client@1849

@emdash-cms/registry-lexicons

npm i https://pkg.pr.new/@emdash-cms/registry-lexicons@1849

@emdash-cms/sandbox-workerd

npm i https://pkg.pr.new/@emdash-cms/sandbox-workerd@1849

@emdash-cms/x402

npm i https://pkg.pr.new/@emdash-cms/x402@1849

@emdash-cms/plugin-ai-moderation

npm i https://pkg.pr.new/@emdash-cms/plugin-ai-moderation@1849

@emdash-cms/plugin-atproto

npm i https://pkg.pr.new/@emdash-cms/plugin-atproto@1849

@emdash-cms/plugin-audit-log

npm i https://pkg.pr.new/@emdash-cms/plugin-audit-log@1849

@emdash-cms/plugin-color

npm i https://pkg.pr.new/@emdash-cms/plugin-color@1849

@emdash-cms/plugin-embeds

npm i https://pkg.pr.new/@emdash-cms/plugin-embeds@1849

@emdash-cms/plugin-field-kit

npm i https://pkg.pr.new/@emdash-cms/plugin-field-kit@1849

@emdash-cms/plugin-forms

npm i https://pkg.pr.new/@emdash-cms/plugin-forms@1849

@emdash-cms/plugin-webhook-notifier

npm i https://pkg.pr.new/@emdash-cms/plugin-webhook-notifier@1849

commit: ad7b420

[swissky]

Thanks for the review — all findings addressed in ad7b420:

• Dead toggles: importMenus / importSiteTitle / importLogo / importSeo are now part of ImportConfig and honored server-side in both the legacy single-shot path and the chunked phases (menus + site identity in finalize, SEO field creation/values in the content phase). Absent flags mean enabled, so older admin clients keep today's behavior. The menus toggle also only renders for the plugin source now — the WXR path never imports menus, so showing a switch there was misleading. Covered by a new regression test (importSeo: false creates neither the seo_title field nor values).
• Localization: the migration-key/URL/credentials placeholders, both setImportError messages, and the FEATURE_COMPARISON feature names/notes now go through Lingui (msg descriptors resolved with t() in the component).
• dark: prefixes: replaced throughout WordPressImport.tsx with Kumo semantic tokens (kumo-brand, kumo-success(-tint), kumo-warning(-tint), kumo-danger(-tint), kumo-info-tint).

Lint, typecheck, and the import test suites pass locally.