Skip to content

test(e2e): stabilize playground smoke - #2834

Merged
ascorbic merged 3 commits into
mainfrom
codex/stabilize-playground-e2e
Sep 2, 2026
Merged

test(e2e): stabilize playground smoke#2834
ascorbic merged 3 commits into
mainfrom
codex/stabilize-playground-e2e

Conversation

@ascorbic

@ascorbic ascorbic commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

What does this PR do?

Stabilizes the workerd-backed Playground smoke after repeated CI runs exhausted its 120-second initialization deadline. The retained trace from this PR confirmed that POST /_playground/init was still pending when the navigation timeout fired.

  • Gives initialization a 240-second budget and the complete smoke a 360-second budget.
  • Retains traces on failures and retries one transient failure in CI with a fresh browser session.
  • Allows data-backed admin assertions up to 60 seconds to finish loading on slower CI runners.
  • Pins Playwright output to the repo-root test-results/ directory consumed by the artifact upload.
  • Runs the Playground smoke in a dedicated CI job rather than after the long Cloudflare shard 1 workload.
  • Includes Node, Cloudflare, and Playground results in the required E2E Tests roll-up.

Failure evidence: run 33536074249 and run 33511649004.

Related issue: none.

Type of change

  • Bug fix
  • Feature (requires maintainer-approved Discussion)
  • Refactor (no behavior change)
  • Translation
  • Documentation
  • Performance improvement
  • Tests
  • Chore (dependencies, CI, tooling)

Checklist

  • I have read CONTRIBUTING.md
  • pnpm typecheck passes
  • pnpm lint passes
  • Targeted tests for this change pass
  • Formatting passes for the changed files with Prettier
  • I have updated the existing Playground E2E test configuration and CI coverage
  • User-visible strings are wrapped for translation (not applicable: no user-visible strings changed)
  • I have added and reviewed a changeset (not applicable: no published package changed)
  • New features link to an approved Discussion (not applicable: test and CI stabilization only)

AI-generated code disclosure

  • This PR includes AI-generated code — model/tool: GPT-5 Codex

Screenshots / test output

No screenshots: this changes test orchestration only.

  • pnpm build
  • pnpm run --filter "@emdash-cms/playground^..." build
  • Current Playground E2E config repeated across three fresh sessions: 3 passed
  • Controlled failure retained its trace under repo-root test-results/
  • pnpm typecheck
  • pnpm lint
  • Workflow YAML parsing, Prettier, and git diff --check

@changeset-bot

changeset-bot Bot commented Sep 1, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: dcbb6de

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@github-actions github-actions Bot added area/ci size/S review/needs-review No maintainer or bot review yet labels Sep 1, 2026
@pkg-pr-new

pkg-pr-new Bot commented Sep 1, 2026

Copy link
Copy Markdown

Open in StackBlitz

@emdash-cms/admin

npm i https://pkg.pr.new/@emdash-cms/admin@2834

@emdash-cms/auth

npm i https://pkg.pr.new/@emdash-cms/auth@2834

@emdash-cms/auth-atproto

npm i https://pkg.pr.new/@emdash-cms/auth-atproto@2834

@emdash-cms/blocks

npm i https://pkg.pr.new/@emdash-cms/blocks@2834

@emdash-cms/cloudflare

npm i https://pkg.pr.new/@emdash-cms/cloudflare@2834

@emdash-cms/contentful-to-portable-text

npm i https://pkg.pr.new/@emdash-cms/contentful-to-portable-text@2834

emdash

npm i https://pkg.pr.new/emdash@2834

create-emdash

npm i https://pkg.pr.new/create-emdash@2834

@emdash-cms/gutenberg-to-portable-text

npm i https://pkg.pr.new/@emdash-cms/gutenberg-to-portable-text@2834

@emdash-cms/plugin-cli

npm i https://pkg.pr.new/@emdash-cms/plugin-cli@2834

@emdash-cms/plugin-types

npm i https://pkg.pr.new/@emdash-cms/plugin-types@2834

@emdash-cms/registry-client

npm i https://pkg.pr.new/@emdash-cms/registry-client@2834

@emdash-cms/registry-lexicons

npm i https://pkg.pr.new/@emdash-cms/registry-lexicons@2834

@emdash-cms/registry-moderation

npm i https://pkg.pr.new/@emdash-cms/registry-moderation@2834

@emdash-cms/registry-verification

npm i https://pkg.pr.new/@emdash-cms/registry-verification@2834

@emdash-cms/sandbox-workerd

npm i https://pkg.pr.new/@emdash-cms/sandbox-workerd@2834

@emdash-cms/x402

npm i https://pkg.pr.new/@emdash-cms/x402@2834

@emdash-cms/plugin-ai-moderation

npm i https://pkg.pr.new/@emdash-cms/plugin-ai-moderation@2834

@emdash-cms/plugin-atproto

npm i https://pkg.pr.new/@emdash-cms/plugin-atproto@2834

@emdash-cms/plugin-audit-log

npm i https://pkg.pr.new/@emdash-cms/plugin-audit-log@2834

@emdash-cms/plugin-color

npm i https://pkg.pr.new/@emdash-cms/plugin-color@2834

@emdash-cms/plugin-embeds

npm i https://pkg.pr.new/@emdash-cms/plugin-embeds@2834

@emdash-cms/plugin-field-kit

npm i https://pkg.pr.new/@emdash-cms/plugin-field-kit@2834

@emdash-cms/plugin-forms

npm i https://pkg.pr.new/@emdash-cms/plugin-forms@2834

@emdash-cms/plugin-webhook-notifier

npm i https://pkg.pr.new/@emdash-cms/plugin-webhook-notifier@2834

commit: dcbb6de

@emdashbot emdashbot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR stabilizes the workerd-backed Playground E2E smoke by moving it to a dedicated CI job, extending the Playwright test timeout from 120 s to 240 s while keeping the webServer startup budget at 120 s, and switching tracing from on-first-retry (ineffective because the Playground config has no retries) to retain-on-failure. It also updates the E2E Tests rollup to gate on Node, Cloudflare, and Playground lanes. The approach is focused and matches the stated problem.

I checked the diff, the full workflow file, the Playground Playwright config, the package scripts, and related workflow files. Everything is consistent except the artifact upload paths in the new test-e2e-playground job: Playwright resolves outputDir relative to the config file's directory, so traces and screenshots land under e2e/playground/test-results/, not the repo-root test-results/ the upload step currently references. Since retaining traces on the first failure is one of the PR's stated goals, this path should be fixed before merge.

Comment thread .github/workflows/ci.yml Outdated
@github-actions github-actions Bot added review/awaiting-author Reviewed; waiting on the author to respond and removed review/needs-review No maintainer or bot review yet labels Sep 1, 2026
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Sep 1, 2026

Copy link
Copy Markdown

🚀 Deploying Preview to Cloudflare 🚀

Preview URL: https://codex-stabilize-playground-e2e.try.emdashcms.com, https://codex-stabilize-playground-e2e-emdash-playground.emdash-cms.workers.dev (commit dcbb6de)

This URL reflects your latest Preview deployment

Preview Deployments by commit

Status Deployment URL Commit Updated (UTC) See this deployment's details
  • Build: Success ✅
  • Deployment: Success ✅

View logs ↗
https://f356142f.try.emdashcms.com, https://f356142f-emdash-playground.emdash-cms.workers.dev dcbb6de 2026-09-02T11:39:48.270Z Visit the dashboard ↗
  • Build: Success ✅
  • Deployment: Success ✅

View logs ↗
https://f2f67ded.try.emdashcms.com, https://f2f67ded-emdash-playground.emdash-cms.workers.dev 0cd5935 2026-09-02T09:57:47.465Z Visit the dashboard ↗
  • Build: Success ✅
  • Deployment: Success ✅

View logs ↗
https://3f90bcb5.try.emdashcms.com, https://3f90bcb5-emdash-playground.emdash-cms.workers.dev fc71857 2026-09-01T18:38:14.377Z Visit the dashboard ↗

@ascorbic ascorbic added the bot:review Trigger an emdashbot code review on this PR label Sep 2, 2026
@github-actions github-actions Bot added review/needs-rereview Author pushed changes since the last review and removed review/awaiting-author Reviewed; waiting on the author to respond labels Sep 2, 2026

@emdashbot emdashbot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review summary

This is a re-review of the Playground E2E stabilization PR. The approach is still focused and sensible: split the Playground smoke into its own CI job so it is no longer gated behind the Cloudflare shard 1 runtime, raise the in-test navigation budget to 240 s, and keep a 360 s per-test ceiling. Switching tracing to retain-on-failure while adding one CI retry means failed runs should now leave useful artifacts.

I re-checked the diff, the full workflow file, and the Playground Playwright config. The only previously raised issue is still unresolved. The new test-e2e-playground job advertises retain-on-failure traces, but its artifact upload step points at the repo-root test-results/ directory. Because the Playground config lives in e2e/playground/playwright.config.ts and Playwright resolves the default outputDir relative to the config file, traces/screenshots actually land in e2e/playground/test-results/. The upload step will therefore collect nothing (and the actions/upload-artifact step will warn or fail because the path does not exist), which directly undermines one of the PR’s stated goals.

No other blocking concerns. The timeout and retry changes are proportionate to the failure evidence, and the rollup job correctly gates on Node, Cloudflare, and Playground lanes.

Still open from the previous review: fix the Playground artifact upload path.

Comment thread .github/workflows/ci.yml
@emdashbot emdashbot Bot removed the bot:review Trigger an emdashbot code review on this PR label Sep 2, 2026
@github-actions github-actions Bot added review/awaiting-author Reviewed; waiting on the author to respond and removed review/needs-rereview Author pushed changes since the last review labels Sep 2, 2026
@ascorbic ascorbic added the bot:review Trigger an emdashbot code review on this PR label Sep 2, 2026
@github-actions github-actions Bot added review/needs-rereview Author pushed changes since the last review size/M and removed review/awaiting-author Reviewed; waiting on the author to respond size/S labels Sep 2, 2026

@emdashbot emdashbot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a re-review of the Playground E2E stabilization PR. The previously raised artifact-path mismatch is now resolved: e2e/playground/playwright.config.ts explicitly sets outputDir to the repo-root test-results/ directory, so the test-e2e-playground job’s upload-artifact step will actually collect traces and screenshots.

The overall approach is still sensible and proportionate to the failure evidence:

  • Pulling the Playground smoke out of the Cloudflare shard 1 job removes it from the long workerd runtime lane.
  • Raising waitForURL to 240 s and the per-test ceiling to 360 s gives the workerd-backed admin init enough headroom.
  • retain-on-failure plus one CI retry aligns trace retention with the goal of diagnosing flaky failures.
  • The rollup job now correctly gates on Node, Cloudflare, and Playground lane results.

I re-read the full ci.yml, the Playground Playwright config, and the spec. No code-level bugs or AGENTS.md convention violations jumped out.

One non-blocking thing to double-check before merge: the new CI build step uses pnpm run --filter "@emdash-cms/playground^..." build (.github/workflows/ci.yml, line 297). That ^... selector pattern does not appear anywhere else in the repo — every other dependency-expansion build step uses <pkg>... without the caret (e.g. --filter emdash..., --filter @emdash-cms/admin..., --filter "emdash-e2e-fixture-cloudflare..."). I cannot execute pnpm here to confirm whether @emdash-cms/playground^... is valid syntax in pnpm 11. If it is a typo, the job will fail with “No projects matched the filters.” If it’s intentional and supported, it’s fine; otherwise drop the caret to match the existing convention.

@emdashbot emdashbot Bot removed the bot:review Trigger an emdashbot code review on this PR label Sep 2, 2026
@github-actions github-actions Bot added review/awaiting-author Reviewed; waiting on the author to respond and removed review/needs-rereview Author pushed changes since the last review labels Sep 2, 2026
@ascorbic
ascorbic enabled auto-merge (squash) September 2, 2026 12:18
@ascorbic
ascorbic merged commit 43b200d into main Sep 2, 2026
59 checks passed
@ascorbic
ascorbic deleted the codex/stabilize-playground-e2e branch September 2, 2026 12:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/ci review/awaiting-author Reviewed; waiting on the author to respond size/M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant