Experiments: A/B Testing v2 #36763

Summary

Rebuild dotCMS A/B testing on the new analytics platform. Four parts, one epic:

1. Migrate Experiments to the new analytics infrastructure — move goal evaluation off CubeJS onto CAEM + ClickHouse
2. Build the new Experiments UI from scratch — per the approved design below
3. Make the SDK and the traditional page experience work — one analytics SDK for both rendering modes
4. Update the docs — rewritten against what ships

These ship together. The migration alone leaves a feature that is hard to operate; a UI rebuild alone has no working data source; SDK work alone fixes delivery for experiments that still cannot be measured or managed; docs written against current behavior go stale immediately. One acceptance bar: a user can run an experiment end to end and trust the result.

Design

Approved UI prototype →

The prototype is the spec for Part 2. It is interactive and covers all four screens — experiments list, configuration, UVE variant editor, results — including empty states, validation, and the configuration-lock behavior. Use the startScreen control to move between screens and lockRunningConfig to see the lock.

Two things it settles, rather than repeating them as a task list here:

• Configuration locking is one rule. When status is RUNNING, ENDED, or ARCHIVED, goal, variants, traffic, and scheduling lock as a set. Running is locked until stopped; ended and archived are read-only.
• Four goal types, no more. Bounce rate, exit rate, reach page, URL parameter. Reach-page and URL-parameter reveal a conditions block. The "coming soon" placeholders get deleted, not built.

This is a redesign of existing capability, not new behavior. Variant promotion, archive/restore, the page picker, weight-split editing, and the Bayesian result values all exist today. The work is rebuilding the surface, not inventing what sits behind it. The one genuinely new screen is the cross-page experiments list — today entry is per-page only (/:pageId), so there is no way to see which pages have experiments in draft, scheduled, running, or finished states.

Background

Experiments evaluates every goal by querying CubeJS, which is being retired in favor of the CAEM + ClickHouse pipeline delivered by #35048. This is a dependency deadline, not a product preference: when CubeJS is removed, experiment reporting stops working.

Three other problems compound:

• Reporting is slow or fails. Reports for in-progress and completed experiments can take a long time to load, and in some environments do not load at all.
• Delivery is split across two unrelated implementations. Traditional pages get experiment code injected server-side against the legacy Jitsu collector (ExperimentWebAPIImpl.getCode, experiment/js/init_script.js). Headless front ends use @dotcms/experiments, which still carries @jitsu/sdk-js as its only runtime dependency. The two share no code and neither emits events into the new analytics pipeline. @dotcms/analytics, which does talk to the new pipeline, has no concept of experiment context.
• Traditional variant delivery is rough. It halts page render, calls POST /v1/experiments/isUserIncluded, then redirects and reloads with a variantName query parameter — visible flicker, and it requires JavaScript.

Full implementation plan (schema DDL, endpoint contracts, per-track sequencing, production migration steps) is the output of spike #36195.

Architecture

• Event context — when an experiment is running, analytics events carry a nested experiment block (experiment id, running id, assigned variant) in the event context, shared across a batch.
• Storage — new experiment_id / running_id / variant columns on analytics.events and the session tables, a bloom-filter skip index on experiment_id, and updated materialized views. Empty string is the sentinel for "no experiment active".
• Query — session-level goals (bounce, exit) are served from the existing session summary via GET /v1/analytics/sessions, extended with new metrics and a variant dimension. Goals requiring raw event ordering (reach-target, URL parameter) are served by a new GET /v1/analytics/sessions/behavior, always scoped by a strong experiment filter to bound the scan.
• dotCMS core — the goal-result classes (BounceRateResultQuery, ExitRateResultQuery, ReachTargetAfterExperimentPageResultQuery, plus a new UrlParameterResultQuery) call the analytics endpoints instead of building CubeJS queries. dotCMS holds all experiment configuration and passes it as query params; the analytics service stores no goal definitions. The result model (ExperimentResults, GoalResults, VariantResults) is unchanged.
• SDKs — @dotcms/analytics gains an experiment-context API. @dotcms/experiments narrows to variant assignment plus setting that context, which removes the Jitsu dependency and the second set of analytics credentials. Consumers configure analytics once.

Acceptance criteria

Part 1 — Migration

• All four goal types return results from the new pipeline, matching CubeJS output within an agreed tolerance
• UrlParameterResultQuery exists — URL-parameter goals currently reuse the reach-target query
• No Experiments code path depends on CubeJS
• Production ClickHouse migration applied, sequenced last, with post-migration verification
• Decided and documented whether existing experiment data migrates from the old pipeline

Part 2 — UI

• All four screens match the approved prototype, including empty states and validation
• A cross-page experiment listing exists, with search and status filtering
• Configuration locks once an experiment is running, ended, or archived
• Reports load reliably at realistic data volume; the SLO is set from the analytics query API's measured performance

Part 3 — SDK and traditional pages

• No Experiments code path depends on Jitsu; @dotcms/analytics is the only analytics SDK required, with one set of credentials
• The same experiment behaves identically on a dotCMS-rendered page and a headless front end
• Variant assignment and rendering verified in the reference Next.js app
• Traditional variant delivery no longer halts page render

Part 4 — Docs

• Headless integration guide with a working reference example. Traditional pages auto-inject experiment code; headless requires explicit SDK wiring, and the current docs do not make that clear enough
• Traditional-page setup guide, goal-type reference, and the configuration-locking rules documented so intended behavior is not reported as a defect
• Published @dotcms/experiments SDK docs refreshed (stale — see @dotcms/experiments 1.0.0: Next.js 15 App Router integration crashes in UVE editor, spurious /api/v1/experiments/DEFAULT 404, and stale npm docs #36225)
• Upgrade note covering what changes for existing experiments

Out of scope

• AI-generated page variants and AI-recommended segments — these belong to the dotAI area and must not gate this epic
• Multi-armed bandit / automatic traffic reallocation; traffic split stays manual
• Multi-page and funnel experiments; single-page experiments only
• Multi-touch attribution models (see [EPIC] Analytics Conversion and Attribution #33553)
• Changes to the personalization rules engine; targeting continues to use dotCMS Rules
• Statistical engine rework — the Bayesian values shown in the new results UI are already computed today

Dependencies

• Blocks the query-layer work in Part 1: the Analytics Query REST API domain-driven redesign (Spike: Redesign Analytics Query REST API #35736, first slice in dot-ca-event-manager), because the experiment endpoints use the response envelope it introduces. Schema, ingest, SDK, and the core result-class migration do not depend on it and can start independently.
• Make Analytics Infrastructure Production-Ready #35048 — the CAEM + ClickHouse pipeline this builds on (closed)
• The Analytics app must be configured before Experiments is usable; this coupling is unchanged

Open questions

• Can outbound-link clicks be surfaced as a conversion, with per-placement attribution when the same CTA appears in multiple locations? The content_click payload already captures the element href, id, class, attributes, containing contentlet, and DOM index — so this may be a query/reporting change rather than new collection. Needs confirmation, and a decision on whether it belongs here or in analytics conversion tracking.
• Is a JavaScript-triggered goal type in scope? Neither the implementation plan nor the approved design includes one.
• Does existing experiment data migrate from the old pipeline to the new one? Spike [SPIKE] Migrate Experiments from legacy (CubeJS) to new analytics infrastructure (CAEM + ClickHouse) #36195 treats this as a feasibility question; the default is not to migrate.
• Is report slowness caused by the CubeJS query path, or by query shape and data volume? If the latter, the migration alone will not fix it and Part 2 needs different acceptance criteria.

References

• Approved UI prototype: https://claude.ai/design/p/36aed5a6-ca42-4624-b187-a465a23ce909?file=Experiments.dc.html&via=share
• Spike + full implementation plan: [SPIKE] Migrate Experiments from legacy (CubeJS) to new analytics infrastructure (CAEM + ClickHouse) #36195
• Analytics infrastructure epic: Make Analytics Infrastructure Production-Ready #35048
• Analytics Query REST API redesign: Spike: Redesign Analytics Query REST API #35736
• SDK defect (Next.js 15 App Router, stale docs): @dotcms/experiments 1.0.0: Next.js 15 App Router integration crashes in UVE editor, spurious /api/v1/experiments/DEFAULT 404, and stale npm docs #36225