Skip to content

0041-observability-stack.md

Latest commit

 

History

History
1056 lines (961 loc) · 80.9 KB

0041-observability-stack.md

File metadata and controls

1056 lines (961 loc) · 80.9 KB
Raw

ADR-0041: Observability on Cloudflare — a sleeping Loki + Grafana box, OTLP inward, Sentry for uncaught errors

Status: Accepted (2026-10-21) — design approved 2026-09-23 (revision 3), implemented, local end-to-end walkthrough, every sandbox probe and the production verification (2026-09-29 to 2026-10-21, DEV-3093) complete (§L "Results"). 14 of 15 exit criteria pass with real evidence, criteria 5, 9, 11 and 13 also in production; criterion 8 (Volume) is Mixed, not a pass — Analytics Engine points and the raw Workers Logs pool both pass at 10× with real margin, but the exported-logs allotment does not (§D above has the numbers and the fallback). The design's own §L trigger (criterion 1, 2 or 7 failing) is not engaged. The last pending reading, criterion 13's real-object retention expiry on the production inbox bucket, passed on 2026-10-21 (DEV-3178). The §E.3 Sentry scope switch was flipped to uncaught on 2026-10-13, held for a week (checked 2026-10-21) and was then removed (DEV-3099): §E.1's split is now fixed in code. Supersedes ADR-0040 decisions A, B, C.2 and C.3; amends ADR-0022 (o11y spend cap, per-script billing rows), ADR-0038 (WAF exception extended to /telemetry/*); adds routes under ADR-0020. No longer deviates from ADR-0007: /grafana/* gates through the same Handsontable login broker as every other internal surface, not Cloudflare Access — see §H. ADR-0042 ships with this ADR and was accepted with it. ADR-0043 follows after launch (not yet dispatched).

Context

The runner reports faults through Sentry and nothing else. Both SDKs run errors-only, Workers Logs keeps seven days at a 10 % head sample, and every number the team looks at is a D1 counter aggregated at write time (usage_daily, analytics_daily, cost_ledger) and rendered by /admin. There are no traces, no metric history, no browser performance data, and the only join between a Sentry issue and a Worker log is the cf-ray header, promoted to a Sentry tag by hand.

The missing thing is the ability to answer, from data, the questions the runner is run on: how long an example takes to show a grid, by tier and Handsontable major; how full the container pool is at 14:00 UTC; whether the last release raised compile errors on the next bucket; what the AI assistant costs per answer; which docs guides people reach for a live example of (ADR-0042).

Constraints:

  1. Self-hosted, on Cloudflare, next to the app, in the main account (ADR-0010). No VPS, no Grafana Cloud, no SaaS beyond Sentry.
  2. OpenTelemetry protocol as the contract from the Worker inward.
  3. Grafana as the UI, Grafana Faro as the browser SDK.
  4. Sentry stays connected for uncaught errors, and those errors also land in the new stack.
  5. Cost at app scale. The runner's containers cost $1–13 a month because they sleep; observability follows the same pattern.
  6. Anonymous by construction stays in force for analytics (analytics.ts, AGENTS.md). Operational logs are governed by the explicit rule in §E.4.
  7. Deploy only after localhost tests, from one feature branch (decision 2026-09-23). Facts that exist only on real Cloudflare are measured with throwaway probes on the sandbox account, never the production account (§L).

Platform facts that shaped the design (verified 2026-09-22/23, including Loki, Sentry and Faro source):

  • Workers tracing and export (open beta): auto spans for fetch, D1, KV, R2, DO, handlers, RPC; custom spans via tracing.enterSpan; OTLP export of traces and logs to any endpoint with custom headers. From 2026-10-01 spans are billed as events in the same 20 M/month Workers Logs pool as log lines, and export includes 10 M events per signal. No spanContext() at the time of writing (it landed later, see §C.4), so no browser-to-worker traceparent join. No metrics export. Sampling is per invocation and per Worker, never per route. Export retry semantics are undocumented. Fetch-handler spans carry url.full, user_agent.original, city and ASN; preview hostnames carry the sandbox token.
  • Workers Logs: every sampled invocation writes an invocation log plus each console.* line; observability.logs.invocation_logs: false removes the former. The Sandbox SDK logs a warning on every request to a stale preview URL, and Tier-2 container stdout lands in the API worker's logs.
  • Containers bill CPU on use and memory/disk on provisioned size, nothing while asleep. Disk is ephemeral. Inbound only via Worker/DO. jurisdiction: "eu" placement exists since 2026-04-05. onStop reports a host loss the DO did not observe as { exitCode: 0, reason: "exit" }, indistinguishable from a clean exit. Cloudflare "does not guarantee that any container instance will run for any set period of time."
  • R2 is S3-compatible; EU jurisdiction on buckets; lifecycle rules work by key prefix.
  • Loki 3.x ingests OTLP natively and synchronously; otlp_config can label only resource attributes. Query-time dedupe needs same stream, same timestamp, same line and same structured metadata (TestMergeIteratorNoDedupDifferentStructuredMetadata), and metric queries dedupe only across iterators, so two copies inside one chunk are both counted. When an OTLP record has no timestamp, Loki stamps time.Now(). POST /flush returns before anything is written. The TSDB index head rotates every 15 minutes and uploads afterwards. The compactor rewrites index only, and its retention markers live on local disk. Default push limits are 4 MB/s with a 6 MB burst; shard_streams is on by default with shard numbers from in-memory state; query_ingesters_within defaults to 3 h.
  • Grafana Faro v2.12: faro.receiver stamps every entry with time.Now(), answers 202 even when its exporter fails, drops after a 2 s timeout, rate-limits at 50 req/s, and looks maps up by URL path. persistent: false session tracking uses sessionStorage. The Performance and CSP instrumentations send full URLs with query strings.
  • Workers Analytics Engine: 20 blobs + 20 doubles + 1 index per point, 3-month retention, 10 M data points and 1 M read queries per month included (currently not billed), SQL API; it samples at write time and read time (_sample_interval). Grafana reads it through the Altinity ClickHouse plugin. No local emulation.
  • Sentry: Team plan, so issue-alert webhooks only, no per-event error.created. The API worker's fetch catch-all converts every throw into an explicit captureException plus a 500, and the snapshot-job alarm reports without rethrowing, so almost no Worker error is "uncaught" in the SDK's sense. Spend alerts reach anyone only through Sentry.captureMessage in reconcile.ts. React 19 render crashes caught by Sentry.ErrorBoundary never reach window.onerror.
  • Edge: the zone rule behind ADR-0038 403s any body containing <script outside /api/*; workers_dev defaults to on and bypasses zone rules.

Alternatives considered and rejected:

  • VPS with the LGTM stack — outside constraint 1.
  • Serverless store (Pipelines → Iceberg → R2 SQL) — no LogQL, two open-beta services. Kept as the escape hatch (§L).
  • Cloudflare's own Workers Observability as the store — 7-day, non-EU, not Grafana. Kept as the temporary safety net (persist: true, §H).
  • Alloy with faro.receiver in the box (revision 2's design) — time.Now() stamping makes every browser record land at drain time and never dedupe on re-replay; 202-on-failure makes the drain's ledger lie; its rate limit refuses backlog replays; its path-based map lookup breaks per-release storage. Once the Worker converts Faro to OTLP itself (§C.1), Alloy has nothing left to do: the drain pushes straight to Loki.
  • Vanilla OpenTelemetry JS in the browser — experimental SDKs, no bundled error or vitals capture. Faro stays as the capture SDK; only its wire format stops at the Worker.
  • Mimir — a WAL on ephemeral disk and ingestion only while awake. Analytics Engine ingests while the box sleeps.
  • Tempo, and exporting traces at all, in the first release — no browser join exists, span attributes carry URLs, user agents, geo and preview tokens, and spans compete with logs for the same 20 M pool. Dropped (§C.4).
  • A Queue in front of the inbox — billed per message operation and a second write path next to the inbox Durable Object, which already batches.
  • Self-hosted Sentry or GlitchTip; @microlabs/otel-cf-workers — not the Grafana stack; deprioritised by its maintainer.

Decision

A. One sleeping box: Loki + Grafana in a Container; R2 is the store

A new Worker handsontable-demos-o11y (workers/o11y/, own wrangler.jsonc with workers_dev: false and preview_urls: false, own deploy job) owns two Durable Object classes: InboxWriter (§B) and GrafanaBox, a Container class running Loki and Grafana on standard-1, pinned with jurisdiction: "eu". There is no collector in the box.

  • Loki, single binary, TSDB single store in an EU R2 bucket, two tenants: browser (browser and beacon streams) and worker (Worker streams). Configuration in §B.4.
  • Grafana served from /grafana/, provisioned from git (one Loki datasource per tenant via X-Scope-OrgID, the Altinity plugin against the Analytics Engine SQL API, dashboards), auth.proxy behind the Worker (§B.5), Grafana Live disabled, sqlite state disposable.
  • Secrets reach the container only as envVars set by GrafanaBox at start from Worker secrets (LOKI_S3_*). In production AE_SQL_TOKEN and the Slack webhook never enter the box (local mode still passes the token via HEADER2_VALUE).
  • Analytics Engine SQL goes through an outbound handler. A container's own egress to api.cloudflare.com is refused with Cloudflare error 1000 (dns_loop, HTTP 403), so the ClickHouse datasource points at http://ae.internal/client/v4/accounts/<id>/analytics_engine/sql. GrafanaBox.outboundByHost (src/ae-outbound.ts) intercepts that host, accepts only GET/POST on that one path, adds Authorization: Bearer ${AE_SQL_TOKEN} in the Worker and makes the real request. The Worker must export ContainerProxy for the interception to work. If the interception setup fails at wake, the box still starts, without the ae.internal route (the ClickHouse datasource errors, Grafana and Loki work), and the Worker logs o11y.ae_outbound.degraded.

Wake. Two triggers only:

  1. A Grafana visit through the login broker (ADR-0007 — not Cloudflare Access). The Worker renews the activity timer on every HTTP request to /grafana/*, including the waking page's own meta refresh poll while the box is still booting (counting a request only once the box is ready would let a visit wake with nothing yet in the backlog SIGTERM itself around 20s into boot, before the person who opened it ever saw Grafana); the box stops after 15 idle minutes, and after 4 hours awake regardless (the next request shows the waking page).
  2. A backlog: a */10 cron in the o11y worker asks InboxWriter.backlog() and wakes the box when the oldest uncommitted object is older than 60 minutes or the backlog exceeds 64 MB. The cron never wakes the box when drains are paused (§G).

Stop protocol, the same for every stop the Worker initiates: the drain finishes; if no Grafana request arrived in the last 10 minutes the Worker calls stop() (otherwise the idle timer does, later, so a drain never SIGTERMs someone reading a dashboard); the container's shutdown script stops Loki gracefully, confirms the index is uploaded to R2, and only then writes a clean-shutdown marker state/wakes/<wakeId>/clean into the Loki bucket, the one bucket its credentials reach. The o11y worker reads state/ through an R2 binding on the same bucket; no lifecycle rule touches that prefix except a 30-day expiry. The marker, not onStop, is what the ledger trusts (§B.3). Exception: a wake that ends without ever having a provisional key (an empty backlog, or a Grafana-visit-only wake with nothing to drain) never gets an index upload and so never gets a marker — that is expected, not an unclean stop, and the ledger now resolves such a wake as clean without requiring one. A wake that did push data still requires the real marker. A box that stays not ready for 10 minutes after its wake began (Loki never answers its readiness probe) is a further stop trigger: the drain reports one error and runs the same stop decision, unless a stop for that wake is already in flight.

Stop grace bound: the container side is bounded step by step: Loki's wait is O11Y_STOP_GRACE_SECONDS (120 s in production), each R2 call has a 15 s limit, and the Grafana wait is limited to 30 s (O11Y_GRAFANA_STOP_GRACE_SECONDS), after which the script logs and exits with the marker decision it already made. The worst case is 240 s (pre-SIGTERM snapshot) + 120 s + 240 s (post-exit snapshot) + 30 s (marker PUT and HEAD) + 30 s ≈ 660 s. The Worker schedules a one-shot destroy() 780 s after stop(), 120 s above that worst case. Cloudflare documents a 15-minute (900 s) SIGTERM-to-SIGKILL window, but that is unverified for a Worker-initiated signal()/stop() and may apply only to platform-initiated shutdowns. The backstop is correct either way: it is a Worker-side destroy() that does not depend on that window, so the container is gone by 780 s whatever the platform does. It fires only if that wake's container is still running and no newer wake has started. A destroy before the marker leaves the wake unclean, so the ledger replays it, which is the safe direction.

Waking page: the Handsontable logo, one line of text, <meta http-equiv="refresh" content="3">, no script, served by the Worker while the box is not ready.

Cost model: drain wakes × measured drain-wake duration + visit hours, at $0.038–0.074 per awake hour on standard-1. The target is ≈ $5–8/month at current traffic; exit criterion L.7 recomputes it from the measured drain-wake duration and fails above $10. Measured on the real sandbox platform, corrected 1×/10× traffic scale (§L.7): $0.21/month at 1×, $0.33/month at 10× — the design's own $5–8 target was itself a conservative upper estimate; drain-wake frequency is capped by the 60-minute backlog-age trigger, not by traffic volume, so 20× more records only adds ~16s of drain time per wake, not 20× the awake-hour cost. Per forgotten tab: every provisioned dashboard ships with auto-refresh off, so an open tab left idle costs nothing beyond the visit that opened it; a viewer who turns refresh back on keeps the box awake for as long as the tab stays open, up to the 4-hour hard cap — $0.15–$0.30 per forgotten tab at the awake-hour rate above.

B. Ingest never waits for the box

B.1 Routes, all owned by the o11y worker on the main hostname, passed as --routes flags (ADR-0020), beside the API worker's /api/*, /d/*, /embed/*:

Route Source
POST /telemetry/collect Faro from the authoring app
POST /telemetry/lite lite beacon from /d and /embed
POST /telemetry/v1/logs Cloudflare OTLP log export
POST /telemetry/deploy CI deploy events
POST /telemetry/hooks/sentry Sentry issue-alert webhook
/grafana/* Grafana, waking page
POST /grafana/_o11y/reopen manual ledger re-open (§B.3)

ADR-0038's WAF exception is extended from /api/* to /telemetry/*: Faro errors and Sentry payloads legitimately contain <script. The compensating controls are the gates in §B.5.

B.2 InboxWriter: one owner of the inbox lifecycle. One instance, named main, addressed through .jurisdiction("eu"). The CPU-heavy steps run in the stateless route handler, so ingest is not serialised through one object; InboxWriter only checks, stores and packs. For each accepted request, in this order:

  1. Decode and scrub in the route handler into OTLP log records (§C.1): convert Faro and beacon payloads; decode Cloudflare's export (protobuf or JSON, whichever spike (b) observes); keep only allowlisted attributes; hoist hot.* and service.* to resource attributes; run the server-side scrubber (§E.4). Nothing derived from the arrival time is added yet. Records over 256 KB are dropped.
  2. Hash each record (SHA-256) at this point, before any arrival-time value exists, so a redelivered body hashes identically.
  3. Stamp timestamps (§C.2), which may use the arrival time as a clamp or fallback. The arrival time itself never becomes part of a stored record; InboxWriter keeps it on the storage row.
  4. Deduplicate in InboxWriter: each hash is checked against a 24-hour set in DO storage; a record already seen is dropped. This is what makes a duplicate Cloudflare delivery a non-event.
  5. Append the records to DO SQLite storage in rows of at most 1 MB, and answer 2xx only after the transaction commits. Nothing is held in memory across requests.
  6. Pack, from a 60-second alarm or at 4 MB stored: write one gzipped NDJSON object per tenant to inbox/<tenant>/<yyyy-mm-dd>/<hh>/<seq>.ndjson.gz, where <seq> is a counter persisted in DO storage, incremented in the same transaction that records the key, zero-padded to 12 digits. The key's state is recorded as written; the packed rows are deleted. "4 MB stored" is a real, enforced bound on the packed object's own decompressed NDJSON size (PACK_OBJECT_MAX_DECOMPRESSED_BYTES), not only a flush-cadence hint — the alarm takes pending rows in arrival order up to that budget (always at least one row, even if a single row alone is over budget) and commits that object; a tenant with more pending rows than fit in one object is packed across several objects, looping within the same alarm invocation up to a per-invocation cap on packed objects and rescheduling itself immediately when rows remain, rather than one unbounded in-memory gzip per alarm. The alarm's own read of pending rows is bounded too: row:<n> is zero-padded (12 digits, matching <seq>'s own width), so native ascending key order equals arrival order without an in-memory sort, and the alarm pages row: in small chunks, accumulated up to one packed object's own byte budget per round, rather than one unbounded list() — see contract §8 and workers/o11y/src/inbox/pack.ts.

One writer means key order equals arrival order; storage-backed buffering means a deploy, eviction or host restart between two alarms loses nothing.

Ingest deadline. The v1/logs, deploy and hooks/sentry routes race the InboxWriter.ingest call against a 10 s deadline (INGEST_DEADLINE_MS, workers/o11y/src/inbox/ingest.ts; an AbortSignal cannot cancel a Durable Object RPC, so the call is abandoned, not stopped). A deadline miss answers 503 with Retry-After: 30 and writes an o11y.ingest point with outcome dropped, reason ingest_timeout; a rejected call answers the same 503 with reason ingest_error. Accounting caveat: if the object commits and only the reply is lost, that batch is counted as ingest_timeout, and its redelivery is deduplicated and counted as duplicate, so a committed batch can go uncounted as accepted. Cloudflare's OTLP exporter is undocumented on its timeout, retry count and backoff, whether it back-pressures the source Worker, and what it treats as a failure; that stays a known unknown, watched passively through the "Observability self" dashboard's o11y.ingest outcome panel. /telemetry/collect uses the same deadline; its own catch (500, invalid_item) remains only as a backstop for Faro body processing. Invalid and oversize records and Analytics Engine points for items without an inbox item are written before ingest, so a Faro retry after a timeout can double-count them. Points for items with an inbox item (browser metrics, hash-only example.* events) are written only for items the ingest reports accepted. When the deadline fires, the abandoned call is kept alive with ctx.waitUntil and, if it commits, writes the points of exactly its accepted items; the client's retry then comes back duplicate and writes none, so they are counted once. A call that rejects later writes none.

Analytics Engine points for browser metrics are written by the route handler after step 3 (§F.1), so they exist while the box sleeps. example.* events (ADR-0042) produce Analytics Engine points only and are never packed into the inbox. The exact first-seen registry for error fingerprints (§F.3) is updated at step 4.

B.3 Drain and ledger. The ledger lives in InboxWriter storage, next to the keys it describes, so backlog and state are readable without starting the container. Each key is written → provisional(wakeId) → committed, or rejected.

  • At each cron tick and at the start of each wake, InboxWriter resolves every wake that still owns provisional keys and is over — a newer wakeId has started, or GrafanaBox's container state reports it not running: marker present → those keys become committed; marker absent → they go back to written (re-opened). A wake that is still running is left alone. POST /grafana/_o11y/reopen re-opens a time window by hand.
  • backlog() counts only written keys, after that resolution step, so a box kept awake by a visitor never triggers wake attempts for its own provisional keys, and a crashed wake's keys count again as soon as the crash is noticed.
  • The drain runs in GrafanaBox, as a Durable Object alarm loop that handles a bounded number of objects per invocation and reschedules itself, under the o11y worker's limits.cpu_ms (set to the value spike (b) needs, at most 300 000). It runs only on a freshly woken box, before anything else is pushed: it replays re-opened keys first, then new written keys, in key order, each object's records pushed to Loki's /otlp/v1/logs with the tenant header in requests of at most 1 MB decompressed. A key becomes provisional(wakeId) only after every one of its requests returned 2xx. 429 and 5xx are retried with backoff within the wake; a 400 (for example too_far_behind) is logged with Loki's message. Two cases defer a key instead (it stays written, nothing is rejected, the batch goes on): a 429 whose body names Loki's stream limit, which is never retried, and an inbox object that cannot be read (a failing source-map read defers the same way). A stream-limited tenant's keys, and every deferred key, are skipped for the rest of the wake so the keys behind them keep draining (contract §8, "Drain refusals"). A single too-old record inside an otherwise-good packed object does not 400 (and so reject) the whole key — drainKey drops individual log records older than reject_old_samples_max_age minus a margin before pushing, counts the dropped ones on the o11y.drain point, and still pushes the good siblings in the same key. A key is marked rejected only when the push itself still 400s after that filtering (a genuine, not-just-stale, rejection), which raises an alert (§F.3). Within one wake, a per-record hash set guarantees no record is pushed twice.
  • What an unclean stop costs. The whole wake's keys are replayed on the next wake. Data that Loki had already indexed before the crash is then stored twice, in different chunks; queries return it once, because timestamps, labels and structured metadata are deterministic (§C.2) and shard_streams is off. The duplicate storage stays until R2 lifecycle deletes it, not until compaction. Re-opening across a Loki configuration or label change is not allowed, because it would break that determinism.

B.4 Loki configuration: ingester.wal.flush_on_shutdown: true; limits_config.shard_streams.enabled: false; max_chunk_age: 2h (a 1-hour out-of-order window, enough because the drain replays in key order into an empty ingester); reject_old_samples_max_age: 7d; query_ingesters_within: 168h, so backfilled data is queryable while the box is awake; ingestion_rate_mb and ingestion_burst_size_mb raised to what spike (b) measures, at least 16/32; max_line_size: 256KB; otlp_config promoting the hot.* resource attributes to labels. Retention through R2 lifecycle, not the compactor: one rule per tenant chunk prefix (browser 30 days, worker 90 days) and the index prefix at 90 days, with max_query_lookback per tenant so nothing past retention is queried. For days 31–90 the shared index still lists browser chunks that lifecycle has already deleted; max_query_lookback: 30d on that tenant means no query ever reaches those entries, and the index bytes they cost are negligible, so the index is not split per tenant. Compactor retention stays off: its markers would live on ephemeral disk, and retention_delete_delay outlasts any wake.

The schema has two periods. The first set index.prefix: "index/"; a slash in the prefix makes the compactor fail to map the table back to a schema period, so it never compacted. The second period (from: 2026-10-03, same store, schema and 24h period) uses index_, so new tables are index/index_<day>/… where the old ones are index/index/<day>/…. Both periods stay in the config to keep old data readable, and the stop check lists both prefixes per day because records up to 7 days old still land in the old-prefix table after from. The new period must be live in the deployed image before from (2026-10-03 00:00 UTC): if the deploy slips past it, move from to at least one day after the real deploy date before merging, otherwise index entries written under index/index/<day> for days at or after from become unreachable. Until from, runs write the old prefix; the stop-roundtrip script covers the new one with a seeded key. The stop check's worst case is 16 listings of up to 15 s each, 240 s per snapshot before and after SIGTERM; with the 120 s Loki grace that is about 600 s, inside the platform's 15-minute SIGTERM-to-SIGKILL window. The old period can be dropped once nothing older than the index/ lifecycle rule (90 days) is left under index/index/ (follow-up, not before 90 days after from).

B.5 Gates, every route authenticated or gated:

Route Gate
collect, lite Origin/Referer host is the production host (or localhost in the local environment), payload environment matches, BOT_RE user-agent filter, size caps, item-kind allowlist, unknown attributes dropped, the Workers rate-limiting binding, then the server-side scrubber. navigator.webdriver is a browser-side check only (§E.4).
v1/logs x-o11y-secret header set on the export destination, constant-time compare
deploy GitHub OIDC token (issuer, audience, repository, workflow), secret fallback
hooks/sentry sentry-hook-signature HMAC
/grafana/*, reopen the Worker's own HMAC-signed session cookie (O11Y_SESSION_SECRET), minted once from a Handsontable login broker token (ADR-0007, K1 — not Cloudflare Access); a client-sent auth.proxy header is stripped

Every drop writes an o11y.ingest point with the gate as the reason.

B.6 The observer does not observe itself: the o11y worker exports no Workers Logs and no traces, with invocation logs off and its own lines persisted in Cloudflare's dashboard only. Its self-metrics (o11y.*) go to Analytics Engine like every other metric, which is not a loop: they never pass through its own ingest routes.

C. OTLP from the Worker inward

C.1 Hops.

Hop Format
Authoring app → o11y worker Faro JSON (the SDK's native transport)
Embeds → o11y worker the §C.5 beacon payload
API worker → o11y worker OTLP logs, Cloudflare's export
o11y worker (ingest) everything normalised to OTLP log records, stored as OTLP JSON in the inbox
drain → Loki OTLP /otlp/v1/logs, synchronous
metrics Analytics Engine points, written at ingest by the o11y worker (browser) and directly by the API worker (server)

C.2 Attributes, identity and time. Every record carries service.name, service.version, deployment.environment.name and the hot.* set (surface, tier, framework, ht_major, outcome) as resource attributes. Per exit criterion 15's own precise wording, Loki labels service.name, deployment.environment.name and every hot.* key — seven of the eight — from containers/o11y/loki/loki-config.yaml's own otlp_config.resource_attributes promotion list. service.version is a resource attribute (queryable, present on every record) but is deliberately never promoted to a label: it is per-deploy-SHA, and a label with that cardinality would fragment Loki's index into one stream per deploy. Exit criterion 15 checks exactly this seven-key set arrives as labels, confirmed live against the real committed config for all four sources (§L). The exact names, allowed values and Analytics Engine slots live in docs/observability-contract.md. hot.demo_id, session.id and cf.ray are structured metadata only; never labels, never Analytics Engine indexes. The user pseudonym, emails, IPs, user-agent strings, query strings, authored code, chat text and console output are never sent to the o11y stack.

  • session.id is an in-memory id minted per page load by the app (§E.4). It joins a page load's browser records with the API calls that page made (sent as x-hot-session); it is not a visitor session and does not survive a reload.
  • service.version is the deploying GITHUB_SHA on each deployable. Authoring and API deploy independently (path-gated in master.yml), so their versions usually differ; each deploy job posts {service, sha, cf_version_id} to /telemetry/deploy, which becomes a Loki line and a Grafana annotation.
  • Timestamps are event time, deterministically: browser item timestamps are clamped to the envelope's received_at ± 5 minutes; beacon timestamps likewise; OTLP records keep time_unix_nano, falling back to observed_time_unix_nano, then to received_at, so no record ever reaches Loki without one.

C.3 Symbolication in the Worker, at drain. CI uploads the authoring build's maps to the EU maps bucket under sourcemaps/<sha>/<original asset path>.map before deleting them from dist (the Sentry Vite plugin's in-build deletion is turned off; one CI step uploads to both destinations, then deletes). At drain, for exception records only, the o11y worker resolves app-chunk frames with @jridgewell/trace-mapping against the map for the record's service.version, parsing lazily per file and caching in the isolate within a fixed memory budget; frames from the Babel compiler chunk and third-party files are left as they are. Maps expire with the browser tenant (30 days). Symbolication never runs on the public ingest route. Exit criterion L.5 bounds its cost.

C.4 Traces are not exported, and Tempo is not built. No trace destination is configured. Worker traces are sampled at 1 % into Cloudflare's own dashboard for the 7-day view while persist: true holds (§H).

Span.spanContext() shipped in workerd 1.20261009.1 (wrangler 4.151.0 bundles 1.20261012.1) with no compatibility flag. It is read-only, and an untraced span returns an all-zero context, so at 1 % sampling most requests have no usable id. The browser join is therefore a correlation, not a parent/child link: the API worker puts the active span's trace id, when real, on its api.request line as trace.id, next to session.id. That is the whole feature. A sandbox Worker with traces at 100 % returned a distinct, non-zero trace id per request, and the same id after an await inside ctx.waitUntil.

Decision: Tempo and trace export are dropped. Cloudflare's dashboard already covers the 7-day trace view, and the trace.id / session.id pair in Loki already joins a browser session to a Worker trace. Tempo would add a container and a new traces ingest route on the o11y worker (the span-attribute allowlist could not run on Cloudflare's native export path, which bypasses our Workers), and spans bill from the same 20 M pool as log lines, whose exported-logs half §D already projects as exceeded at 10x (about 6.6 M against 5 M). No production measurement of span volume exists, and the gain does not justify the cost and risk. traceparent propagation from the authoring app is dropped too: spanContext() is read-only, so the browser cannot parent a Worker span. Reopen this only with a concrete need for trace retention beyond 7 days, and then with measured production volumes.

C.5 Lite beacon for /d and /embed. The existing ES5 reporter in monitor.ts gains a standalone mode: with no parent runner frame, it sends uncaught errors (up to the existing ceiling) and sampled web vitals (10 % of page views) with navigator.sendBeacon to same-origin /telemetry/lite, under 2 KB per payload. It is injected at the serve seam in share.ts with the monitor-inject.ts guards and the DEV-2580 rules (self-removing tag, no whitespace). Docs pages link with noreferrer and send strict-origin-when-cross-origin, so no embed payload carries a docs page path; embeds are identified by demo id.

D. Worker signals

  • observability.logs: head_sampling_rate: 1.0, invocation_logs: false, persist: true, destinations: ["o11y-logs"].
  • observability.traces: head_sampling_rate: 0.01, persist: true, no destination.
  • One structured JSON line per non-proxy request (route class, status, duration, cf.ray, session.id, hot.demo_id, service.version), replacing the bracketed-prefix convention, plus an api.request Analytics Engine point. The preview proxy path emits nothing per request; stale-preview requests are answered before the Sandbox SDK where the code can recognise them, so its per-request warning does not fire.
  • Every error that escapes a handler is logged as a structured line by our code: the fetch catch-all, the snapshot-job alarm's report path, the Durable Object alarms and the cron handler. The Worker half of constraint 4 therefore does not depend on how Cloudflare exports an uncaught exception with invocation logs off.
  • Custom spans (tracing.enterSpan) around session start, container boot, snapshot build, chat, theme AI, import and payload boot, for the dashboard view.
  • A */5 cron in the API worker, which owns the KV session meters and D1, writes pool.gauge, budget.gauge (tier, percent of ceiling) and checks the o11y heartbeat (§F.3).
  • Volume budget (exit criterion L.8): at current traffic, Workers Logs events (lines plus spans) stay under half of the 20 M pool, exported log events under half of the 10 M logs allotment, Analytics Engine points under half of 10 M. Tier-2 container stdout is counted in the first. Because every count and alert reads Analytics Engine, which is not sampled at ingest, lowering the log sampling rate is the fallback that costs text, never alerts. Measured (projected from real per-session/per-request counts × traffic-baseline.md, at the ADR's own required 10× headroom): Analytics Engine points pass comfortably (≈4.14M of the 10M dataset, well under half). The raw Workers Logs pool also passes (≈6.6M of 20M, under half) — but the exported-logs allotment does not clear its own half at 10×: real measured Tier-2 container stdout (12–22 lines for a Vite-family starter's boot alone, ~22 for a webpack/Angular-family starter's boot, plus 2 lines per 60-second keepalive poll — Cloudflare's own Sandbox SDK's structured logging of its own health checks, not the dev server's own output) pushes the projected 10× total to ≈6.6M/month against the 5M half of the 10M exported-logs allotment. This is exactly the situation this paragraph's own fallback exists for: lower head_sampling_rate before or during launch if real production volume confirms this projection, which drops exported/logged text but never drops a count, an Analytics Engine point, or an alert. See docs/run-and-deploy.md's Launch plan for the concrete pre-launch measurement and the sampling-rate action.
  • The comment at wrangler.jsonc:10-13 ("full fidelity is a spike amplifier") is answered by invocation_logs: false, the silent proxy path and the budget above, not reversed.

E. Sentry: what "uncaught" means, what moves, what stays

E.1 Definition. Uncaught means an error that escapes a handler:

  • Browser: window.onerror, unhandledrejection, and render crashes caught by Sentry.ErrorBoundary.
  • Worker: anything that escapes a fetch, alarm or scheduled handler, including errors the fetch catch-all turns into a 500 and snapshot-job failures the alarm reports without rethrowing.

These stay in Sentry, keep its grouping and regression detection, and also reach the new stack (§E.2).

Moves to the new stack only: diagnostic reports about handled conditions — upstream failures reported with tags (npm registry, import URL), the preview boot-window report, reportError calls for recoverable UI failures, and demo-runtime preview events. They become Faro reports or structured lines with an error.handled point, and Sentry no longer receives them at all; the exact new-fingerprint registry (§F.3) replaces its "new issue" signal as a dashboard table, not a push notification. The Outcome of each implementing task lists every call site and its classification.

Stays in Sentry, unconverted: the budget-alert captureMessage in reconcile.ts and its rehomeBudgetAlert hook. It is the spend alert channel.

E.2 Tee. Faro's errors instrumentation sees window.onerror and rejections; Sentry.ErrorBoundary's onError also calls the facade, so render crashes reach Faro. Worker errors reach Loki through the structured lines of §D. Sentry's beforeSend pushes the Sentry event id as a Faro event, and the Faro page-load id becomes a Sentry tag. The issue-alert webhook (new issue, regression, resolved) becomes a Loki line with issue id, title, release and link.

E.3 Switch (removed). The trim shipped behind SENTRY_SCOPE (API worker var) and VITE_SENTRY_SCOPE (authoring build), full by default, where moved reports went to both Sentry and the new stack. Both were flipped to uncaught on 2026-10-13, after data had been seen end to end in Grafana, alerts had fired and volume sat inside the budget. The flip held for a week, and the switch and its full code path were then deleted (DEV-3099): the authoring Sentry client always runs with only the global handlers and dedupe, and reportDiagnostic never calls Sentry. Rolling back means reverting that change.

E.4 Faro configuration and the operational-log rule.

  • Faro: session tracking disabled; the facade mints the page-load id in memory and sets it on every item; no user meta; only the errors and web-vitals instrumentations (Performance, CSP, console and view instrumentations off, because they send full URLs or console text); transport to same-origin /telemetry/collect.
  • One scrubber, scrubTelemetry, runs in the browser and authoritatively at ingest for Faro, beacon and OTLP records alike: strip query strings and fragments from every URL-valued field; redactPreviewHosts on every string; reduce any browser meta to the device and browser classes analytics.ts uses; remove Babel code frames explicitly (stripCodeFrame: the gutter-numbered source lines and caret markers), because normalizeMonitorMessage does not; drop unknown attributes.
  • The rule: operational logs may carry a page-load id, a demo id and a cf-ray as structured metadata, and nothing else that identifies a person or a request's content. Browser streams are kept 30 days, Worker streams 90 days. This is a separate class from analytics, which keeps its rule unchanged: example.* events are counts only and never reach Loki.
  • Local testing: Faro runs on a local path only when the build was made with VITE_TELEMETRY_LOCAL=1 and the host is localhost/127.0.0.1, with environment local. That path checks neither import.meta.env.DEV nor navigator.webdriver, because Playwright serves a production vite preview under automation. The production gate (resolveReporting) is unchanged and stays closed under automation; a post-build check fails if the local path survives into a production bundle.

F. What is metered

F.1 Store. Counts and latencies go to Analytics Engine; Loki holds the text. (Contract §6's own table matches this ruling — a Faro measurement or web-vitals item is AE-only, never a stored Loki record.) The positional slot layout and the full metric registry, with allowed outcomes, are in docs/observability-contract.md §4–§5; this section names the signals, the contract fixes their shape. Every count is SUM(_sample_interval * count), every percentile a weighted quantile, and every query goes through one helper that allowlists Analytics Engine's documented functions. Browser metrics are extracted from Faro measurements at ingest (§B.2); server metrics are written by the API worker.

F.2 Catalogue.

Journey Signals
Play preview.ready_ms (pick → data-preview-status="ready", by tier/framework/ht_major — the headline metric); sandpack.compile_ms; sandpack.compile_error (fingerprint, no code); sandpack.bundler_unreachable; preview.runtime_error groups (ladder-deduped); version.switch; bucket.resolve_ms
Edit live session.start (server; outcomes ready, at_capacity, container_starting, boot_timeout, budget_denied, error) and session.start_ms (client, cold/warm); container.boot_ms; hmr.roundtrip_ms where a reliable hook exists; session.end with reason and awake seconds; pool.gauge every 5 min
Share & build snapshot.build; serve.share, serve.d, serve.embed (demo id as a blob); web vitals on /share and /d
Embed on docs beacon error.uncaught and web_vital by demo id and ht_major; broken-embed and slow-embed lists by demo id
Assist chat.answer (model, tokens, USD, latency, outcome), chat.edit, theme.ai, import.url, payload.boot
Examples ADR-0042
Platform api.request (route class, status class, duration); error.handled; budget.gauge every 5 min; reconcile.run; o11y self: o11y.ingest, o11y.drain, o11y.wake, o11y.backlog, o11y.alert

Sampling: authoring Faro 100 %; beacon errors 100 %, vitals 10 %; Worker lines 100 %; Worker traces 1 % into Cloudflare's dashboard only.

Dashboards, provisioned from git: Runner overview (with deploy annotations), Tier-2 sessions, Tier-1 playground, Version health (framework × ht_major, next highlighted), Docs embeds, AI assist, Examples & features (ADR-0042), Observability self. Cost moves to ADR-0043, because spend truth is D1.

F.3 Alerting runs outside the box, with state. Grafana holds no alert rules: its state would not survive a sleep.

Signal Owner Latency
Uncaught error, new issue, regression Sentry seconds
Spend thresholds (200/500/800) reconcile.ts captureMessage to Sentry, as today nightly
at_capacity rate, 5xx rate from api.request, preview-ready rate per tier, Tier-2 liveness (no ready preview in 6 h), Tier-2 slow previews (ready p95), session start p95, embed error rate per demo id, compile-error rate per ht_major day over day, snapshot-build failed rate per framework, a wake or reload that ran without the ae.internal interception (o11y.ae_degraded), LiteLLM error rate (chat.answer + theme.ai), inbox backlog age, a rejected inbox key, the o11y spend cap o11y worker */10 cron over Analytics Engine and InboxWriter → Slack minutes
New handled-error fingerprint the exact first-seen registry in InboxWriter (not sampled data), excluding surface = demo-runtime, whose keystroke ladders are authored-code output. Not a page: the */10 cron writes one o11y.new_fingerprint point per new fingerprint and the Observability self dashboard lists them in a table; nothing goes to Slack minutes, dashboard only
The o11y stack itself stale (no cron tick or ingest for 30 min) the API worker's */5 cron reads the o11y heartbeat over a service binding and sends captureMessage to Sentry minutes

Alert state (firing, resolved, last notified) lives in Durable Object storage: a rule notifies once when it fires and once when it resolves, never on every tick. Thresholds are starting values, tuned after launch: preview-ready below 97 % (Tier-1) or 95 % (Tier-2) over 6 h, evaluated per tier only with at least 10 non-abandoned previews and at least 3 failed (error or timeout) in that window (preview.ready_ms carries no demo or session id, so the failures cannot be required to span distinct sources); Tier-2 liveness fires when no Tier-2 preview reached ready in 6 h with at least 5 non-abandoned attempts; Tier-2 slow previews fire when the ready time-to-ready p95 is above 180 s over 6 h with at least 10 ready previews (a starting value above today's worst per-framework p95 of 167 s); session start p95 above 20 s, evaluated only with at least 20 ready starts in the hour (below either floor the rule is not firing, so a firing alert resolves through the normal path); at_capacity above 5/h; 5xx above 1 % over 1 h, evaluated only with at least 100 requests after the exclusions (at 1 % one error exceeds the threshold only below 100 requests, so a single 500 cannot page); LiteLLM errors above 5 % over 1 h, evaluated only with at least 20 non-denied calls (same reasoning at 5 %); compile errors on one ht_major doubling day over day, with at least 20 today; any o11y.ae_degraded point in 2 h (written only at a wake or reload, so a short window would resolve before anyone read the page); snapshot builds failing above 50 % per framework over 30 min with at least 10 failed; an embed above 20 % errors with at least 50 views in 7 days; backlog older than 2 h.

G. Cost is its own number, with its own cap

  • recordContainerUsage takes the SKU as a parameter; o11y_container carries the box's awake seconds, reported by the o11y worker to the API worker over the API service binding (the API worker owns D1 and KV; the o11y worker binds neither).
  • reconcile.ts iterates over the scripts it reconciles, handsontable-demos-api and handsontable-demos-o11y, and writes each script's billing rows under distinct SKUs (o11y_container, o11y_workers), so the per-SKU upsert never overwrites the app's rows.
  • /admin shows app, observability and total.
  • O11Y_BUDGET_USD (default $15/month) joins the guardrail settings. When month-to-date observability spend crosses it, drains pause, visit wakes still work, and one Slack line is posted. Metrics and alerts keep working, because they read Analytics Engine, not Loki. Log text is kept only while the pause is shorter than the inbox's 7-day lifecycle and Loki's 7-day reject_old_samples_max_age; a longer pause loses the oldest log text, and that is the price of the cap.
  • Why the cap ($15) sits above the exit ceiling ($10): the ceiling is a design check at current traffic, measured once in exit criterion 7; the cap is a runtime brake. A cap at the ceiling would pause drains on the first busy month or heavy week of Grafana use, which is exactly when the logs matter. The $5 gap is headroom for traffic growth; a month that reaches it is a signal to revisit the design, not normal operation. Product tiers keep acting on the total. The bound is honest: drains are capped by the pause; visits are bounded by the 15-minute idle stop and the 4-hour limit (§A), not by construction.

H. Access, jurisdiction, retention

  • Login broker, not Cloudflare Access: every @handsontable.com account, as for /admin — the same Handsontable login broker (ADR-0007). A callback page under /grafana/_o11y/ reads the broker's fragment token once and exchanges it for the Worker's own HMAC-signed session cookie (gates/session.ts); Grafana Viewer via auth.proxy. This conforms to ADR-0007 rather than deviating from it. The earlier claim in this section — that the broker "hands a JWT to a SPA and cannot gate a proxied third-party HTML application" — was wrong: hot-mcp's own create_app runtime gates a proxied app the identical way, and a point-in-time production probe (the §M access-broker delta has the exact curl command and its 302 result, dated 2026-09-24) confirmed the callback host was allowed as of that date — expected to still hold, but re-run that probe before launch rather than assuming it (see the §M access-broker delta also for the one risk this design inherits rather than fixes, DEV-3088).
  • EU-pinned: the container (jurisdiction: "eu"), both Durable Objects, the inbox, Loki and maps buckets.
  • Deploy order for the mutual service bindings: the o11y worker first (binding the existing API worker), then the API worker with its O11Y binding.
  • Not EU-pinned, stated plainly: Cloudflare's own Workers Observability store keeps full lines and 1 % of spans for 7 days while persist: true, which is switched off 30 days after this ADR is accepted; Analytics Engine; the Workers edge; Sentry, whose ingest host is ingest.us.sentry.io (unchanged); D1, which has an EEUR location hint, not a jurisdiction, and cannot gain one later.
  • Retention: Loki browser 30 days, worker 90 days (§B.4); inbox objects 7 days; maps 30 days; Analytics Engine 3 months (platform); D1 rollups unbounded; analytics_visitors 180 days as today.

I. Local development

Layout: workers/o11y/ (Worker, both DOs), containers/o11y/ (Dockerfile, Loki and Grafana config, provisioning, compose.yml), pipeline/fixtures/otlp/, pipeline/o11y-*.test.mjs. The whole new stack runs locally with Docker: the box through wrangler dev or compose.yml; the o11y worker with Miniflare's R2, DO and cron; Loki on Miniflare's local S3 endpoint for R2 or MinIO; Analytics Engine replaced by a ClickHouse container holding an AE-shaped runner_events table, queried with the same SQL through the allowlisting helper; the login broker's session check by a fail-closed DEV_ADMIN bypass in .dev.vars. Cloudflare's OTLP export does not run locally; its fixtures are real bodies captured by the sandbox probe, scrubbed, plus hand-built edge cases. pnpm o11y:dev starts the box, the Worker and the fixture replay.

J. ADR-0042 and ADR-0043

ADR-0042 (example analytics) ships with this ADR: it needs the ingest path and Grafana, and its events are counts only. ADR-0043 (/admin reads in Grafana) follows after launch; the manual ledger re-open does not wait for it and lives on /grafana/_o11y/reopen.

K. Tests

Every implementing change carries tests that fail with the change reverted (docs/TESTING.md, the presence gate): gate tests per §B.5 row; InboxWriter tests for normalisation, dedupe, storage-backed buffering across a simulated restart, sequence persistence and key order; ledger tests for provisional/committed/re-opened/rejected transitions and marker handling; a label test asserting the Loki series for each source carry exactly the contract labels (exit criterion 15); scrubber tests per rule on real inputs (a Babel code frame, a preview-host URL, a user-agent string); a config test pinning the Loki keys of §B.4 and the observability block of §D; symbolication against a real vite build; beacon injection and an acorn ES5 parse; e2e/o11y-local.spec.ts, which drives a real browser against the real o11y worker and asserts the resulting points land in the local Analytics Engine stand-in (ClickHouse rows) — not a Loki query. Loki queryability itself is proven separately: containers/o11y/local/stop-roundtrip.mjs's own query_range calls against a real local Loki (the clean-stop/reopen roundtrip, criteria 1–2), and the local end-to-end walkthrough's live Grafana Logs-panel checks (docs/run-and-deploy.md's local-dev section).

L. Delivery and exit criteria

Probes. Cloudflare-only facts are measured with throwaway resources on the sandbox account: separate Worker names, a probe-only config, wrangler whoami before every deploy, synthetic traffic only, every resource deleted afterwards and listed.

Order: shared contract and scaffolds; the box and its stop protocol (spike a); ingest and InboxWriter (spike b); drain, ledger and Grafana access; API worker signals; Faro and the beacon; alerts and cost; ADR-0042; dashboards; CI and runbook; the local end-to-end walkthrough and launch.

Exit criteria, each a pass/fail with recorded evidence:

  1. Clean stop: lines pushed, clean stop, fresh wake → 100 % of lines queryable and the marker present, written with the production-scoped R2 token, not a local all-bucket credential. Plan B if Loki does not upload the index on graceful shutdown: the stop protocol waits for the next 15-minute index rotation and its upload before stopping, and wakes are thinned to backlog > 3 h or > 128 MB, which keeps the cost model within criterion 7; if that fails too, the serverless store replaces Loki.
  2. Unclean stop: SIGKILL mid-drain, next wake → re-open → count_over_time and a log query both equal a single clean replay.
  3. Event time: stored timestamps equal event time (clamped for browsers) for Faro, beacon and OTLP records, including OTLP records without time_unix_nano.
  4. Duplicate delivery: the same export body delivered twice produces one copy.
  5. Symbolication: an exception from a real vite build resolves to src/… file and line using at most 500 ms CPU and 64 MB of isolate memory; Babel-chunk frames are skipped, not parsed.
  6. Cold start: wake-to-ready at most 90 s, worst of five runs on Cloudflare.
  7. Drain wake: one hour of production-shaped traffic drains in at most 5 minutes of wall time, no alarm invocation exceeds the configured cpu_ms, CPU per object is recorded, and the cost model recomputed from the measured duration stays at or under $10/month at current traffic.
  8. Volume: each allotment of §D under half, projected from measured per-session and per-request counts times production traffic.
  9. Idle tab: an open, idle Grafana tab lets the box stop at the 15-minute idle timeout.
  10. Placement: the Container's Durable Object namespace accepts .jurisdiction("eu") and the instance runs in an EU region.
  11. Worker errors: an error thrown in a fetch handler, a DO alarm and the cron handler each produce a structured line in Loki with invocation_logs: false.
  12. Stop semantics: what onStop reports for our own stop() is recorded, and a Worker-initiated stop does not escalate to SIGKILL before the clean marker is written. A stop that outlasts the 780 s bound is destroyed and resolved unclean.
  13. Retention: R2 lifecycle deletes expired chunk prefixes per tenant, and queries past max_query_lookback return nothing.
  14. Image: compressed size recorded and at most 1 GB, and it fits the instance disk.
  15. Labels: records from each source (Faro, beacon, Cloudflare export, deploy events) arrive in Loki with every hot.*, service.name and deployment.environment.name label populated, and hot.demo_id, session.id and cf.ray present as structured metadata only, never as labels. Checked with Loki's label and series APIs, locally and on the sandbox probe with a real Cloudflare export.

If criterion 1, 2 or 7 fails with its plan B, the design is rewritten toward the serverless store before more is built.

Results (local end-to-end pass plus every sandbox probe):

# Criterion Result
1 Clean stop, production-scoped token PASS — sandbox probe with a real bucket-scoped R2 token (T03B), local compose/wrangler dev both pass (T03-D2 fixed: the drain now posts a real OTLP resourceLogs envelope, not bare NDJSON)
2 Unclean stop, reopen PASS, fully — real SIGKILL mid-drain on the sandbox platform (T03B): reopen, replay, count_over_time and a log query both equal one clean replay. Independently reproduced locally (T11): an interrupted wake's canary record reopens and replays to exactly one Loki line, both query forms agreeing
3 Event time PASS (local) — clamped browser timestamps, un-clamped OTLP time_unix_nano (T02); a real, old-dated captured OTLP fixture was genuinely rejected by Loki's 7-day window this pass, which is only possible if its stored timestamp preserved real event time
4 Duplicate delivery PASS (local) — the same body delivered twice produces one copy; re-confirmed live and repeatedly this pass (o11y.ingest outcome duplicate)
5 Symbolication PASS in production (2026-10-01) — browser exceptions in Loki (browser) resolve to src/App.tsx:2659:34 and src/App.tsx:327:20. Symbolication runs in the GrafanaBox drainStep, a Durable Object alarm: 497 alarm invocations (2026-09-29 12:30 to 2026-09-30 10:30 UTC) had CPU p99.9 22.6 ms and a per-minute peak of 22.6 ms, and a wrangler tail of a backlog drain showed 25 ms CPU, against the 500 ms budget. The only invocations over 500 ms are the first GrafanaBox.isAwake poll after a box start (520–661 ms; 580 ms in the tail), which is not on the symbolicator path. Memory: the o11y Worker peaked at 9.04 MB (limit 64 MB), but the Durable Object dataset has no memory field, so the alarm's own memory is not measured; no out-of-memory outcome was observed
6 Cold start PASS — sandbox: 46.5s worst-of-5 (T01), 3–22s after the T03-D3 fix (T03)
7 Drain wake (time + cost) PASS at the corrected traffic scale — sandbox (T03B): 28s wake-to-drain-complete at 1× (≈432 records/hr, T05's own per-session line count), 44s at 10× (≈4325/hr); cost $0.21/month at 1×, $0.33/month at 10× — both far under the $10 ceiling
8 Volume Mixed, measured, not a breakeven guess — Analytics Engine points and the raw Workers Logs pool both pass at 10× with real margin; the exported-logs allotment does not (§D above has the numbers and the fallback); production, 2026-10-01/02: the API Worker's export is not affected by the o11y Worker's "only 1% of events are being recorded" banner (a per-day cap that reset; Loki (worker) held 3569 api.request lines and Analytics Engine a weighted 3563 for the same hour). The monthly count for o11y-logs is at most ~0.85M events/month at current traffic (Observability → Usage per service, since Cloudflare exposes no per-destination count: 84,225 handsontable-demos-api log events in the 7 days to 2026-10-02, export live since 2026-09-29; ~0.36M/month if spread over all 7 days), against the ~5M/month half-allotment. That confirms the §D projection: at 10× it is 3.6M to 8.5M, straddling the line, so the row stays Mixed and lowering head_sampling_rate stays the named fallback, needed only if traffic grows
9 Idle tab PASS in production (2026-09-30) — /grafana/dashboards loaded at 10:40:09 UTC and the tab was left untouched; the production GrafanaBox was running until 10:54:26 and stopped at 10:55:29 UTC, 14.3–15.3 min after the last request. Sandbox (T01): 17.65 minutes with zero HTTP requests
10 Placement PASS — sandbox: EU region mxp04 (Milan)
11 Worker errors → structured line PASS — fetch-handler and cron paths confirmed live (T05, T11); in production (2026-09-30) one anonymous GET /api/versions/exists?v=<600 characters> returned 500 and produced both a Sentry event (api-production) and the fetch-catch-all error line plus the api.request 500 line in Loki (worker). The DO-alarm path is unit/pipeline-tested (T01–T03) but not independently reproduced live
12 Stop semantics PASS — onStop is recorded and, by design, claims nothing about cleanliness (T01); "no SIGKILL before the clean marker" is the same platform behaviour criteria 1 and 2 already confirm
13 Retention PASS in production (2026-10-21) — at 2026-10-21 09:27 UTC the production handsontable-demos-o11y-inbox held 8,476 objects (8,227 inbox/worker/, 249 inbox/browser/); the oldest, inbox/worker/2026-10-14/09/000000014907.ndjson.gz, was written 2026-10-14T09:16:55Z, 7 days and 11 minutes earlier, and every lower sequence number written since 2026-09-29 is gone. 10 objects were 0–11 minutes past 7 days, within R2's asynchronous lifecycle lag. Loki's own 30/90-day expiry and max_query_lookback cannot be read in production until the oldest data is that old (from 2026-10-29 for browser/); they rest on the read-back rules below and the sandbox/local evidence. Earlier: R2 lifecycle rules apply and read back correctly (T01, T10); the production rules were read back on 2026-09-30: inbox inbox-7d, loki browser/ 30 d, worker/ 90 d, index/ 90 d, state/ 30 d, maps maps-30d. Real expiry on the sandbox (2026-10-02): o11y-probe-t03-loki still has its 1-day rule and holds 0 objects, 0 B, where the test objects were written on 2026-09-23 (consistent with expiry; bucket info alone can't rule out a manual delete, and the bucket's own delete reported "not empty" at 0 objects, likely leftover multipart uploads). The sandbox test (t03-retention-clock-test/) started 2026-09-23T14:15:22Z
14 Image size PASS — 212.9 MB compressed, real linux/amd64 build (T01), under the 1 GB bound; uncompressed size against the standard-1 8 GB disk was not separately recorded by any task
15 Labels PASS, all four sources, both tenants — confirmed live against the real committed loki-config.yaml: Faro (demos-authoring) and the lite beacon (demos-embed), browser tenant; the Cloudflare export (demos-api) and deploy events (demos-o11y), worker tenant — all seven labels populated, service.version present as a resource attribute but deliberately never promoted to a label (see §C.2), hot.demo_id/session.id/cf.ray never labels

§L's own trigger (criterion 1, 2 or 7 failing its plan B) is not engaged — all three pass. The last production reading, criterion 13's real retention expiry on the production inbox bucket, passed on 2026-10-21 (DEV-3178), so this ADR is Accepted. Criterion 8's exported-logs finding is real and measured, not a missing-evidence gap: the monthly count (about 0.36M to 0.85M events/month today) is recorded in the row above and confirms the §D projection, so lowering head_sampling_rate stays the named fallback, to be used only if traffic grows toward the 10× case. The row is carried as a named action in docs/run-and-deploy.md, not as a blocker to this ADR's status.

M. Implementation deltas (full detail in git history under

the deleted runner/tasks/o11y/)

Deltas already folded as direct edits above (§A cost, §A wake/stop, §B.3 drain-rejection, §C.2 labels, §D volume, §L results) are not repeated here. The rest, grouped by section, where they add information beyond what §A–§L already say:

  • §B.2 ingest. Hashing (step 2) uses each record's own raw, un-clamped source timestamp alongside its body and attributes — not the clamped time_unix_nano a later step computes — so two real deliveries of the same content at different real times still hash differently, and the same body redelivered still dedupes. One aggregated o11y.ingest point is written per request (not per record), so a batch of N duplicate records reads as one duplicate point with count = N, not N separate points. Every §3 resource attribute a source has no natural value for defaults to "none" ("unknown" for service.version specifically, confirmed against real Cloudflare export samples that never carry it at all) — this default is what makes exit criterion 15 pass for worker-origin sources, not a defensive fallback. A real Cloudflare OTLP export's ray id arrives as cloudflare.ray_id, remapped to the contract's own cf.ray. HotAttrs fields with no dotted hot.* resource- attribute counterpart (bucket, reason, fingerprint, and others with no browser call site emitting them yet) travel over an AE-only channel, read from a Faro item's raw context before the browser's own scrub allowlist would otherwise drop them — this channel needed its own allowlist extension (AE_ONLY_ATTRIBUTE_KEYS) before it worked for real.

  • §B.2 pack. The pack alarm's "at 4 MB stored" trigger (step 6 above) is a real, enforced upper bound on a single packed object's decompressed size (PACK_OBJECT_MAX_DECOMPRESSED_BYTES = 4 MB), not only a flush-cadence hint. An over-threshold burst is capped by splitting into extra keys, not by cutting the alarm's own accumulation short: packTenant takes pending rows in arrival order up to the budget and returns only that prefix (consumedRowKeys); InboxWriter.alarm() loops packTenant/commitPackedObject per tenant over the leftover rows until nothing remains or a 25-packed-object per-invocation cap is hit, rescheduling the alarm immediately (setAlarm(Date.now())) when objects still remain. A single row over budget on its own is still packed alone (row size is already bounded to INBOX_ROW_MAX_BYTES, ~1 MB, well under the 4 MB object budget) rather than blocking progress.

  • §B storage API, DO 128-key batch limit (confirmed platform fact). Cloudflare's SQLite-backed Durable Object storage API caps get/put/delete at 128 keys/key-value pairs per call (https://developers.cloudflare.com/durable-objects/api/storage-api/, fetched 2026-09-24: "Supports up to 128 keys at a time" / "up to 128 key-value pairs at a time"). Local workerd accepts 500+ in one call with no error, so this codebase's own test doubles (workers/o11y/src/inbox/storage.ts#memoryStorage(), pipeline/fixtures/o11y-harness.mjs) enforce the same cap explicitly rather than relying on workerd to catch a violation. Every multi-key call in InboxWriter — dedupe's checkDuplicates, the fingerprint registry's writes/prune, pruneLedger, wake resolution, markKeysProvisional, manual reopen, the pack commit, and ingest's own transaction put — chunks through storage.ts's getManyChunked/putChunked/deleteChunked. finalizeWakeResolution and reopenWindow run their whole put+delete sequence inside one storage.transaction() — chunking alone, without that, would let a crash between chunks leave a partial write (an orphaned provisional:<wakeId> key whose wake:<id> is already gone).

  • §B.3 drain reads whole objects. Each drained key's packed object is read into memory whole before its records are pushed to Loki — this is bounded, not unbounded, because the object it reads was itself capped at write time (PACK_OBJECT_MAX_DECOMPRESSED_BYTES, ~4 MB), plus at most one further oversized single row (INBOX_ROW_MAX_BYTES, ~1 MB) packed alone when it alone exceeds the object budget. So one drain-time read is bounded to roughly 4–5 MB, never the whole tenant's backlog at once. This bound is a property of the PACK side (inbox/pack.ts), restated here as the drain-side consequence, not as a claim about drain.ts's own internals.

  • §B.2 ingest, worker tenant. A Worker's own console.log(JSON.stringify(...)) line (the structured request/error lines §D describes) arrives through Cloudflare's real OTLP log export as opaque body text, not as OTLP attributes — confirmed with a real captured export. The o11y worker parses a JSON-object body and merges its keys into the same attribute bag a real OTLP attribute would land in, through the existing allowlist, with every §3 resource-attribute key stripped from the parsed body first and given the lowest merge priority — a body key cannot spoof service.name/deployment.environment.name/ any hot.* label.

  • §B.2 ingest, worker tenant — fingerprint (live at merge). The API worker's own handled-error lines (reportDiagnostic, workers/api/src/telemetry/diagnostic.ts) carry hot.fingerprint (contract §3 AE-only key) in the same structured JSON body the bullet above describes. The read half (workers/o11y/src/normalise/otlp.ts#toIngestItem/apiFingerprintFeed) reads bodyJsonAttrs["hot.fingerprint"] (the pre-hoistAttributes bag tryParseJsonBodyAttrs already builds) and feeds it into the fp: registry only when ALL of: the REAL resource service.name === "demos-api" (read from finalResourceAttrs, the resource attribute after hoisting/defaults — never from bodyJsonAttrs, the same anti-spoof rule RESOURCE_ATTR_KEY_SET already enforces for every other resource attribute); the parsed body's log.kind === "error"; the value matches contract §7's own shape, via isValidFingerprint — ONE shared validator, also used by the browser path's resolveFingerprint, never a second, independently drifting copy: a validator anchored on the FIRST : would reject the :-joined call-site paths reportDiagnostic's own real callers send — "npm-registry:version-exists", "npm-registry:versions" — so neither could ever satisfy this condition, gate aside.

    Two preconditions, both must hold:

    1. Service-name remap: a real Cloudflare OTLP export's resource service.name is the deployed script's own name (handsontable-demos-api), not the contract's short demos-api — normalise/otlp.ts#remapCloudflareServiceName strips the shared handsontable- script-name prefix whenever what remains is one of the contract's own SERVICE_NAMES, applied before hoistAttributes, general across every deployable. Confirmed against the captured real-export fixtures (pipeline/fixtures/otlp/json/console-log-line*.json, cloudflare-invocation-log.json), every one of which carries the raw script name.
    2. Shared fingerprint validator: it accepts a :-joined context, so reportDiagnostic's own real call sites' fingerprints pass the shape check at all — see above.

    Without BOTH, this gate is a correctly-gated no-op against real production traffic, not a silent bypass; with both, it fires for real. This makes it LIVE at merge (the o11y worker's */10 new-fingerprint cron runs unconditionally) — docs/run-and-deploy.md's runbook keeps this as a standing check.

    The fourth condition — not Tier-2 container stdout — is attempted by construction, not guaranteed: the B cross-note fix (two bullets below) makes tryParseJsonBodyAttrs refuse to parse ANY body whose own log.kind isn't one of this worker's trusted shapes, so bodyJsonAttrs is empty for a body with no matching sentinel — but the sentinel is body TEXT, not a resource attribute, and (known gap, not fully resolved — see normalise/otlp.ts#tryParseJsonBodyAttrs's own doc comment for the full analysis) nothing in this pipeline can currently tell a genuine lines.ts line apart from a Tier-2 container's own authored stdout that happens to print the same shape, since both would share this Worker's service.name once the service-name remap normalises it. Accepted, bounded residual: a forged line can only mint a fp: entry and a notify-only, mrkdwn-escaped Slack line, the same noise class already accepted for the browser path — never Sentry, PII or code execution. Deliberately NOT hot.surface !== "demo-runtime" (the browser path's own rule) — a worker-tenant record's hot.surface resource attribute defaults to "none" when nothing sets it, which would admit any body reaching /telemetry/v1/logs, forged or not.

  • §C.1 hops. Faro's real browser transport posts a TransportBody ({meta, exceptions?, logs?, measurements?, events?, traces?}), not an array of self-contained items the way every contract function's own types assume — the ingest route reconstructs items from the four typed arrays.

  • §C.3 symbolication. A Faro exception's stack trace reaches the drain as V8-shaped text in the record body — the pre-implementation contract had no field carrying frame data for this to resolve at all.

  • §D Worker signals. container.boot_ms (not session.start's own boot_timeout outcome) is what fires when the Tier-2 boot window is exceeded — the original design would have double-counted a session that later times out after already reporting session.start ready once (a design correction made before shipping, not after). Two §5 reasons have no emitter at all: pool.gauge reason="builder" (no signal tracks BuilderSandbox concurrency the way live sessions are tracked; telemetry/cron.ts emits live only) and session.end reason="sleep_after" (nothing observes the Sandbox SDK's own idle-timeout stop) — named gaps, not silently dropped, and no data will appear for them. snapshot.build reason="inline" is emitted by the synchronous create/update path in share.ts and is observed in production; an earlier draft of this note listed it as unobserved. A cron failure inside ctx.waitUntil() is structurally unreachable by @sentry/cloudflare's own auto-capture (its scheduled instrumentation only wraps the synchronous handler invocation) — every cron branch calls Sentry.captureException explicitly in its own catch (confirmed live: a forced cron failure without this produced zero Sentry envelopes; with it, exactly one).

  • §E Sentry. The full call-site inventory found one real §11 violation: App.tsx's versions-fetch diagnostic was unconditional before this ADR's switch existed, exactly the shape §E.1 already names as "handled." Demo-runtime preview events kept their pre-ADR Sentry path (including the DEMO_SURFACE environment re-homing) until the §E.3 switch was removed; that path, the re-homing and its relay budgets are deleted.

  • §B.3 drain, a key with a mixed 400/2xx outcome ("accepted chunks skip §B.3"). Pushing every chunk of a key even after an earlier one 400'd, while still classifying the whole key rejected if ANY chunk 400'd — including when another chunk landed 2xx — would be wrong: a rejected key never becomes provisional, so those already-accepted bytes never pass the §B.3 marker/commit check any wake's clean stop confirms durability through: an unclean stop right after the push, before Loki's own local flush, could lose them with no automatic replay (only a manual reopen, which — being a full key replay — would re-derive the identical classification anyway). Instead: a key with at least one accepted (2xx) chunk stays provisional, following the normal durability path; only a key with ZERO accepted chunks stays rejected. The permanent chunk loss stays operator-visible via a rejectedEvent: audit log (ledger.ts#recordPartialReject, contract §8) rather than the key's own ledger state.

  • §B.3/§F.3 storage housekeeping, remainder ("prune ceiling"). Three gaps in the prune mechanism: (1) a 500-row/tick batch limit falls behind ADR §D's own 10× traffic projection at roughly 3× today's traffic — raised to 5,000/tick (still chunked to the real 128-key limit per call, see the DO storage batch-limit bullet above), with the exact arithmetic in dedupe.ts#HASH_PRUNE_BATCH_LIMIT's doc comment; (2) newFingerprintsSince listed the entire (alphabetically, not chronologically, ordered) fp: prefix every ten-minute alert tick — a new fpts:<firstSeenMs>:<fingerprint> time-ordered secondary index (contract §8) makes this a bounded range read instead, with a truncation-safe cursor in alerts/rules.ts#newFingerprintRule (never skips an unread fingerprint, at the cost of a bounded duplicate report under sustained flood — the same trade-off already accepted for the cursor's grace window); (3) rejected-inbox-key fired on rejectedKeyCount() > 0 and never resolved (rejected key: entries are deliberately never pruned by date alone — see the row-19 bullet's rejectedEvent: log and the §B storage API bullet above) — it now fires on a RECENT (last hour) count from that same log instead, resolving once new rejections stop.

  • §F metering. ADR-0042's example.* events needed the same AE-only attribute-channel extension as §B.2 above (kind→hot.metric_kind, since hot.kind is reserved for the Faro item kind, ref, area) before kind/ref/area survived the browser scrub at all. A post-fork landing needs a one-shot, non-storage URL marker (?fork=1, stripped via history.replaceState on read) to classify as entry="fork" rather than "deep-link", because onFork's navigation is a full page reload — the same hard-navigation pattern the rest of the app already uses for every route change, which destroys any in-memory alternative.

  • §H access. ACCESS_AUD is still the committed "" placeholder as of this ADR's own fold — no Access application was ever minted; docs/run-and-deploy.md's Launch plan names this as the first pre-condition to confirm before any real deploy.

  • §H access (supersedes the bullet above). The /grafana/* Access application named above was never created before launch. It was replaced with the Handsontable login broker (ADR-0007) instead of finishing it — gates/session.ts (session cookie, DEV_ADMIN bypass), gates/broker.ts (the one-time /broker/userinfo call), grafana/login.ts (login/callback/session/logout). A real production probe (curl against mcp-auth-proxy-j0tb.onrender.com/broker/login, 2026-09-24) confirmed 302 to Google for return_to=https://demos.handsontable.com/grafana/_o11y/callback?n=…, so the callback path is allowed today (see docs/run-and-deploy.md's step 5 for the exact command; a separate local round trip against a stubbed broker proves the Worker's own code, not the real broker's live configuration, and should not be read as a second production probe). This widens DEV-3088's blast radius, it does not just inherit it: the broker's return_to allowlist is host-suffix-only, so it also admits anonymous Tier-2 preview hosts under *.demos.handsontable.com, letting anyone harvest a team member's 1h broker token. Before this change, a stolen token could not reach Grafana at all (ACCESS_AUD was "", so Access refused everything); now, it can be exchanged for a Grafana session. This narrows the exposure: gates/session.ts#computeSessionTtlSeconds caps the session at min(now + 12h, brokerTokenExp) (falling back to 1h when the token carries no readable exp) instead of a flat 12h, so a stolen token buys close to its own remaining lifetime, not up to 11 extra hours — narrows, does not close. DEV-3088 itself remains open and is filed and tracked separately from this ADR.

  • §I local development. wrangler dev's local Container reaches compose.yml's standalone minio/clickhouse services (started without the box service) via Docker's own host.docker.internal, since the two are never on the same Docker network.

Consequences

  • ADR-0040 decisions A (hour dimension) and B (usage_hourly) are not built; awake-seconds per hour and sampled peak concurrency become Analytics Engine points (C.2, C.3). C.1 stands as written: at_capacity is a usage_daily counter, and is also emitted as a session.start outcome. D (privacy) stands.
  • ADR-0022 gains a subordinate o11y ceiling and per-script billing rows; recordContainerUsage takes a SKU.
  • ADR-0038's WAF exception grows by one path, /telemetry/*.
  • ADR-0007: no longer deviated from — /grafana/* gates through the same Handsontable login broker as every other internal surface (§H).
  • ADR-0020: more route patterns on the main hostname, still in deploy commands.
  • Sentry keeps uncaught errors (as §E.1 defines them) and spend alerts; handled diagnostics moved and lost grouping; the per-event environment re-homing was removed with the §E.3 switch, so no new events reach the demo-runtime environment.
  • Source maps are no longer deleted inside vite build; CI uploads them to Sentry and to R2, then deletes them before deploy.
  • We own: the Faro-to-OTLP converter, the scrubber, the symbolicator, the drain and its ledger, and the alert evaluator. That is more code than revision 2, in exchange for synchronous acknowledgements, event-time timestamps, and one path for every record.
  • New operational surface: one image (Loki + Grafana), three EU buckets, two Durable Object classes, provisioning in git, a fixture set, one Slack webhook, one O11Y_SESSION_SECRET (no Access application; /grafana/* gates through the existing login broker instead).
  • Accepted limits: no traces in Grafana (dropped, §C.4; trace.id on api.request is the browser-to-worker correlation); no alert rules or durable UI state in Grafana; the first visit after a sleep waits behind the waking page; an unclean stop costs a replay and duplicate storage until lifecycle; embeds have no docs page attribution; handled errors have no issue grouping; the non-EU items listed in §H.
  • Cost: ≈ $5–8/month target, $10 exit ceiling, reported separately, capped separately, summed under the same product ceiling. Measured (real platform, §A/§L.7): $0.21/month at 1× traffic, $0.33/month at 10× — both far under target.
  • Volume: Analytics Engine and the raw Workers Logs pool both pass exit criterion 8 at 10× with real margin; the exported-logs allotment does not, measured (§D) — docs/run-and-deploy.md's Launch plan carries the pre-launch action (a real Tier-2 stdout measurement to confirm or refine the projection, and the head_sampling_rate fallback if it holds).