[design] event_{seq}: server-assigned per-run sequence as canonical log order, with fenced appends

[pranaygp]

Summary

Replace the event log's ordering contract — today a client/ingest-minted ULID that serves as sort key, pagination cursor, and identity all at once — with a server-assigned, dense, per-run sequence number (seq) assigned atomically at commit, plus fenced (compare-and-swap) appends for replay-derived writes. One primitive replaces five mechanisms, closes an entire corruption class by construction, and makes the remaining classes loud instead of silent.

This is the synthesis of the 2026-07-29 CORRUPTED_EVENT_LOG investigation (repro on all three worlds; commit-order forensics on postgres; storm measurement matrix) plus comparative source studies of Temporal and DBOS.

Problem

One identifier is doing three jobs, and it's minted at the wrong time:

• Event IDs (ULIDs) are minted before commit on every path, in every world (world-vercel server ingest, world-postgres storage.ts, world-local createImpl). No path allocates under a lock except one hand-built case (world-postgres step_started, whose comment names the missing invariant: "Without a sequence, this is the local ordering guarantee we can provide").
• The log is sorted and cursored by that ULID (ORDER BY id / WHERE id > cursor; Dynamo sk range scan; local FS sort).
• Under concurrency, ULID order ≠ commit order. Measured on postgres with pg_xact_commit_timestamp: 22% of events in corrupted runs occupy a different position by commit order than by ULID order; mint→commit invisibility windows run p50 31ms / max 2.0s for step_created; hook_received (a tiny out-of-band write, p50 0.4ms) systematically overtakes earlier-minted step events (334 inversions avg 148ms vs 6 reverse inversions ≤2.6ms).

Consequences:

1. A reader's snapshot is not guaranteed to be a prefix of the final log — an event can commit behind a cursor the live run already passed. Measured rare (0.8% of 12,403 instrumented reads) but structurally permitted, and the class our positional correlation IDs cannot survive.
2. The existing guards are blind to it by construction. The stateUpdatedAt watermark detects "something newer than my snapshot exists"; this bug is "something older than my view committed after I read." In the hook interleaving the stale invocation's watermark equals the marker timestamp — the comparison is t < t, guaranteed to pass.
3. We've accumulated compensating machinery: a Redis per-run event-count index (24h TTL, 16-id window, region-local), the outside-event marker, stateUpdatedAt preconditions, proven-delta completeness arithmetic, and SDK delivery-order barriers. Each patches a symptom of the same missing invariant.

Scope honesty (important): the forensics showed the dominant corruption cause is engine wake-order nondeterminism between concurrent replaying invocations, amplified by positional correlation IDs — not the ordering contract (storm data: sequential replays alone moved 698→650 corrupted of 1400; call-site correlation IDs moved 698→106). event_{seq} fixes the contract class, deletes the compensating machinery, and provides the fencing primitive — but it must ship alongside the call-site correlation-ID work (#3179) and the engine determinism fixes (#3196) to reach zero. See "What this does NOT fix" below.

Design

The invariant

Log position is assigned at commit, by the store, densely, per run. Nothing that has a position was ever invisible to a reader of a lower position.

This is Temporal's core invariant (their history_node separates node_id = dense server-assigned position from txn_id = commit identity; a late commit can rewrite a position, never insert below a cursor). We collapse today's ULID back to what it should be — an identity/idempotency key — and add seq as the position.

Two write classes

Class	Events	Who assigns seq	Fencing
Unfenced (facts)	hook_received, step_completed, step_failed, stream chunks…	Server, at arrival/commit (atomic increment + insert)	None — facts are accepted as they come; idempotent by correlationId
Fenced (decisions)	step_created, wait_created, hook_created, attr_set, run_completed/run_failed…	Client submits expectedSeq (= its loaded log length); server CAS-appends	Reject if the log grew past expectedSeq — the replay's view is stale, it must re-derive

The distinction is not the event type per se — it's whether the write encodes a decision derived from a log prefix (stepName/args computed from wake order) versus a fact about the world. Fenced rejection is exactly the invalidation we want: "a hook arrived while you were thinking; re-run."

Rules

1. The fence rides the append. Never check-then-write (that's the count guard's TOCTOU). The CAS and the insert are one atomic operation: TransactWriteItems(counter CAS + event puts) on Dynamo; counter UPDATE … RETURNING inside the insert transaction on postgres; in-process on local.
2. Batch fenced appends per drain/suspension window. One conditional multi-event append, not one CAS per event (Temporal batches all events of a workflow task into one append). This also fixes the partial-suspension-write hazard found in the count-guard review — a suspension's hook_created + step_created + wait_created commit atomically or not at all, so a rejection can never strand an orphan entity.
3. The loser adopts or re-derives — never re-posts. On CAS failure: discard in-memory state, re-load, re-run the replay in-process (Temporal: failed CAS → Clear(), OperationPossiblySucceeded=false; DBOS: the conflict loser fetches the winner's output and returns normally). The current withPreconditionRetry shape — re-invoking the same closure that captured stale-derived names — is the known landmine: after reload the count matches and the corrupt write gets accepted. The retry must re-derive.
4. Bounded restarts with cross-invocation accounting. In-process restart cap, then re-enqueue with a restart counter carried in the queue message (the replay-divergence path already does exactly this) and non-zero backoff. reinvoke(0) with per-message delivery counts is an unbounded hot loop.
5. Density is asserted, loudly. Readers assert seq[i+1] == seq[i] + 1 on load and fail fast on a gap (Temporal's Rust SDK: eid != last_processed_event + 1 → fatal "History is out of order"). Today's cursor skip is silent and surfaces three replays later as corruption; under seq it's an immediate, diagnosable error.
6. seq becomes the read order and the pagination cursor. Multi-page loads are gap-checked; an incremental read from seq=N is provably complete.
7. Fast paths may skip persistence but may never mint positions from an uncommitted view (Temporal's speculative-workflow-task rule). The lazy step-start, async step_completed, and inline event-delta optimizations must be audited against this rule.

What gets deleted

Today	Under event_{seq}
Redis per-run event-count index (TTL, 16-id window, region-locality, verifier, metrics)	Deleted — the count is seq, atomic with the insert, no side index
stateUpdatedAt watermark + outside-event marker	Deleted — the fence is exact, not time-granular; replay-vs-replay writers are finally visible
Proven-delta completeness arithmetic	Trivial — delta = everything above your seq, complete by density
Cursor-skip / non-prefix snapshots / hook-overtaking	Impossible by construction
SDK delivery-order barriers (pendingDeliveryBarriers)	Eventually deletable — live delivery order ≡ commit order ≡ replay order, so there is nothing to reorder (see Phase D)

Inspirations

From Temporal (server: service/history/**, common/persistence/**; sdk-core)

• Position ≠ commit identity: history_node (tree_id, branch_id, node_id, txn_id); reads range-scan node_id; txn_id only picks winners among duplicate positions.
• The counter is not a DB sequence: in-memory nextEventID++ made durable by a conditional write on the run record (IF db_record_version = ?). One row per run, updated in the same transaction as the append.
• Batching: one durable append per workflow task containing many events — the amortization that makes per-append CAS affordable.
• Buffered stimuli: signals arriving mid-task are position-less (BufferedEventID = -123) until flushed at a quiescent boundary — and the flush canonicalizes order (reorderBuffer: signals before activity completions regardless of arrival). Only possible because buffered events have no position yet.
• Discard-on-stale, twice: stale task token → the worker's entire command set is dropped; UNHANDLED_COMMAND refuses run-close while buffered events exist, forcing a fresh task.
• Contention escape valve: lease ID ranges (~1M per shard update) — only if per-append CAS measures hot; don't pre-build.
• Explicitly rejected: a literal single-writer lease. In Temporal's own stack the per-run mutex is a throughput optimization; the CAS is the correctness mechanism. A lease in a serverless fleet buys lease-expiry, fencing tokens, and split-brain — the costs without the guarantee.

From DBOS (dbos-transact-ts)

• No total order at all — the deepest contrast. Durable state is an independently-keyed set (operation_outputs PK (workflow_uuid, function_id)); external stimuli are converted into positionally-keyed decisions at first observation (recv = select oldest message + mark consumed + checkpoint, one transaction). Arrival order is never recorded, so there's no interleaving to reproduce. We can't retrofit this (our contract, observability, and replay model are built on an ordered log), but it explains why we corrupt: we record stimulus arrival order as a durable fact that must reproduce identically twice.
• The fence rides the durability write: INSERT … ON CONFLICT DO UPDATE … RETURNING + compare — zero extra round trips. (Copy the shape, not the bug: they discriminate insert-vs-conflict by comparing a millisecond timestamp; use a structural discriminator like RETURNING (xmax = 0).)
• Adopt-the-winner: the conflict loser fetches the winner's result and returns normally. Converts a category of concurrent-replay errors into non-events.
• Concurrent replay is made rare by policy, not mechanism: first-writer-wins ownership at start + recovery strictly through an atomic queue dequeue (FOR UPDATE SKIP LOCKED). No leases, no heartbeats. Their model keys on stable process identity, which serverless lacks — but the policy insight transfers: fencing detects racing writers; admission control reduces how often they race. We currently run up to 20 backends replaying one run within 140ms as normal operation.
• Escape hatch: DBOS.patch/deprecatePatch (≅ Temporal GetVersion) for intentional positional shifts when users edit workflow code. Both systems shipped one; we should plan for it.
• Anti-pattern to avoid: their replay is O(N) database round trips (one checkpoint read per step, no prefetch). Our snapshot-load replay is strictly better — keep it.
• Sobering mirror: DBOS's functionID is a bare wake-order positional counter and child workflow IDs are literally parentID + '-' + funcId — the same amplification scheme we have. Their mitigation is a comment ("reserve the function ID synchronously, before any await") at seven call sites plus a fatal DBOSUnexpectedStepError. They survive because concurrent divergent replay is exceptional in their environment; it is routine in ours.

Gotchas (hard-won; each of these cost us)

1. bigserial/sequences reproduce the exact bug. Postgres sequences are non-transactional: writer A takes 5, B takes 6, B commits first, a cursor reader skips 5 forever — a server-assigned number with the same defect. The counter must be a row updated in the same transaction as the insert.
2. Handler-evaluated fences leave TOCTOU windows. The merged count guard evaluates in the handler and writes after (S3 payload drain + Dynamo transaction in between). The CAS must be a condition on the write.
3. Multi-phase suspension writes strand orphans. Sequentially-awaited commit phases sharing one snapshot mean a mid-sequence rejection leaves earlier phases durable (orphaned hook_created → token conflict on the corrected replay, no compensating delete). Atomic batch per suspension.
4. Healthy writers get fenced. Commit inversions occur in 84–98% of clean runs — a predicate keyed on "someone appended after your read" fires constantly on runs that were never going to corrupt. The append-tail-fence experiment measured worse than baseline (805/1400 vs 698/1400 corrupted; hook-storm 177→252) for exactly this reason plus a mispriced retry convoy (20 concurrent writers vs an in-process retry cap of 2). Hence: fence only decision writes; accept facts unfenced; pair with admission control; price the convoy at production concurrency.
5. Rejection handling must re-derive, not re-post (the withPreconditionRetry landmine above). And the restart budget must be cross-invocation (queue-message counter + backoff), or a persistent mismatch becomes an infinite zero-delay loop.
6. Side indexes decay: the Redis count index's TTL (runs sleeping >24h unprotected — a headline use case), 16-id window (indeterminate on busy runs), and physical-region locality (cross-region racing pairs invisible) are all consequences of keeping the correctness artifact next to the data instead of in it. seq lives in the log row; none of these failure modes exist.
7. Read order on Dynamo is the expensive part. The base-table sk (eventId) is immutable, and a [runId, seq] GSI is eventually consistent — which would forfeit the ConsistentRead the events query was specifically built to have. Options, in preference order: (a) store seq as an attribute, keep the ULID sk, sort by seq after read (runs' pages are small; server-side sort preserves ConsistentRead); (b) real sk migration with dual-write window; (c) GSI only if measurement shows (a) untenable. Postgres/local: trivial (ORDER BY seq).
8. Per-run stickiness: a run is entirely pre-seq or post-seq (stamp the scheme on the run at creation, e.g. via executionContext). Mixing ordering schemes inside one run voids the guarantee. Wire format is additive: readers prefer seq, fall back to eventId ordering for old runs — fits the server-first compat model.
9. Multi-region: the counter needs a single home per run. Runs are already region-pinned; enforce it for the counter path explicitly.
10. Escape hatch: plan the patch/GetVersion analog for intentional decision-shape changes across deploys, or users will hit fenced rejections they can't resolve.

What this does NOT fix (and what must ship alongside)

Wake-order nondeterminism within a single replay of a single log. Two invocations replaying the same log can wake concurrent branches in different orders and allocate correlation IDs differently — no server-assigned number touches this, because the problem is not what the server assigns but the order in which the client asks. Temporal solves it only by owning the scheduler; DBOS's answer is a comment and a fatal error. Evidence this is our dominant class: the postgres forensics caught two invocations swapping two adjacent positions live (stored recoverStep/finalizeStep vs the mirror, ms apart, no missing events), and the engine corrupts on a synthetic perfectly-ordered in-process log at every consumer hop count.

Companion tracks, already in flight:

• [measurement] Call-site-addressed correlation ids #3179 — call-site correlation IDs: removes the amplifier (one ordinal of skew stops renaming every downstream entity). Measured −85% (698→106 corrupted) in the storm matrix; with the guard on top, the residual is genuine control-flow divergence that fencing correctly converts to retry.
• fix(core): close the replay-engine determinism gaps (barrier retirement, quiescence, buffered-hook claim ordering) #3196 — engine determinism fixes: barrier retirement tied to the barrier's own delivery ("delivered" = continuation-quiesced), bounded idle detection, buffered-hook claim-window closure. Under full event_{seq} + strict in-order delivery these barriers become deletable (Phase D), but the fixes are needed now and de-risk the transition.
• Admission control (new, orthogonal): best-effort single-runner per run (enforce what ownerMessageId only hints at today, or queue-level same-run dedupe). Reduces the CAS conflict rate; the fence remains the correctness backstop. Fence for correctness, admission for rate.
• One enforceable discipline neither comparison system has: "reserve IDs synchronously before the first await." DBOS states it in comments; we own the SDK and the compiler plugin, so we can enforce it at build or runtime.

Performance notes

• The fence is free at the round-trip level: it rides the write durability already requires (DBOS's proof: their fence is the checkpoint INSERT). Allocation is an in-memory/in-transaction increment.
• Batching amortizes: one CAS per drain window covering many events (Temporal: one append per workflow task). Dynamo TransactWriteItems costs ~2× WCU on the items in the transaction — budget it; it replaces the Redis round-trips and the verifier reads.
• Contention is per-run only (counter on the run item/row; runs are partition-local). Conflict rate = concurrent-writer rate, which is precisely the signal we want surfaced — and which admission control cuts. Escape valve if a fan-out-heavy run measures hot: range-lease blocks of seq (Temporal's RangeSizeBits pattern). Don't pre-build it.
• Keep snapshot-load replay (avoid DBOS's O(N)-round-trip replay). seq makes incremental deltas provably complete, which strengthens the inline-delta optimization rather than weakening it.
• The one real tension: commit-before-deliver on the inline-step hot path (delivery order can't equal commit order if we deliver optimistically before committing). Options: reserve seq synchronously + commit async; or keep inline delivery optimistic and fence only the subsequent decision writes. Needs a benchmark-backed decision — flagging as the main open design question.
• Ambient wisdom from both systems: polling is correctness, push (LISTEN/NOTIFY, streams) is latency; and when the store is down, block rather than guess (DBOS dbRetry: "trading off availability for correctness").

Rollout plan

• Phase A — additive seq (no behavior change): server assigns seq on every event insert in all worlds (Dynamo attribute + run-item counter via TransactWriteItems; postgres column + same-transaction counter + UNIQUE(run_id, seq); local in-process). Emit density/inversion telemetry. Readers ignore it.
• Phase B — seq becomes read order + cursor for new runs (per-run stamp). SDK asserts density, fails fast on gaps. Old runs keep eventId ordering (fallback path).
• Phase C — fenced batched appends for decision events + adopt/re-derive loser path + cross-invocation restart accounting. Delete the Redis count index, count guard, watermark/marker machinery, proven-delta arithmetic.
• Phase D — delivery simplification: live delivery order = commit order (deliver events as committed; replay in seq order) → retire pendingDeliveryBarriers. Gated on fix(core): close the replay-engine determinism gaps (barrier retirement, quiescence, buffered-hook claim ordering) #3196's fixes landing first (they're the safety net during the transition) and on resolving the inline-hot-path question.
• Validation gate at every phase: the event-log-race-repro storm (label on any PR, or workflow_dispatch) — pass = 0 CORRUPTED_EVENT_LOG with bounded retries, and for fenced phases, non-zero rejections whose verification verdicts are true positives (a silent guard passing 1400 runs has told us nothing). Baseline for comparison: main = 698/1400 corrupted (step-storm 583/600, hook-storm 115/600, control 0/200).

Relationship to in-flight work

PR	Relationship
#3145 + workflow-server count guard	Interim detection for the missing-event class; superseded by Phase C (the count is seq). Worth shipping now with its two hazard fixes — it's the bridge.
workflow-server tail fence (#670)	Superseded by this design; its measurement (805/1400, worse) is the evidence for gotcha #4. Hold.
#3179 call-site correlation IDs	Complementary and ships first — it's the measured −85% lever and stays as defense-in-depth after seq.
#3196 engine determinism tests+fixes	Complementary; prerequisite for Phase D.
#3189	The storm harness / regression-gate vehicle.

---

Design synthesized from the 2026-07-29 corruption investigation: three-world reproduction, postgres commit-order forensics (108 corrupted runs, 12,403 instrumented reads), the 7-run storm measurement matrix, and read-only source studies of Temporal (temporalio/temporal, sdk-core) and DBOS (dbos-inc/dbos-transact-ts).

🤖 Generated with Claude Code

[pranaygp]

Implemented Phases A–C for world-postgres and world-local in #3269, with the storm run locally against both worlds (seq branch vs main, 24+24 attempts each, details and forensics in the PR):

condition	world	step-storm	hook-storm
main	local	24 corrupted / 0 completed	18 corrupted / 0 completed
event_{seq}	local	23 corrupted	9 corrupted
main	postgres	20 corrupted / 4 completed	5 corrupted / 11 completed
event_{seq}	postgres	2 corrupted / 22 completed	0 corrupted / 24 completed

Storage invariants held everywhere on the seq branch (postgres: 34,722 events, 0 gaps/dups/inversions; local: 8,590 events, 0 gaps/unpositioned/orphans) — every residual corruption sits on a perfectly ordered, dense log, and the forensics isolate it to the complete-view wake-order class this issue scopes to #3179 / Phase D (two current-view writers derive different ULID-draw assignments and both flush; the fence rightly admits both). Two additions for the gotchas list from the implementation: on a non-transactional store the fence must run before any side effect (a 412'd run_completed otherwise strands the terminal marker and wedges the run), and per-run append serializers must not span directory-scan reads (a stale-lock break on a healthy holder mints duplicate positions — caught by the rule-5 density assertion within minutes, working exactly as designed).