Backwards DTS at a failover switch: the always-present-step explanation is refuted, and the mechanism is unknown

[rainmanjam]

Important

Update 2026-08-13 — retitled. The rate framing is retired, and the leading
explanation is refuted.

One run on main at c3a1bc1 emitted no NEARMISS at all across nine
seams: not one backward step of any magnitude, sub-millisecond included. The
"a small step is present at every seam and a timing change merely widened it"
explanation does not hold.

The old title said 3 runs out of 10, and only on the core-review branch.
Both halves are now misleading: that branch is long merged and deleted, and
the rate was never the useful quantity — with no mechanism, more runs make a
number more precise without making it more meaningful. Deliberately NOT
pursued for that reason.

The one lead is seam 6, which hit its stop deadline with a 12-second
teardown, meaning the outgoing feed was not confirmed stopped before the
incoming one began. Measured on a passing run. See the comment below.

Important

Update 2026-08-11 — the rate below is stale, and stale in the direction
that gets this deprioritised.

This issue was filed on a single unreproduced CI failure, and the "What is
not known" section proposed running the suite ~20 times a side to settle it.
That experiment was run. Commit 79b16f8 (PR #152) reports:

a backwards decode timestamp at a failover switch: 3 runs of 12 under one
timing change, 0 of 12 under another, and the mechanism is still not
established.

So it reproduces at roughly 1 in 4 under one timing condition and not at
all under another. It is not a one-off. Read the update at the bottom before
judging severity from the original text.

What happened

acceptance-failover failed once on PR #125 with a single assertion:

FAIL  1 backwards DTS step(s) — a platform drops the connection on these
      25 passed, 1 failed

Re-running the identical commit passed. It has not reproduced since.

Observation	Result
CI, PR #124 (parent)	pass
CI, PR #125 attempt 1	FAIL — 1 backwards DTS step
CI, PR #125 attempt 2, same SHA	pass
Local run 1	pass, 26/26
Local run 2	pass
Local run 3	pass

Why this is not #94

#94 documents the known
acceptance flake as a binary stall: a suite consuming its whole 20-minute
budget, with measured medians of 185s and no duration tail. This failure ran to
completion in 4m26s, reached step 9, and failed one assertion. Different
signature. It should not be filed under #94 and it should not be assumed benign
because #94 exists.

Why it matters more than a flake normally would

The check is zero-tolerance by design, and the reason is in the script:

A feed started without -output_ts_offset republishes from zero, so a switch
hands the destination a timestamp behind the one before it, and a platform
answers a backwards jump by dropping the connection.

So a single backwards DTS step is not a test artifact — it is the failover tier
occasionally doing the exact thing failover exists to prevent, on a real switch,
in a real recording. The suite caught a genuine intermittent. Whether it happens
once in four runs or once in forty, a dropped platform connection mid-broadcast
is the outcome.

What is not known

Whether the rate changed. PR #125 refactors the selector (extracted to
selector.go), changes primaryFeedSig to include the upstream hub's port, and
coalesces status pushes on a 150 ms leading-edge window. Any of those could
perturb switch timing relative to a GOP boundary. Establishing whether the rate
moved needs N runs on main against N on the branch — roughly 5 minutes each,
so ~20 runs a side to say anything with confidence. That has not been done.

The honest position is: one occurrence, not reproduced, cause unknown, and the
possibility that it is a pre-existing intermittent has not been separated from
the possibility that this branch made it likelier.

Superseded 2026-08-11 — see the banner at the top and the update below.
The 12-runs-a-side experiment this section asked for was carried out.

Suggested next step

Run the suite ~20 times on main and ~20 times on a branch with these changes,
counting backwards-DTS occurrences on each side. If main shows the same rate it
is pre-existing and wants its own investigation into the switch seam. If only the
branch shows it, the status coalescing is the first thing to look at.

---

Update 2026-08-11 — measured, still unfixed

Recorded here because the numbers lived only in a commit body, and this issue
is what anyone triaging reads.

The rate

From 79b16f8 (PR #152): 3 runs of 12 under one timing change, 0 of 12 under
another. The mechanism is still not established. What the original text called
"one occurrence, not reproduced" is now a measured intermittent under a known
condition.

Two proposed fixes, both refuted, both for the same reason

BOTH REFUTED FIXES FOR #126 RESTED ON THE SAME UNMEASURED ASSUMPTION.

The argument in ensureFeed is about whether the incoming feed's offset is
taken before or after the teardown — and that only means anything once everyone
agrees what -output_ts_offset does with the number. Nobody had measured it.
There is no -copyts in the tree, so the input's own start time is a live
variable, and MPEG-TS does not start at zero. An offline bench against the
production relayFeedArgs was added rather than against a copy of it.

Why the old evidence could not settle anything

The suite printed one number for the whole file, which cannot distinguish
"one sub-millisecond step somewhere" from "a sub-millisecond step at EVERY
seam". Those are the leading hypothesis and its refutation: if a small backward
step is present at every switch all the time, a timing change that widens it
merely pushed an existing step over the threshold, and the offset was never the
cause.

What landed instead of a fix

A seam ledger — one Info line per handover, written where the handover
happens, recording which feeds swapped, where on the tier clock, how long the
teardown took, what the ledger predicted, and how many backward steps of any
magnitude landed in that seam's interval. Step 9 of the acceptance suite now
attributes each backward DTS step to the switch that produced it.

79b16f8 is explicit: "No fix is claimed and #126 stays open. This only makes
the next occurrence" attributable.

Next step

The next reproduction should now name its seam. Until one is captured with the
ledger active, the mechanism stays open — but this should be triaged as a
reproducible intermittent in failover, not as a single unexplained CI blip.

[rainmanjam]

Measured. It is a regression, not a flake.

flake-rate.yml, 10 runs a side, same runner image, unchanged trees:

	result
main @ ba128db	0 / 10 failed
refactor/core-review-2026-08-08 @ bfdb0fd	3 / 10 failed (runs 4, 7, 9)

All three failures are the same assertion, and it is the one this issue is about:

FAIL  1 backwards DTS step(s) — a platform drops the connection on these

With the original PR #125 CI failure that opened this issue, that is 4 failures
on the branch and 0 on main across 11 and 10 observations.

I was leaning towards "flake". The data says otherwise: this branch hands a
destination a backwards decode timestamp at a switch roughly 30% of runs, which
is the exact failure the failover tier exists to prevent.

Ruled out

The selector extraction is a pure move. startFeed, reconcileSelector,
applySourceChoice and ensureFeed are byte-identical between ba128db's
engine.go and the branch's selector.go. Only detachFeedForSilence differs,
and that difference is deliberate (six lines).

So the cause is one of the behavioural changes, not the file split.

Candidates, in order of suspicion

1. primaryFeedSig now includes the upstream hub's port (662202c) — it
   directly decides when the primary feed is torn down and rebuilt, and a feed
   respawn is exactly where a timeline discontinuity would come from.
2. Status publishing coalesced on a 150 ms leading edge (a54d4a3) — the
   suite polls status to decide when to trigger each switch, so this moves when
   switches land relative to a GOP boundary.
3. relay.deliverMu held across fanout and measure (5eac880) — adds a
   mutex to the per-datagram path on the hub the destination reads.
4. effectiveSourceKnown reading measured instead of probed (52efbff).

Next step: bisect with this workflow

Each side is ~35 minutes for 10 runs, which makes a real bisect affordable.
Suggested first point is 662202c, the midpoint and the commit carrying
candidate 1.

Do NOT merge PR #125 until this is resolved.

[rainmanjam]

Bisected. It is the status coalescing.

Ten runs a side, flake-rate.yml, on the runners where it actually happens:

commit	change	failed
ba128db	main	0/10
c4651c8	relay deliverMu	0/10
00b96d3	primaryFeedSig + hub port	0/10
b923bf6	perf + structural splits	0/10
57d079a	the deferred nine	2/10
bfdb0fd	branch tip	3/10

Then inside 57d079a, one suspect removed per variant:

variant	failed
mid-session publisher drop removed	4/10 — still fails
status coalescing reverted	0/10 — clean

Both suspects I ranked first were wrong. That is what the measurement was for.

Withdrawn, not fixed

The coalescing is reverted. publishStatus is synchronous again.

The optimisation was real — Status() costs three database queries plus a
routing.Compile per idle destination, and onState fires it per process
transition — but it is not worth three broadcasts in ten. A backwards DTS makes
a platform drop the connection, which is the failover tier failing at the one
thing it exists to do.

Why this stays open

It is not established that the coalescing CREATED the defect. The 150 ms
delay may simply have exposed a latent race in the switch path that main also
has, at a rate ten runs has never sampled. The two readings have very different
consequences:

• Created it — the revert is the whole fix, and this can close.
• Exposed it — main ships a switch that occasionally hands a platform a
  backwards timestamp, and nobody knows the rate.

Distinguishing them means running main at much higher N (50-100) and looking
for a single occurrence, and reading the switch seam for a genuine ordering
hazard rather than a timing one. Neither has been done.

The suite's own comment is the reason to bother: "a platform answers a
backwards jump by dropping the connection." If main can do that at all, the
rate matters more than the branch that made it visible.

[rainmanjam]

Confirmed on the branch tip: 0/10

Ten consecutive passes with the coalescing reverted, matching main.

The experiment commits are preserved as tags

The bisect/* branches are deleted, but two of them held commits that exist
nowhere else, and this issue's open question may need them again. They are tagged
so they survive both the branch deletion and a squash-merge of #125:

tag	what it is	measured
evidence/126-deferred	the commit that introduced it	2/10
evidence/126-nodrop	same, mid-session publisher drop removed	4/10
evidence/126-nocoalesce	same, status coalescing reverted	0/10

git fetch --tags && git checkout evidence/126-nocoalesce reproduces any of them.

Still open, and this is the part that matters

Whether the coalescing created the backwards DTS or merely exposed a
latent race in the switch path that main also has. Ten runs cannot tell 0% from
2%, and the two readings have very different consequences for the product rather
than for this branch.

Next step remains: main at N=50-100 looking for a single occurrence, plus a
read of the switch seam for a genuine ordering hazard rather than a timing one.

[rainmanjam]

Stopped at 18/18 clean, because 40 was the wrong number

main ran 18 consecutive clean before this was cancelled. Against the branch's
3/10, that settles the question it was asked to settle.

What 0 failures rules out, at each N:

true rate on main	P(0 in 18)	P(0 in 40)
30% (the branch's rate)	0.2%	0.0%
15%	5.4%	0.2%
10%	15.0%	1.5%
5%	39.7%	12.9%
3%	57.8%	29.6%

18 runs already excludes anything resembling the branch's 30%, and excludes
10% at better than 6-to-1. Going to 40 buys the ability to exclude ~7% instead
of ~13% — a real gain, but not the one that mattered.

And the question actually left open — is main 0% or 3% — is not answerable
at any N this method can afford. Separating 0% from 3% with confidence needs
low hundreds of runs, at three minutes each. Twenty-two more runs was two hours
spent narrowing a bound that was already narrow enough, while leaving the
interesting question exactly where it was.

I proposed 40 without doing that arithmetic. The number should have come from
what it could rule out, not from the workflow's maximum.

What this establishes

• The regression was real and was mine: 3/10 on the branch, 0/18 on main.
• main has no defect at anything like that rate.
• v0.5.0 is not obviously carrying a broadcast-dropping bug.

What it does not

Whether a rare latent race exists in the switch seam. More runs is the wrong
instrument for that, and always was. The right one is reading the code: the
feed teardown and -output_ts_offset handoff between the outgoing and incoming
feed, looking for an ordering hazard rather than sampling for a timing one.

That is the next step here, and it does not need a runner.

[rainmanjam]

The switch seam, read rather than sampled

There is a structural hazard here, and it explains why the rate is
timing-dependent rather than constant.

ensureFeed swaps a feed in this order (selector.go):

e.teardownFeed(cur)                                       // blocks
feed := e.startFeed(s, want, upstream, silenceSig, now)   // `now` predates the line above

and startFeed derives the new feed's timeline from that now:

offset := now.Sub(sel.startedAt).Seconds()   // becomes -output_ts_offset

now is captured before the teardown, and the teardown blocks.
teardownFeed calls proc.Stop(ctx) with stopTimeout, which is 12 seconds.

So for a teardown lasting T:

	timestamp of last/first frame
outgoing feed, last output	now + T - startedAt
incoming feed, first output	now - startedAt

The new feed's timeline begins T behind where the old one ended. The step is
backwards, and its size is exactly how long the teardown took. -output_ts_offset
pins the timeline to offset rather than to when frames actually appear, so the
encoder's own startup delay does not absorb it.

Why this is consistent with everything measured

It is not a race with a window — it is present at every switch, and whether it
crosses the check's 1 ms threshold depends on how long that particular teardown
took and on what the destination's muxer absorbs downstream. That is precisely
the shape of a defect that sits at 0/18 on one timing profile and 3/10 on
another, without either being wrong about what it saw.

It also explains why the two things I suspected first were clean: neither
primaryFeedSig nor the relay mutex touches when now is read.

The candidate fix

Capture the timestamp after the teardown, so the new feed's offset describes
when it actually starts rather than when the swap was decided. One line, and it
makes the seam continuous by construction rather than by luck.

Confidence

Hypothesis, not conclusion. The reasoning above is from reading; it has not
been demonstrated. Confirming it means instrumenting a switch to print the last
outgoing DTS against the first incoming one, and showing the gap tracks teardown
duration. That is a cheap experiment and the right next step — considerably
cheaper than the hundreds of runs sampling would have needed.

[rainmanjam]

Another occurrence, worth recording because of where it happened.

acceptance-failover failed with 1 backwards DTS step(s) on PR #133 — a pull request whose entire diff is web/src/ and one notes file. Nothing in it touches the engine, so this is a clean observation of the flake rather than a regression. It passed on re-run.

The part that matters: this is after #131 merged. #131 fixed the two timestamp faults at the switch seam (the feed's -output_ts_offset being taken before a blocking teardown, and the respawn backoff being measured from the decision rather than the start). Those were real, but they were not this — the backwards DTS survives both fixes.

So the seam work should not be treated as having addressed #126. Whatever produces the backwards step is still there, and the earlier flake-rate sampling put it around 1-in-6.