🐛 fix(contribute): make a dropped SSE event observable — seq on every frame, a gap frame when events are lost

[Danathar]

Summary

A client of /api/contribute/events can now tell "nothing happened" from
"I missed events".

Dropping an event for a subscriber whose channel is full stays exactly as it was
— a slow observer must never back-pressure the hub's event path, which is worker
assignment. What changes is that the drop is no longer invisible:

• every frame carries seq, a monotonic stream position, and
• a connection that had events discarded gets a gap frame naming how many
  went.

The connection stays open and reconnect-with-replay is still the recovery path.
What was missing was any way for the client to know it should take it.

Closes #6218.

Why this shape

@castrojo offered three options and preferred (3), close the subscriber.
This implements (1) + (2) instead, and I want to be explicit about the
trade-off rather than quietly picking a different one.

Closing turns a cheap discard into a reconnect, and a reconnect here is not
cheap: the hello frame runs a full admission sweep plus a 150-item queue
snapshot. A flapping slow client would trade silent data loss for a sweep storm
on the hub, and the existing "a momentary drop is self-healing" comment would
stop being true. A signal leaves the decision with the client — the one that
knows whether it can afford to re-sync — and a client that wants option (3)'s
semantics can implement them in three lines on top of the gap frame.

What a client sees

// on connect — the position you joined at
{"type":"hello","replay":[...],"queue":[...],"seq":41}
// normal flow — contiguous
{"type":"activity","activity":{...},"seq":42}
// your channel filled; three events went
{"type":"gap","dropped":3,"seq":46}
{"type":"activity","activity":{...},"seq":47}

seq is assigned once at fan-out, so the same event carries the same number
for every subscriber — a per-subscriber counter would be useless for comparing
two clients or reasoning about the stream. hello.seq is captured under the
same lock as the registration, so the position and the set of events the
subscriber will receive cannot disagree.

Both fields are omitempty, existing clients ignore them, and a stream that has
broadcast nothing serialises exactly as before.

Mechanics worth review

• The counter is atomic, not registry-locked. The writer must never take the
  registry lock — the entire point of the non-blocking send is that a slow
  client cannot reach the hub's event path.
• Swap-to-zero, so a gap is reported exactly once and a drop landing between
  the read and the write is carried into the next frame rather than lost.
• The check runs on both loop branches, and they are not equal. In practice
  the event branch fires: a drop implies a full channel, so ~32 queued events
  sit behind it and the check runs before each pop. The heartbeat branch is
  the guarantee — it makes "reported" unconditional rather than contingent on
  another event ever arriving, which is precisely the quiet-stream case the
  report describes. The code says this plainly rather than implying both are
  equally load-bearing.
• Ordering is stated honestly in the source. The frame means "you are
  missing events", not "the ones after this are what you missed" — the discarded
  event was newer than the queued ones, so it fires while the client is still
  draining good ones. Telling a client early that it is behind beats telling it
  late, and it is exactly why seq matters: the numbers locate the
  discontinuity, the frame is the prompt to go looking.

Testing

Six cases: the numbering (monotonic, and identical across subscribers), the
recorded start position, the recorded drop, the end-to-end gap frame over a
still-open connection, the idle-stream heartbeat path, and a healthy stream
gaining neither a gap nor a discontinuity.

Mutation-checked — removing the drop counter, the numbering, the start
position, or either gap check turns the suite red:

Mutation	Caught by
stop recording drops (the pre-fix silent discard)	2 tests
stop numbering events	3 tests
stop recording the subscriber's start position	2 tests
drop the gap check from the heartbeat branch	idle-stream test
drop the gap check from the event branch	end-to-end gap test

That table cost me a rewrite, and it is the reason I trust it. My first
idle-stream test drove broadcast to create the drop — which fills the channel,
so the writer reported the gap on the event path while the test's name claimed
it covered the heartbeat. Removing the heartbeat check left it green. The
surviving mutant is what surfaced that; the test now marks the pending drop
directly on the subscriber, with the reasoning recorded in the test so the next
person does not "simplify" it back.

sseHeartbeatInterval becomes a var solely so that branch is reachable
without a 25-second wait — the same seam CACertPath uses in pkg/proxy;
production never reassigns it.

Also run: full pkg/dashboard package green; -race over the SSE tests with
-count=2 clean (this adds concurrent state, so that felt mandatory);
golangci-lint 0 issues for the package. No new routes, so the OpenAPI
route-parity guard is untouched — I updated the existing
/api/contribute/events description and response text to document the frame
shape and the gap contract.

Client

The /contribute page — which already polled /fleet as a hedge, one of the
two workarounds the issue names — now handles type:"gap" by re-reading the
reliable endpoints immediately instead of waiting out its 6s timer. It is a
small win there; the clients this actually rescues are the headless ones like
projectbluefin/review that trusted the stream and had no way to learn they
were behind.

@castrojo — you offered to test against your downstream monitor, and the two
fields are exactly what that needs: track seq for precise detection, or just
watch for type:"gap" and re-sync. Happy to adjust the shape if it does not fit.

— hive: backend=claude model=claude-opus-5

[kubestellar-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign clubanderson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment