Replace StreamChunk with explicit token/done/error events

[nosyndicate]

Why

In decode_loop, we had two different terminal shapes:

• EOS termination appended a separate, token-less chunk after the real tokens.
• Reaching max_new_tokens marked the last real token as terminal — and that token's decoded text could legitimately be "".

v1 relayed both through the same StreamChunk construction, so for any stream past the first token the two cases arrived at the client as byte-identical JSON. Nothing on the wire discriminated them, and a chunk-count heuristic only relocates the error: counting the token-less terminal chunk fixes max-tokens termination but breaks EOS-at-the-last-permitted-step. The bench client's rule (skip a token-less terminal chunk) was the better of two wrong defaults, but still undercounted in that narrow case.

Surfacing v1's own tally was not a fix either: v1 counted non-empty token strings, dropping every empty-decoding token, so it would have handed the client a differently-wrong number wearing the authoritative output_tokens_source: "server" label — worse than the honest client fallback. The discrimination had to originate in decode_loop, which is what this PR does: it removes the ambiguous terminal shape from the protocol entirely.

What changed

The SSE streaming endpoints now emit an explicit event protocol instead of overloaded chunks: every payload carries a type discriminator, each sampled non-EOS token is delivered immediately as its own event, and a separate terminal done event carries the authoritative token count, finish reason, and timing metrics. This removes the terminal-chunk heuristics the client previously needed to distinguish the v1 stream from v2+.

• Replace StreamChunk with StreamTokenEvent / StreamDoneEvent / StreamErrorEvent: discriminated union on type, extra="forbid", and done events require finish_reason, prompt_tokens, output_tokens, and non-optional timing metrics
• Stop holding the last token back in v2/v3/v4 streams; every token is emitted immediately and done is sent as a separate terminal event
• Bring the v1 stream onto the same protocol; generation failures surface as an error event
• decode_loop yields structured GenerationStep records: EOS produces a completion step with no token event, and a token whose decoded text is empty still carries its index and is counted
• generate_text reports the sampled token count instead of re-tokenizing the output text, which undercounted when the decoded text did not retokenize cleanly
• TokenEvent drops is_first/is_last; DoneEvent gains finish_reason; FinishReason moves to server/model/types.py as a str enum
• Bench client now validates the protocol: contiguous zero-based token indices, done count must match observed token events, no events after the terminal — violations are recorded as protocol_error

[Copilot]

Pull request overview

This PR standardizes the server-sent events (SSE) streaming protocol across v1 and v2+ endpoints by replacing the overloaded “chunk” payload with an explicit discriminated event protocol (type: token|done|error). It also refactors model/executor internals to preserve token identity (including empty decoded strings) and to report authoritative token counts without re-tokenizing output text.

Changes:

• Introduce explicit stream event schemas (StreamTokenEvent, StreamDoneEvent, StreamErrorEvent) and update v1 + v2+ streaming endpoints to emit per-token events immediately followed by a separate terminal done (or error) event.
• Refactor decoding/generation to yield structured GenerationStep records and to use sampled token counts (including empty decoded tokens) rather than re-tokenizing output strings.
• Update bench client + tests to validate and exercise the new protocol (token index contiguity, done counts, terminal semantics).

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/scripts/bench/test_runners.py	Updates bench-stream tests to use explicit token/done/error events and adds protocol-validation expectations.
tests/model/test_hf_runner_smoke.py	Adjusts smoke test assertions to the new GenerationStep-based streaming output.
tests/model/test_hf_runner_behavior.py	Adds coverage for empty decoded tokens and for sampled-count reporting; updates EOS behavior assertions.
tests/executor/test_sinks.py	Updates TokenEvent construction after dropping is_first/is_last.
tests/executor/test_schedule_engine.py	Updates scheduler-engine expectations for explicit token events + separate done event.
tests/executor/test_events.py	Updates event emission tests for new EOS semantics (no EOS token event) and finish_reason on DoneEvent.
tests/api/test_routes.py	Adds schema-level tests for the explicit SSE event models; updates stream expectations and FinishReason usage.
tests/api/test_collector.py	Updates DoneEvent construction to require finish_reason; updates TokenEvent construction.
server/model/types.py	Introduces FinishReason as a string Enum in model types.
server/model/hf_runner.py	Adds GenerationStep and refactors decode loop + generation APIs to yield structured steps and sampled token counts.
server/executor/types.py	Removes FinishReason from executor types; drops TokenEvent.is_first/is_last; requires DoneEvent.finish_reason.
server/executor/events.py	Updates event emission: EOS produces no TokenEvent; terminal steps emit DoneEvent with required finish_reason.
server/api/v1.py	Updates v1 streaming endpoint to emit explicit token/done/error SSE events.
server/api/schema.py	Replaces StreamChunk with explicit Stream*Event models and a discriminated union.
server/api/routes.py	Updates v2+ streaming generator to emit explicit token/done/error SSE events immediately.
scripts/bench/runners.py	Updates bench stream parsing/accumulation to enforce explicit protocol and report protocol errors.
scripts/bench/models.py	Updates bench result docs to reflect the new terminal done event source for server metrics.
scripts/bench/cli.py	Updates comment wording from “final chunks” to “done events”.
benchmarks/README.md	Updates protocol documentation to describe explicit token/done events and client-side validation rules.

Suppressed comments (1)

scripts/bench/runners.py:176

• The PR description says done events have non-optional timing metrics, but the accumulator currently accepts missing/null total_ms, queue_wait_ms, execution_ms, and tokens_per_s (they become None via _float_or_none). That makes protocol regressions harder to detect and can also lead to downstream calculations mixing clocks.

            self.server_ttft_ms = _float_or_none(chunk.get("ttft_ms"))
            self.server_total_ms = _float_or_none(chunk.get("total_ms"))
            self.server_queue_wait_ms = _float_or_none(chunk.get("queue_wait_ms"))
            self.server_execution_ms = _float_or_none(chunk.get("execution_ms"))
            self.server_tokens_per_s = _float_or_none(chunk.get("tokens_per_s"))

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.