perf(streaming): incremental committed-prefix rendering (fixes #21)

[jonathanKingston]

Fixes the O(n²) committed-prefix re-rendering described in #21. Plain-prose DOM streaming now grows ~2×/doubling (down from ~4×) — ~12× faster at 100 paragraphs — while staying byte-identical to a fresh whole-string render.

Layer 1 — tokenize once per update (5166457)

A single update() / renderStreamingMarkdown() call re-tokenized the streamed string ~5–8×. This threads the tokens computed during the split back through the helpers:

• splitForStreaming returns tokenizeBlocks(content) via the new StreamingSplitWithTokens; the boundary logic moves to an internal splitForStreamingCore.
• getIncompleteFenceSource / getIncompleteTableSource / completeEndsInOpenTable / pendingLineBelongsInTable accept an optional pre-tokenized array.
• renderMarkdown accepts options.tokens.

Instrumented result: ~4.5–6.0 → ~1.5–2.0 tokenizations per update (now ~1 amortized after the review fixes' per-commit cache). Stateless refactor, no behavioral change.

Layer 2 — freeze the settled prefix, re-render only the tail (e566e60)

stream-complete is split into a frozen region (top-level groups that can never change again; rendered/sanitized once, permanent node identity) and a tail group (the last still-mergeable group; the only thing re-rendered per commit, morphed so a growing table/list keeps node identity). FrozenTailRenderer.update is constant-time per commit regardless of prefix length. Every uncertainty falls back to today's exact full-morph behaviour.

Scaling fixes (2610f02)

1. Tail-scope the pending-sync queries — pending elements always live at the streaming tail, so the per-frame whole-subtree querySelectors (~700µs each in jsdom) become lastElementChild-scoped (~8µs); pending-only frames are O(1).
2. Keep the frozen boundary monotonic without a fallback storm — complete is not append-only (inline-hold retreat); never retreat frozenEnd, and fall back only when a later chunk extends an earlier block over the boundary (tokenStraddles).

Adversarial review + fixes (e8e287b)

A multi-agent review (line-by-line scan, removed-behavior audit, cross-file trace, plus differential fuzzes vs main) found 4 confirmed regressions, all reproduced, all fixed with regression tests:

1. Indented pending list item after a paragraph — the tail-scoped findOpenListItemHost could return the pending <li> as its own nesting host, detaching the pending subtree and accumulating empty <ul>s. Now skips the pending item and reuses the trailing wrapper list.
2. Non-append-only rewrites kept stale frozen DOM — # Alpha → # Bravo showed Alpha forever (every guard passed). New frozenSource byte-prefix guard (complete.startsWith(frozenSource), a memcmp) falls back on mismatch.
3. Unclosed benign raw inline tags broke byte-parity — whole-string sanitize re-opens an unclosed <b> into later blocks (formatting-element reconstruction); per-fragment sanitize can't. A delta with unbalanced benign tags is never frozen; the stream degrades to full-morph (correct output).
4. Node identity destroyed at each block's freeze frame — the delta was re-parsed instead of adopting the byte-identical existing tail nodes (selection/animation/media reset once per block). Delta+tail are now reconciled in one morphInnerHtmlFrom; insertFrozenDelta deleted.

Plus: serializeLinkRefs no longer embeds literal NUL bytes (the file was git-binary — its diff is now reviewable); pending-only frames no longer re-tokenize complete or re-scan for | (per-commit instance cache, reusing split.blocks when everything commits); delta/tail token slices are binary-searched instead of two O(#tokens) filters.

Verification

• 445 tests + CommonMark conformance green.
• Exhaustive convergence fuzz (STREAMING_FUZZ_ALL=1, every prefix of every baseline example) plus two independent review-time differential fuzzes vs main (~1,400 randomized streams).
• Byte-parity at every fully-committed frame of a full streaming history; deterministic node-identity guards (pre→post freeze, and across 30 later commits).
• Targeted tests for every settling hazard: setext retro-conversion, loose-list flip, table-row append, blockquote blank-run continuation, indented raw-HTML, trailing-open re-tokenization, lazy-continuation absorption, in-place rewrite, unclosed raw inline HTML, the forward-reference fallback, seam/sweep edges, and pending-kind transitions.
• bench-streaming.mts scaling section + regression guard (mean growth < 3×/doubling; currently ~2.2×).

Known bounds (documented in the module header)

• Limitation K: the per-update tokenizeBlocks and per-commit parseLinkReferenceDefinitions string scans stay O(prefix) (measured: 4% of wall-clock at 9 KB, 17% at 37 KB, 32% at 74 KB), so the asymptote is still Θ(n²); true linearity needs an incremental tokenizer + link-ref scan.
• Long open lists: a trailing group that never settles (one long bullet list) keeps the whole group in the tail, degrading toward old per-commit cost for list-shaped output. Tail-bounding is follow-up work.
• Layer 3 (folding lastComplete into frozenEnd) intentionally skipped.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LnAFvAgDY3RVYT6zJqN1fy