Expose the transform-stream layer; UTF-8 text fast path; O(active) end-tag dispatch by sosukesuzuki · Pull Request #3 · oven-sh/lol-html

sosukesuzuki · 2026-09-08T06:24:43Z

Three changes for Bun's HTML-to-Markdown converter (oven-sh/bun#41527), which needs every tag/text event from the tokenizer but none of the rewriter machinery.

lol_html::transform — TransformStream, TransformController, Token, TokenCaptureFlags, LocalName/LocalNameHash, SharedMemoryLimiter, SharedEncoding etc. were only exported behind integration_test. They are now public (and documented) under lol_html::transform. Additions: TransformStream::controller(), LocalNameHash::{as_u64, from_ascii (const), decode} so a controller can identify tags from the LocalName alone. Driving the tokenizer this way instead of through HtmlRewriter (a * element handler + text handler + per-element end-tag closures) roughly halved Bun's conversion time, and it sees every end tag rather than only those HtmlRewriter matches to an open element.

text_decoder — a valid UTF-8 chunk went out through the fast path but the empty remainder still went through the streaming-decoder loop, allocating a decoder per text node just to remember that the closing chunk was owed, and pushing any further chunk of the node onto the buffered path. A flag now carries that state; the decoder is created only when needed. ~5% of rewriter time on text-heavy pages.

handlers_dispatcher — do_for_each_active_and_remove scanned every registered end-tag handler per end tag. Active handlers are always a suffix, so it now stops once the active count is spent. Long lists/tables with omitted </li>/</td> plus any on_end_tag user were quadratic (100k <li><a>x</a>: ~3 s → 0.1 s).

cargo test --lib: 172 passed (including the suspension tests, which exercise the text-decoder resume path).

Follow-up commit: RAW_TEXT capture + TransformController::handle_raw_text (text as raw byte runs, no TextChunk/decoder), StartTag::for_each_raw_attribute and Attribute::name_raw/value_raw (attributes without allocation), LocalNameHash::from_ascii (const).

CI note: the workflows on this repository currently fail at "Set up job" for every branch (the org requires actions pinned to full SHAs), so CI cannot report on this PR. Locally: cargo test --lib and cargo test --features=integration_test pass, cargo clippy --all-targets (with and without integration_test) is clean for the library crate, and Bun's HTMLRewriter test suites pass against this branch.

A handler can return Err(SuspensionRequest) to park the current rewritable unit on the heap and exit write()/end() with the non-poisoning RewritingError::Suspended; HtmlRewriter::resume() continues from a StateMachineBookmark. - take_owned() on each rewritable unit deep-copies borrowed byte slices so the unit outlives the write() stack frame - Arena::shift advances a start offset instead of memmoving the tail (a suspended rewrite re-feeds its unconsumed tail on every resume) - the pending captured-text flush is hoisted from Dispatcher::handle_tag into the lexer actions so every suspension point has a uniform shape - end_mut() is the non-consuming end() for a rewriter that may suspend in a document-end handler Bun uses this so an async JS content handler can suspend the rewrite and resume it from a promise reaction instead of spinning a nested event loop inside write(). See oven-sh/bun#36733.

The upstream ci.yml triggers on push: branches: [master] only, so an additive bun.yml runs cargo test --features=integration_test on pushes/PRs to bun without touching ci.yml (keeps rebases onto upstream clean).

…epth The selector-VM open-element stack had two per-token scans that go quadratic (or worse) on pathological nesting: * Stack::pop_up_to did a full rposition() over the open stack for every end tag. A stray end tag that matches nothing still walked the whole stack, so N unclosed opens plus N stray closes was O(N^2): "<a>".repeat(100_000) (~700 KB) burned ~25s of CPU. * try_exec_hereditary_jumps_without_attrs walked every ancestor and re-executed its hereditary_jumps for every start tag. For a selector like "div div" over N nested <div>s every item carries the same address range, so the walk is O(depth) per tag and O(N^2) overall; k-arm chains ("div div div") compound to O(N^k). Both paths are reachable from HtmlRewriter with any element-handler registration, so untrusted HTML can pin a CPU core via sub-MB input. Memory stays linear so max_allowed_memory_usage does not bound it. Changes: * Stack now keeps open_name_counts: HashMap<LocalName, usize> so pop_up_to rejects a name not on the stack in O(1). When the name is present the rposition scan length equals the subsequent drain length, so total scan work over a document is bounded by total pushes. * Stack now keeps active_hereditary_jumps: Vec<(AddressRange, usize)>, the distinct hereditary-jump ranges currently contributed by any open item, each paired with the shallowest depth that introduced it. Descendant matching iterates this flat list (bounded by program size, not depth) and pop_up_to prunes it by depth. The per-item has_ancestor_with_hereditary_jumps flag and the HereditaryJumpPtr.stack_offset resume coordinate are removed. * LocalName gets a manual Hash impl that ASCII-lowercases the Bytes arm so it agrees with the existing case-insensitive PartialEq. Without this, keying a HashMap on LocalName is unsound for any name that falls off LocalNameHash (custom elements, >12 chars, chars outside a-zA-Z1-6). This also fixes the same latent contract violation in TypedChildCounterMap. Re-executing the same address range against the same start tag was already a no-op after the matched-id set union in add_execution_branch, so dedup-once execution is observationally equivalent.

do_for_each_active_and_remove walked every registered end-tag handler on every end tag, i.e. O(elements currently open) per end tag. Registered handlers belong to open elements in stack order and the ones an end tag activates are always the most recently registered (an end tag pops a suffix of the open-element stack; end_tag_handler_idx already relies on removals only happening at the tail), so walking from the end and stopping once the active count is spent visits only the handlers that actually run. Documents that leave many elements open in the tokenizer's view -- long lists and tables with omitted </li>/</td>/</tr>/, which is valid HTML -- otherwise go quadratic as soon as anything inside them registers an end-tag handler: 100k `<li><a>x</a>` with an on_end_tag handler took about three seconds.

feed_text handed a valid UTF-8 chunk to the handler directly but then still entered the streaming-decoder loop with the (empty) remainder, which allocated an encoding_rs decoder per text node purely to remember that the node's final `last_in_text_node` chunk was still owed, and made every further chunk of the same node take the buffered 1 KB path. Track the "clean UTF-8 node open" state with a flag instead; the decoder is only created when non-UTF-8 (or split) input actually needs it. The flag is cleared before any handler runs so a handler that suspends on the closing chunk is not handed it again on resume. For a text-heavy 1 MB page this removes roughly 5% of total rewriter time.

HtmlRewriter is built on TransformStream / TransformController: the tokenizer reports each start and end tag by name and the controller decides, per tag, which tokens to have lexed in full. That layer was only exported behind the integration_test feature. Bun's HTML-to-Markdown converter needs every tag and text event but none of the rewriter machinery (selector matching, per-element handler closures, output serialization), and going through HtmlRewriter for that cost about as much as its whole conversion. This makes the layer public under lol_html::transform, documents it, and adds the small pieces a non-rewriting consumer needs: - TransformStream::controller() to reach the controller after parsing; - LocalNameHash::as_u64(), from_ascii() (const, for building match tables) and decode() (the packed hash back to the lowercase name; the encoding is reversible), so a controller can identify tags from the LocalName it is given without the lexed token; - doc comments for the items that become nameable. TransformController::should_emit_content() returning false already lets such a consumer skip serialization entirely; that is now documented.

…h HtmlRewriter Bumps lol-html to oven-sh/lol-html#3, which makes the transform-stream layer underneath HtmlRewriter public (`lol_html::transform`), and moves the converter onto it: a TransformController receives every start and end tag by name and requests the fully lexed token only for the few elements whose attributes are read. Selector matching, per-element handler closures and output serialization are no longer involved, and every end tag is visible, so stray ``/`` and mis-nested formatting (`abc`) now come out as in a browser; the workaround that stopped reopening unreachable formatting elements is gone. Tag names are identified from lol-html's packed name hash with an integer match; text and attribute bytes go into a bump arena instead of typed_arena's per-byte extend. Start tags dropped at the depth cap are counted per tag so their end tags are dropped as well. The same lol-html change carries the end-tag dispatch fix that was a local patch here and a UTF-8 fast-path fix in its text decoder; the stack fix is already on the fork's bun branch. patches/lolhtml is removed. Release build, min of 30 runs: WHATWG parsing spec (769 KB) 7.9 ms, Node fs docs (1.08 MB) 10.6 ms, Wikipedia "Rust" (996 KB) 5.1 ms, github.com/oven-sh/bun 1.6 ms -- previously 13.1 / 19.7 / 8.6 / 3.2 ms through HtmlRewriter, and 5.8 / 8.1 / 4.4 / 1.5 ms with html5ever. turndown on the same run: 150 / 237 / 105 / 38 ms. Binary delta against main: +160 KB.

…ollers Two additions for controllers that only read the document: - TokenCaptureFlags::RAW_TEXT with TransformController::handle_raw_text: text is handed over as the undecoded byte runs the tokenizer sees, bypassing TextChunk tokens, the streaming text decoder and the last-in-text-node bookkeeping. A consumer that already knows its input is UTF-8 (and does its own character-reference decoding anyway) skips a second validation pass and a token allocation per text node. TEXT keeps precedence when both are set; the bytes still reach the output sink as part of the unconsumed input. - StartTag::for_each_raw_attribute (Attributes::for_each_raw) and Attribute::name_raw / value_raw: attribute names and values as byte slices of the input, without materializing the Attribute list or a String per name and value. For Bun's HTML-to-Markdown converter these remove roughly 10% of total conversion time on text-heavy pages.

The organization requires every action reference to be a full-length commit SHA; with tag references every job on this repository (including the bun branch itself) has been failing at "Set up job" before running anything. Same actions and versions, pinned.

…ags by hash Bumps lol-html (oven-sh/lol-html#3) for two more transform-layer additions and uses them: - Text arrives through TokenCaptureFlags::RAW_TEXT as byte runs of the already-validated input instead of TextChunk tokens, so it is neither decoded through lol-html's streaming text decoder nor UTF-8-validated a second time, and is inserted into the tree straight away (only an unfinished `&…`/CR tail is held back) rather than staged in a String. Buffer inputs, which may be shared memory, are snapshotted first (convert_shared_utf8_bytes); string inputs are parsed in place. - Attributes are read with StartTag::for_each_raw_attribute, so nothing is allocated for attributes that are not kept, and the start-tag token is only requested where attributes are actually read (no longer for inline <code>). - Element identity is the tokenizer's packed name hash mapped to Tag by an integer match; only names outside the table (custom elements, unhashable foreign names) are ever decoded or stored. Common inline, SVG and MathML names were added to the table for that reason. - Scope walks test a Tag bitset, "close a p element" is skipped when no is open, the adoption agency has a fast path for the well-nested case, and scan.rs finishes short tails with one masked word instead of a byte loop. Same output as before on the page corpus and test suite. Release build, min of 30 (previous commit in parentheses): WHATWG spec 5.7 ms (7.9), Node fs docs 7.7 ms (10.6), Wikipedia "Rust" 4.0 ms (5.1), github.com/oven-sh/bun 1.2 ms (1.6) -- at or below the html5ever-based version on every page. Binary delta vs main +176 KB.

With the workflows actually executing again, -Dwarnings turns up three pre-existing issues on the bun branch: rustfmt drift in selectors_vm/mod.rs, an unused `Cell` import in the suspension tests, and an elided lifetime in the `text!` macro's type hint (elided_lifetimes_in_paths). No behaviour change.

… fixes; no code change for Bun)

…d bound The suspension change made UserData::set_user_data require `Send` data; the C API still passed the raw `*mut c_void`, so lol_html_c_api no longer compiled (hidden while CI could not start). Store it in a `UserDataPtr` newtype that is `Send` -- the C API only stores and returns the pointer, and thread-safety of what it points to has always been the C caller's contract. c-api clippy and the c-tests suite pass again.

cargo-semver-checks compares against the published lol_html of the same version. The bun branch intentionally adds API on top of 2.7.2 (RewritingError::Suspended from the suspension work, lol_html::transform) and is not published, so on this fork the check can only fail; keep it for cloudflare/lol-html.

…m-layer changes Reverts 6a170ff (pin workflow actions), eeea10d (warning fixes), 4afc2ca (c-api user-data Send wrapper) and 021637b (semver check scope). Those addressed pre-existing breakage on the bun branch that is unrelated to this change and can be handled separately.

…h HtmlRewriter Bumps lol-html to oven-sh/lol-html#3, which makes the transform-stream layer underneath HtmlRewriter public (`lol_html::transform`), and moves the converter onto it: a TransformController receives every start and end tag by name and requests the fully lexed token only for the few elements whose attributes are read. Selector matching, per-element handler closures and output serialization are no longer involved, and every end tag is visible, so stray ``/`` and mis-nested formatting (`abc`) now come out as in a browser; the workaround that stopped reopening unreachable formatting elements is gone. Tag names are identified from lol-html's packed name hash with an integer match; text and attribute bytes go into a bump arena instead of typed_arena's per-byte extend. Start tags dropped at the depth cap are counted per tag so their end tags are dropped as well. The same lol-html change carries the end-tag dispatch fix that was a local patch here and a UTF-8 fast-path fix in its text decoder; the stack fix is already on the fork's bun branch. patches/lolhtml is removed. Release build, min of 30 runs: WHATWG parsing spec (769 KB) 7.9 ms, Node fs docs (1.08 MB) 10.6 ms, Wikipedia "Rust" (996 KB) 5.1 ms, github.com/oven-sh/bun 1.6 ms -- previously 13.1 / 19.7 / 8.6 / 3.2 ms through HtmlRewriter, and 5.8 / 8.1 / 4.4 / 1.5 ms with html5ever. turndown on the same run: 150 / 237 / 105 / 38 ms. Binary delta against main: +160 KB.

…ags by hash Bumps lol-html (oven-sh/lol-html#3) for two more transform-layer additions and uses them: - Text arrives through TokenCaptureFlags::RAW_TEXT as byte runs of the already-validated input instead of TextChunk tokens, so it is neither decoded through lol-html's streaming text decoder nor UTF-8-validated a second time, and is inserted into the tree straight away (only an unfinished `&…`/CR tail is held back) rather than staged in a String. Buffer inputs, which may be shared memory, are snapshotted first (convert_shared_utf8_bytes); string inputs are parsed in place. - Attributes are read with StartTag::for_each_raw_attribute, so nothing is allocated for attributes that are not kept, and the start-tag token is only requested where attributes are actually read (no longer for inline <code>). - Element identity is the tokenizer's packed name hash mapped to Tag by an integer match; only names outside the table (custom elements, unhashable foreign names) are ever decoded or stored. Common inline, SVG and MathML names were added to the table for that reason. - Scope walks test a Tag bitset, "close a p element" is skipped when no is open, the adoption agency has a fast path for the well-nested case, and scan.rs finishes short tails with one masked word instead of a byte loop. Same output as before on the page corpus and test suite. Release build, min of 30 (previous commit in parentheses): WHATWG spec 5.7 ms (7.9), Node fs docs 7.7 ms (10.6), Wikipedia "Rust" 4.0 ms (5.1), github.com/oven-sh/bun 1.2 ms (1.6) -- at or below the html5ever-based version on every page. Binary delta vs main +176 KB.

… fixes; no code change for Bun)

robobun · 2026-10-18T11:48:39Z

A note on how this PR and #13 meet.

#13 adds tests that compare the text chunks of a handler that suspends with the chunks of a handler that does not. They pass on bun. When #13 is merged with this branch (6166e4c, no textual conflict), two of them fail: suspended_feed_yields_the_same_chunks and text_suspension_in_the_middle_of_a_lexeme.

The cause is how 6e9ace3 and resume_feed work together. While no handler suspends, a later lexeme of a valid UTF-8 text node stays on the fast path and is one chunk. After one suspension, resume_feed runs the decoder loop. The loop creates the streaming decoder, and the later lexemes then arrive in 1024-byte chunks. So a handler that suspends sees other chunks than a handler that does not.

To see it, run cargo test --lib suspended_feed_yields_the_same_chunks on the merge. The text node has three lexemes. The second lexeme is one 65-byte chunk without a suspension, and five 13-byte chunks with one (13 bytes is the chunk length in tests).

Whichever PR lands second has to keep the fast path in resume_feed when no input is left to decode.

robobun and others added 6 commits August 2, 2026 02:58

README: note the Bun fork; add bun-branch CI workflow

725ce49

The upstream ci.yml triggers on push: branches: [master] only, so an additive bun.yml runs cargo test --features=integration_test on pushes/PRs to bun without touching ci.yml (keeps rebases onto upstream clean).

sosukesuzuki mentioned this pull request Sep 8, 2026

Add Bun.markdown.fromHTML(): native HTML to Markdown conversion oven-sh/bun#41527

Open

sosukesuzuki added 2 commits September 8, 2026 08:14

sosukesuzuki added a commit to oven-sh/bun that referenced this pull request Sep 8, 2026

deps: lol-html at oven-sh/lol-html#3 head (pinned CI actions, warning…

d4652eb

… fixes; no code change for Bun)

sosukesuzuki added 3 commits September 9, 2026 00:39

sosukesuzuki added a commit to oven-sh/bun that referenced this pull request Sep 9, 2026

deps: lol-html at oven-sh/lol-html#3 head (same sources as d2efffb)

6aa5c49

sosukesuzuki added a commit to oven-sh/bun that referenced this pull request Sep 9, 2026

deps: lol-html at oven-sh/lol-html#3 head (pinned CI actions, warning…

3ed8ce0

… fixes; no code change for Bun)

sosukesuzuki added a commit to oven-sh/bun that referenced this pull request Sep 9, 2026

deps: lol-html at oven-sh/lol-html#3 head (same sources as d2efffb)

c72517e

Jarred-Sumner force-pushed the bun branch from ea6125b to e1a306d Compare September 17, 2026 02:28

This was referenced Oct 15, 2026

Write a lossy text chunk back from its source bytes #12

Draft

Copy the rest of a suspended text lexeme once, not at every suspension #13

Open

robobun mentioned this pull request Oct 18, 2026

HTMLRewriter: make an async text handler on a long text node linear oven-sh/bun#45912

Open

Navigation Menu

Repository navigation

Expose the transform-stream layer; UTF-8 text fast path; O(active) end-tag dispatch - #3

Expose the transform-stream layer; UTF-8 text fast path; O(active) end-tag dispatch#3
sosukesuzuki wants to merge 12 commits into
bunfrom
sosuke/transform-controller-api

sosukesuzuki commented Sep 8, 2026 •

edited

Loading

Uh oh!

robobun commented Oct 18, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

sosukesuzuki commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

robobun commented Oct 18, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sosukesuzuki commented Sep 8, 2026 •

edited

Loading