Two separate defects found while investigating #2838, both triggered by a file that is one very long line. Both are present on master and are independent of the highlight-perf work in #2841.
Reproduction file: the fp4_poll.json attached to #2838 — 451,855 bytes, zero newlines (one 441 KiB line), valid JSON.
1. Highlighting turns uniform after the first few wrapped rows
Past roughly 43 KB into the line, the highlighter emits one enormous Comment span instead of string/structure spans, so the rest of the view renders in the comment colour.
Spans returned for viewport windows on the same file:
| viewport |
spans |
first span |
| 0..4,000 |
81 |
String@1..8, Number@18..24, … |
| 50,000..54,000 |
1 |
Comment@43059..62192 |
| 200,000..204,000 |
1 |
Comment@195575..212192 |
| 430,000..434,000 |
118 |
String@430001..430013, … |
Byte 43,059 is exactly the // in src=\"https://forum.fairphone.com/.... The parse state has already fallen out of the enclosing JSON string, so // inside a URL opens a line comment — and because the file is a single line, that comment runs for ~19 KB until something re-synchronises.
Notes from investigation:
- Not a size threshold on its own: synthetic JSON with a single string padded to 8/16/32/48/64/96/128 KB containing a URL keeps the correct
String category at every size. The real file's content (escaped quotes, HTML, unicode escapes) is needed to trigger it.
- Leading hypothesis: syntect's regex engine gives up on the string rule when matching tens of KB in one
parse_line call (backtracking/stack limit), after which the parser continues outside the string context. Needs confirmation by instrumenting parse_line_into_spans in crates/fresh-editor/src/primitives/highlight_engine.rs.
- A naive mitigation (cap the bytes fed to syntect per call) is not sufficient alone: a fragment starting mid-string parsed from a line-start state would still be coloured wrong. The fragment's opening state has to be carried across chunks.
2. Wheel scroll after a scrollbar click collapses to a 100 KB boundary
Steps: open the file, click the vertical scrollbar to jump partway down, then scroll the wheel.
Observed viewport state:
initial top_byte=0 offset=0
after scrollbar click top_byte=0 offset=935
after wheel-down #1 top_byte=100000 offset=3
after wheel-down #2 top_byte=100000 offset=6
after wheel-down #6 top_byte=100000 offset=18
The first wheel event discards the scrollbar position: top_byte snaps to exactly 100,000 — the MAX_LINE_BYTES chunk size in LineIterator — and the row offset resets from 935 to 3. After that top_byte stays frozen and each notch advances only 3 rows, so the view appears to jump back toward the top of the file and then barely move. Scroll math is treating a 100 KB read chunk as if it were a logical line boundary.
Repro environment
Driving the real binary under tmux works, with two gotchas: the session needs an attached client (setsid script -qfc "tmux attach -t <session>" /dev/null &), and the workspace-trust dialog must be dismissed first (K, then Enter) or it swallows all input. Mouse events go in as SGR sequences via tmux send-keys -l, e.g. wheel-down \e[<65;100;25M.
Two separate defects found while investigating #2838, both triggered by a file that is one very long line. Both are present on master and are independent of the highlight-perf work in #2841.
Reproduction file: the
fp4_poll.jsonattached to #2838 — 451,855 bytes, zero newlines (one 441 KiB line), valid JSON.1. Highlighting turns uniform after the first few wrapped rows
Past roughly 43 KB into the line, the highlighter emits one enormous
Commentspan instead of string/structure spans, so the rest of the view renders in the comment colour.Spans returned for viewport windows on the same file:
String@1..8,Number@18..24, …Comment@43059..62192Comment@195575..212192String@430001..430013, …Byte 43,059 is exactly the
//insrc=\"https://forum.fairphone.com/.... The parse state has already fallen out of the enclosing JSON string, so//inside a URL opens a line comment — and because the file is a single line, that comment runs for ~19 KB until something re-synchronises.Notes from investigation:
Stringcategory at every size. The real file's content (escaped quotes, HTML, unicode escapes) is needed to trigger it.parse_linecall (backtracking/stack limit), after which the parser continues outside the string context. Needs confirmation by instrumentingparse_line_into_spansincrates/fresh-editor/src/primitives/highlight_engine.rs.2. Wheel scroll after a scrollbar click collapses to a 100 KB boundary
Steps: open the file, click the vertical scrollbar to jump partway down, then scroll the wheel.
Observed viewport state:
The first wheel event discards the scrollbar position:
top_bytesnaps to exactly 100,000 — theMAX_LINE_BYTESchunk size inLineIterator— and the row offset resets from 935 to 3. After thattop_bytestays frozen and each notch advances only 3 rows, so the view appears to jump back toward the top of the file and then barely move. Scroll math is treating a 100 KB read chunk as if it were a logical line boundary.Repro environment
Driving the real binary under tmux works, with two gotchas: the session needs an attached client (
setsid script -qfc "tmux attach -t <session>" /dev/null &), and the workspace-trust dialog must be dismissed first (K, then Enter) or it swallows all input. Mouse events go in as SGR sequences viatmux send-keys -l, e.g. wheel-down\e[<65;100;25M.