fix(parser): revive a stack whose own lex mode read a wider lookahead by xidus90 · Pull Request #1632 · odvcencio/gotreesitter

xidus90 · 2026-11-12T08:59:39Z

Fixes #1636. Refs #1559. Stacked on #1629: this branch is #1629's head a2030de (rebased on main 0f3af76) plus one commit of its own, d5dffe9. Please review and merge #1629 first. The diff of this PR on its own is git diff a2030de..d5dffe9.

What does this PR do?

On the plain DFA route, melonds_GPU.cpp from #1559 now parses to EOF with no ERROR, like C. On main it stops with no_stacks_alive at 75.5%. It also fixes the minimal valid case int f(){ v = (d[c] >> 8) | ((d[c] & 0x80) << 1); }.

Why this approach?

Root cause

This is cause B from the findings on #1559. After d[c], two stacks are live:

One (state 3926) reads >>.
The other (state 8556) reads the template-closing > (sym 190).

compareAngleTokenPreference elects the single >, so the >> stack dies at once. The > stack reduces into state 9092, which has no action for >, so no stack is left. C lexes each version with its own lex mode and keeps the >> version.

The angle rule cannot go: nested template closers need it. With it disabled, TestCppNestedTemplateCloseAngleMatchesC, TestApexGenericLocalDeclarationMatchesCShape, TestTypeScriptGenericLineageRegressions and TestTypeScriptShiftOperatorNotSplitAsCloseAngle fail. Nor can the election tell the two cases apart: it sees states, not stacks, and whether the > survives depends on the reduce chain.

Fix

The fix leaves the election and the lockstep loop alone and adds a fallback for the case where the shared token kills every stack.

At dispatch start, ownLexModeToken (from fix(parser): let each GLR stack read the shared lookahead with its own lex mode #1629) already lexes the shared token's bytes with the stack's own lex mode. It now also reports a reading that starts at the same byte and ends later, if the stack's state has an action for it. The parser records that reading with the stack's state and depth, before the stack reduces.
If every stack is dead at the start of the next iteration, reviveWiderStackReading revives the stacks that are still at their recorded state and depth with the first such reading. It returns that reading as the lookahead, and the parser moves the DFA lexer to the reading's end. tryRelexCurrentStateDFA repositions the lexer the same way.

The change has no language-name checks (D6). It does not run with included ranges.

Cost against #1629. #1629 returns before its probe when the election scanned no same-named rival (electionHadSameNameRival). This PR moves that check after the probe, where it gates only the same-span swap, because the wider reading must be seen without a rival. With the check left in front, TestCppShiftAfterSubscriptSurvivesCloseAngleElection fails again (no_stacks_alive at byte 24). So on multi-stack dispatches without a rival, this PR runs the probe where #1629 does not. #1629's electedTokenReusable shortcut, the ts_parser__can_reuse_first_leaf rule, still runs first. I have not measured the cost; CI's editor_latency will show it.

Correctness

First measured on main 0577027 + #1629 (cd88a46). Re-run after the restack on main 0f3af76 + #1629 (a2030de): the new test fails on a2030de (no_stacks_alive at byte 24) and passes with this PR, TestReviveWiderStackReading passes, melonds_GPU.cpp is accepted with 0 ERROR, grammars passes, the root package fails only the timing-based tests that also fail on main, and the ledger, v1guard and v1layout -base origin/main pass. Windows amd64, go1.27.0.

New grammars/TestCppShiftAfterSubscriptSurvivesCloseAngleElection: full S-expressions for int f(){ v = (d[c] >> 8) | ((d[c] & 0x80) << 1); } and x = i[] >> 8;, plus the empty input, with a 2 s timeout.
- On fix(parser): let each GLR stack read the shared lookahead with its own lex mode #1629 alone: FAIL, no_stacks_alive at byte 24 of 50.
- With this PR: ok.
- The expected trees are the C trees from py-tree-sitter with tree-sitter-language-pack, minus field names. These are not the locked commits.
New unit test TestReviveWiderStackReading in glr_test.go. It covers: a live stack blocks revival, a stale stack index, a stack that moved on, two different readings, and the same reading on two stacks. reviveWiderStackReading has 100% statement coverage. ownLexModeToken has 90.9% through the grammar tests; the uncovered line is the early return when no DFA token source is present.
The four angle tests above still pass.
go test -count=1 . ./grammars:
- grammars passes.
- The root package fails 10 tests. All of them also fail, at least intermittently, on main. Two examples: TestMalformedTokenInvariantGoLeafDoesNotRunBaseMergeRetry failed 1 of 8 runs on main, 3 of 8 on fix(parser): let each GLR stack read the shared lookahead with its own lex mode #1629 and 0 of 8 with this PR. The Awk/CMake/C# incremental tests are the timing-based ones described in fix(parser): stop no-action re-lex steps from undoing each other #1630.
go run ./cmd/v1guard and go run ./cmd/v1layout -base origin/main pass.
Witness files, plain route:

Input	main	#1629	this PR	C (pack)
`melonds_GPU.cpp` (upstream, 47,975 B)	`no_stacks_alive` at 36,225	same	`accepted`, 0 ERROR	0 ERROR
`(d[c] >> 8) \| …` snippet	`no_stacks_alive` at 24 / 50	same	`accepted`, matches C
`i[]>>&;` (invalid)	`no_stacks_alive`, error	same	`accepted`, no error	1 ERROR

Known gap, i[]>>&;: after the revive, the parse ends in LEGAL-REPLAY-RESYNC, which drops >> & without an ERROR. That is the false-clean replay from parser: legacy top-level replay returns false-clean trees outside the C recovery port #1530. With the prepared parser: legacy top-level replay returns false-clean trees outside the C recovery port #1530 change applied on top, the tree is (translation_unit (ERROR (expression (subscript_expression …))) (expression_statement)), as in C. I did not include parser: legacy top-level replay returns false-clean trees outside the C recovery port #1530 here.
Not run: the Docker cgo_harness parity suites. Please run the cpp, apex and typescript suites in CI.

Counters

go run ./cmd/perfcounterledger: counter ledger passed: 412 rows, 206 languages. No row moves past the 2 percent threshold against main (on d5dffe9).

Focused package or unit tests ran for the touched behavior
Affected parity validation ran in Docker (not run, see above)
Documented anything intentionally left unvalidated in this PR
Engine PR: before-and-after deterministic counters are in the PR body
Invariant gate: root covers the input on every witness. Incremental equals fresh was not run separately for this change.
No language-name comparison in engine files (D6)

Performance

Not measured with scripts/run_randomized_benchmarks.sh. On the hot path, the change appends to a slice only when a stack's own reading is wider. The revive check runs at iteration start only when that slice is non-empty.

Self-review

Reviewed my own diff end to end
Second opinion welcome: the revive keeps only the stacks recorded before they reduced, so the wider token is dispatched from the state C would lex it in. Stacks that already reduced on the narrow token are never revived.

Commits are signed.

…n lex mode In C++ 'A<B<T> >' and 'A<B<T>>', the shared GLR token election picks the binary '>' over the template-closing '>' for the stacks that hold B<T> as a template type. They accept both symbols, so the per-stack no-action re-lex never runs and the outer template is read as a comparison: aliases and declarations misparse, and inside a class template the parse runs out of stacks. C lexes each version with its own state's lex mode before it reduces. With several live stacks, each stack now re-reads the shared token once with its head state's lex mode and takes a same-span, action-bearing, non-keyword reading of an internal-DFA token. The DFA probe is shared with relexTokenForStackLexState.

The per-stack own-lex-mode reading also swapped tokens that a grammar tells apart by lex state alone. In the committed earthfile sample it fired hundreds of times (plain vs immediate string words, two `.` and `-` tokens), and the sample, which C and main parse cleanly, ended with an ERROR over its last 375 bytes; the counter ledger flagged new_nodes 4399 -> 6844. Take the reading only where the stacks disagreed about how to spell the same bytes: the union election scanned a same-named rival of the shared token in some live state's lex mode, and the reading has that name too. This keeps the C++ `A<B<T>>` fix and restores the earthfile sample.

…ition The per-stack re-read lexed the shared token again from its own start byte, after the whitespace the shared lexer had skipped. C lexes each version from its own position with its own lex mode, and lex modes read bytes after whitespace differently from bytes right after the previous token. Read from the token's start, Julia's ':' in 'c ? a : b', F#'s '<' and '+' and Earthfile's immediate string fragments got readings C never makes there. The same-named-rival limit already keeps those fixtures clean; this makes the reading itself match C wherever the limit lets it through. The re-read now starts at the stack's byte offset, lets the mode skip the gap itself (restarting after whole-token skips, without immediate tokens after a restart), and leaves modes with an after-whitespace start state or gaps over included ranges alone. The no-action re-lex keeps its start.

…he C token source On the registered C++ route (grammars.ParseFile, LangEntry.Parse) the C token source read 'A<B<T>>' with one '>>' token, because the comparison version accepts it. C lexes each version in its own mode: the template versions read the first '>' alone, since the template closer outranks '>>'. One shared token cannot be both, so the token source now emits the single '>' when a live state's own lex mode reads one byte there, as the DFA route's election already does. 'A<B<T>> x;' and the class-template aliases now parse like C on both routes.

…split against locked C

C keeps a version's lookahead through its reductions and looks the same symbol up in each new state (ts_parser__advance, parser.c:1720-1745). The per-stack reading lasted one dispatch: stackRelexRestoreTok put the shared token back, and after a single reduce the stack read again in its post-reduction state. A stack that reduced on its own reading could then shift the shared one. A stack that dispatched on its own reading and reduced in place now goes on dispatching with that reading until it shifts. TestOwnTokenIsKeptThroughAReduction is the reviewer's table: Q reduces on its own x and lands where the two readings shift to different states. The root was rule_xa and is now rule_xb, as C's rule gives.

C lexes a version again only when its cached token cannot be reused in the version's state (ts_parser__can_reuse_first_leaf, parser.c:470-503): the version's lex mode differs from the one the token was lexed in, and the entry is not reusable or the state has an external lex state. ownLexModeToken now applies that rule before it scans, through the existing transcription incr.FirstLeaf. The state the shared token was lexed in is the election's winner: nextGLRUnionDFAToken moves the elected scan to the front of its scan cache, and electedScan reads it from there. Every other reader of the cache looks scans up by state. electionHadSameNameRival also returns false before it walks the cache when the token did not win the last election. GLR dispatch asks it once per stack. On one W5 go one-byte edit pair (src/debug/elf/elf.go) that is 93,450 calls, none of which found a rival; each is now rejected in O(1). On the committed fixtures the rule skips 4 of 6 readings in cpp, 2 of 6 in earthfile and 45 of 68 in less. Trees of all 204 fixtures, on the DFA and token-source routes, are unchanged against main, and the generated counter ledger equals main's in all 412 rows.

…reading The own reading covers the shared token's bytes after the same gap, but it did not carry the shared token's skipped-prefix flag and start. A read that skips the gap inside one DFA run sets them itself. A read that restarts after a whole skip token, as Lexer.next does, has no record of the gap, so the new start position does not make the point moot. The reading now takes the flag and start from the shared token it replaces. TestOwnLexModeTokenReadsFromStackPosition checks them on every accepted case; "skip token, then a reading" failed before.

The shared GLR token election prefers a single `>` over `>>`, which nested template closers need. In C++ `(d[c] >> 8)` that kills the stack reading `>>` at once, and the stack that took `>` reduces into a state with no action for it, so the parse stops with no_stacks_alive. C lexes each version with its own lex mode and keeps the `>>` version. At dispatch start, record a stack's own-lex-mode reading when it starts at the shared token's byte but ends later and the stack's state has an action for it. If every stack then dies on the shared token, revive the stacks still at their recorded state and depth with that reading, and move the lexer to its end. The election and the lockstep loop are unchanged otherwise. Refs odvcencio#1559

This was referenced Nov 12, 2026

cpp: Parser.Parse stops with no_stacks_alive and a truncated root on five real files that locked C parses to the end #1559

Open

fix(parser): let each GLR stack read the shared lookahead with its own lex mode #1629

Open

xidus90 force-pushed the fix/glr-close-angle-width-fallback branch from 599d522 to 7c1eadd Compare November 12, 2026 09:33

xidus90 added 6 commits November 12, 2026 12:42

test(parser): cover the per-stack lookahead re-read

c149ff1

test(cgo_harness): pin the per-stack re-read and the C++ close-angle …

dc9ad6c

…split against locked C

xidus90 mentioned this pull request Nov 12, 2026

[parser]: cpp: >> after a subscript loses the shared-token election to > and kills every stack #1636

Open

xidus90 added 4 commits November 12, 2026 13:21

xidus90 force-pushed the fix/glr-close-angle-width-fallback branch from 7c1eadd to d5dffe9 Compare November 12, 2026 14:19

Navigation Menu

Repository navigation

fix(parser): revive a stack whose own lex mode read a wider lookahead - #1632

fix(parser): revive a stack whose own lex mode read a wider lookahead#1632
xidus90 wants to merge 10 commits into
odvcencio:mainfrom
xidus90:fix/glr-close-angle-width-fallback

xidus90 commented Nov 12, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

xidus90 commented Nov 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why this approach?

Root cause

Fix

Correctness

Counters

Performance

Self-review

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

xidus90 commented Nov 12, 2026 •

edited

Loading