[xidus90]

Fixes #1628.

What does this PR do?

In C++ A<B<T>> / A<B<T> >, the GLR stacks that hold B<T> as a template type now read the closing > as the template closer instead of the binary operator. Aliases, declarations and class bodies then parse like the C runtime, on two routes:

• the plain DFA route (Parser.Parse);
• cpp's registered token-source route (grammars.ParseFile, LangEntry.Parse, the cgo harness).

Commits, rebased onto main 0f3af76 (#1631). The earlier hashes are in parentheses.

1. d5654112 (e49472a5) fix(parser): let each GLR stack read the shared lookahead with its own lex mode
2. c149ff13 (9a38de8a) test(parser): cover the per-stack lookahead re-read
3. 331e9239 (cd88a469) fix(parser): limit the own-lex-mode reading to same-named rivals
4. aedcefc0 (9c130f9a) fix(parser): start the per-stack lookahead re-read at the stack's position
5. 92ca9413 (d239be39) fix(grammars): split a close-angle run for C++ template versions in the C token source
6. dc9ad6c1 (b5d35445) test(cgo_harness): pin the per-stack re-read and the C++ close-angle split against locked C
7. dfccea66 fix(parser): keep a stack's own reading through its reductions
8. 301e62df perf(parser): read no own token where C would reuse the shared one
9. a2030deb fix(parser): carry the shared token's skipped-prefix record onto the reading

Commits 7 to 9 answer the review of the first revision. They are described under "Changes after review" below.

Why this approach?

Root cause

tree-sitter-cpp has two tokens spelled >:

• the binary operator (sym 36);
• the template-argument-list closer token(prec(1, '>')) (sym 190).

After A<B<T> is closed by its first >, three GLR stacks are live:

• States 4092 and 4100 have B<T> as a template type inside A<…>. Their lex mode reads the next > as the closer (190).
• State 2787 holds the B < T comparison fork. Its lex mode reads > as the operator (36).

C lexes each version with its own state's lex mode, from that version's position, before reducing (ts_parser__advance, ts_parser__lex). So the template versions get 190 and close A<…>.

gotreesitter lexes one shared token per iteration. nextGLRUnionDFAToken scores 190 at 2 and 36 at 1. It still elects 36, because compareSpecificTokenPreference, a tie-break between same-named symbols, is checked before score. The template stacks then reduce B<T> and land in a state that has an action for both > symbols. They take the operator, so A < B<T> > … becomes a comparison.

The existing per-stack re-lex (relexTokenForStackLexState) cannot help:

• It runs only when a stack has no action for the shared token, and these stacks have one.
• It probes the post-reduction state. C lexes in the pre-reduction state.

The registered C++ token source has a second, separate problem with the unspaced A<B<T>>. It emits >> as one token, because the comparison version accepts >>. C's template versions read the first > alone: the template closer has lexical precedence 1, so their lex mode stops after one byte. One shared >> token consumes the byte that the template versions need to close A<…>.

Fix

1. Per-stack re-read (parser, e49472a5, cd88a469, 9c130f9a). When more than one stack is live, each stack re-reads the shared lookahead once with its own head state's lex mode, before it dispatches. This is the state C lexes in. A stack takes the reading only when all of the following hold:

   • The shared token comes from the internal DFA, not the external scanner, and is not the keyword-capture token or a reserved word.
   • The union election that produced the token also scanned, in some live state's own lex mode, a same-named rival at the same span (electionHadSameNameRival), and the reading has that name too (cd88a469). This limits the re-read to elections where the stacks disagreed about how to spell the same bytes, such as the two >. Grammars that tell many tokens apart by lex state alone are left alone, for example earthfile's plain and immediate string words.
   • The reading has exactly the same byte span, so the stacks stay in lockstep. It has a different symbol, which is not the keyword-capture token, and the stack's state has an action for it.

   The read starts where C starts this version (9c130f9a): at the stack's byte offset, before any whitespace the shared lexer skipped, so the lex mode has to skip that whitespace itself. Lex modes read bytes after whitespace differently from bytes that directly follow the previous token. After a whole-token skip, the read restarts in the same lex state, as Lexer.next does, and an immediate token after such a restart is not a reading. The probe leaves alone two cases it does not model: a lex mode with its own after-whitespace start state, and a gap across included ranges.

   The reading is scoped to its stack. The shared token is restored before the next stack, through the existing stackRelexActive / stackRelexRestoreTok mechanism. The DFA probe of relexTokenForStackLexState moved into scanTokenInStackLexState, and both paths share it. The no-action path keeps its start position (tok.StartByte) and its behaviour, including the zero-width external rescue.

2. Close-angle split (C token source, grammars/runtime/c_lexer.go, d239be39). When a close-angle run such as >> matches, the token source asks each live GLR state's own lex mode, using the grammar's DFA. If one of them reads a one-byte token there, it emits the single >. This is the choice the DFA route's election already makes (compareAngleTokenPreference). The choice between 36 and 190 then follows the token source's existing preference and the no-action re-lex.

No language-name checks in engine files (D6). The probe reuses the parser-owned relexProbeLexer.

How the two limits on the re-read relate

The first revision (e49472a5) read every token again from its own start byte, for every stack. In real fixtures, stacks then took readings right after a space that C never makes:

grammar	shared → own reading (read from the token's start)	site
julia	":"(51) → ":"(35)	? 0x94cb2bb20a28ce96 : 0x1f60a087
earthfile	_string_base_alpha → _immediate_string_base_alpha, "+"(73) → "+"(139)	RUN sbt compile, FROM +deps
fsharp	infix_op_token2 → "<", "+"(138) → infix_op_token1	i < span.Length, i <- i + 1

At every such site, reading from the stack's position gives back the shared symbol, or no token. So the read position is the root cause.

The same-named-rival limit (cd88a469) independently stops all of them: no election scanned a same-named rival at those spans. With cd88a469 alone, the julia, earthfile and fsharp fixtures and the C++ cases already behave like the full branch, and the start-position commit changes no fixture or counter on top of it. 9c130f9a stays because it makes the reading itself match C wherever the rival limit lets it through. Without it, a stack in such an election would still be read from after the whitespace.

The C++ readings come out as 190 under both limits.

Alternative considered: put score ahead of specificPreference in nextGLRUnionDFAToken. This would also fix the DFA repro, but it changes the shared election for every grammar and every stack, and it still gives the minority stack the wrong symbol.

Changes after review

1. The stack keeps its own reading until it shifts (dfccea66).
   • C keeps a version's lookahead through its reductions and looks the same symbol up in each new state (ts_parser__advance, parser.c:1720-1745).
   • Before this commit the reading lasted one dispatch. stackRelexRestoreTok put the shared token back, and the stack read again in its post-reduction state.
   • Now a stack that dispatched on its own reading and reduced in place goes on dispatching with that reading (goto retryAction) until it shifts.
   • TestOwnTokenIsKeptThroughAReduction is the reviewer's keptTokenLanguage table, unchanged. It gave rule_xa before this commit and gives rule_xb with it.
   • Scope: this covers the single-reduce path. On the conflict path and in the packed-transaction path, a stack still gets the shared token back after a reduction.
2. No read where C would reuse the shared token (301e62df).
   • ownLexModeToken applies ts_parser__can_reuse_first_leaf (parser.c:470-503) before it scans. It uses the existing transcription incr.FirstLeaf. The state the shared token was lexed in is the election's winner.
   • nextGLRUnionDFAToken moves the elected scan to the front of its scan cache, and electedScan reads it from there. Every other reader of that cache looks scans up by state. dfaTokenSource and glrStack keep their sizes.
   • electionHadSameNameRival returns false in O(1) when the token did not win the last election.
   • Effect on the committed fixtures: the rule skips 4 of 6 readings in cpp, 2 of 6 in earthfile and 45 of 68 in less. No tree changes.
   • Tests: TestOwnLexModeTokenSkipsAReusableSharedToken covers four cases: a token lexed in another mode, one lexed in the stack's own mode, a reusable entry, and a reusable entry in a state with an external lex state. TestElectionHadSameNameRival gains "token did not win the last election". Both new rejections failed before the commit.
3. The reading carries the shared token's skipped-prefix record (a2030deb).
   • The new start position does not make this point moot.
   • A read that skips the gap inside one DFA run records the gap itself. A read that restarts after a whole skip token does not.
   • TestOwnLexModeTokenReadsFromStackPosition now checks the flag and its start on every accepted case. "skip token, then a reading" failed before the commit.

Correctness

Host checks of this revision (Docker not re-run)

The parity suites, the corpus and fixture comparisons against the pinned C parser, the invariant gate and the benchmarks were not re-run in Docker for this revision. The tables further down are from the earlier head b5d35445. On this head, all checks ran on the host: Windows, Go 1.27, anchored -run.

As the oracle I used internal/ccore, the port of the C core, driven through bindtest. A scratch test that is not committed registers cpp's scanner for the comparison. The oracle reproduces the earlier Docker numbers:

• main gives 6 of 18 equal on the token-source route, as C did for the maintainer;
• main gives 173 corpus blocks equal on that route, as the earlier Docker run against C did.

check	main 0f3af76	this head
focused root tests (TestOwnTokenIsKeptThroughAReduction, TestOwnLexMode*, TestRelexTokenForStackLexStateNoAction, TestElectionHadSameNameRival, TestGLRUnion*, size budgets)	n/a	pass
go test ./grammars -run 'Cpp|OwnLexMode'		pass
18 C++ inputs vs ccore, DFA / token source (set listed below)	6 / 6	12 / 12
tree-sitter-cpp corpus c0092228, 198 blocks incl. test/corpus/c, vs ccore, DFA / token source	190 / 173	195 / 174; no block regresses
trees of all 204 committed fixtures (internal/benchfixtures/testdata/real), DFA and token source, vs main		identical
go run ./cmd/perfcounterledger	pass	pass, 412 rows; the generated ledger equals main's in every row
go vet ., v1guard -base origin/main, v1layout -base origin/main		pass

The 18 inputs:

• the 10 blocks of "Nested template calls", one per block;
• the three regression inputs;
• class A { H<I<J>> method() { typename K::template L<M<N>>::type …; typename K::template L<M<N>>::template O<P>::template Q<R>::type …; } };;
• the four std:: lines from the review.

The first revision (d5654112, rebased) gives 12 / 6 on this set. No input that it got right is lost. The reviewer counted 14 for that revision on their set, so the two sets or routes differ.

About the corpus count: my block extraction finds 198 blocks, where the reviewer counted 202. It also normalizes CRLF, so crlf.txt tests nothing in this comparison. On the DFA route, main differs from the oracle in 8 blocks and this head in 3, which matches the reviewer's 194 → 199.

The julia, earthfile, fsharp and pascal fixtures have unchanged trees against main on this head. They were not compared with C again.

Tests

• grammars/TestCppNestedTemplateCloseAngleMatchesC checks using X = A<B<T> >;, A<B<T>> x; and a class template with typename A<B<T>>::type and A<B<T> >::type aliases.

  • It compares full S-expressions and full-input coverage against C's trees, on the DFA route and on the registered token-source route (LangEntry.Parse).
  • v0.57.0 fails on both routes. Before d239be39, the token-source route failed for 2 of 3 inputs.

• grammars/TestOwnLexModeReadingKeepsEarthfileSampleClean (cd88a469): the earthfile fixture parses without error.

• grammars/TestOwnLexModeReadAfterWhitespaceMatchesC checks two things:

  • the Julia repro h = (UInt === UInt64) ? 0x94cb2bb20a28ce96 : 0x1f60a087, as an exact tree;
  • the root shape of the julia, earthfile and fsharp fixtures (error flag, end byte, child count), each as C has it.

  It fails on 9a38de8a and passes from cd88a469 on.

• Unit tests (parser_relex_state_test.go):

  • TestOwnLexModeTokenGuards gains cd88a469's cases for a rival under another name and for no rival at the span.
  • New TestOwnLexModeTokenReadsFromStackPosition covers seven cases:
    • a mode that skips the space;
    • a mode that does not skip it;
    • a stack past the token start;
    • an after-whitespace start state;
    • a skip token followed by a reading;
    • a skip token followed by an immediate reading;
    • an immediate reading after a skipped space.
  • parser_dfa_token_source_test.go covers electionHadSameNameRival.

• cgo_harness/TestOwnLexModeReadMatchesLockedC and TestOwnLexModeFixturesKeepLockedCRoot compare against the pinned C parser:

  • the Julia repro and four C++ inputs, exact tree, on the harness route and the DFA route;
  • the julia, earthfile and fsharp fixtures, root shape.

  Results in Docker:

  • this branch: all pass;
  • 9a38de8a's engine and token source (go -overlay): 8 subtests fail. Those are julia on both routes, three of the four C++ inputs on the token-source route, and all three fixtures.

Docker validation of the earlier head b5d35445 (cgo_harness, pinned C)

Not re-run for this revision.

Environment:

• Docker Desktop on WSL2 with a 4 GiB VM, so --memory 3500m and --gomemlimit 3GiB.
• Image gotreesitter/cgo-harness:go1.25-local, go1.25.14 linux/amd64.
• C oracle tree-sitter-cpp c0092228.
• Baseline is origin/main 934a39e (add(lexexact): add exact-lexer records for 195 grammars #1624).
• Probes compare with GTS_PARITY_COMPARE_HAS_ERROR=1, and baseline and branch ran the same probe program.

check	baseline 934a39e	this branch
TestParity{FreshParse,IncrementalParse,HasNoErrors,Highlight,GLRCanarySet}: cpp, c, c_sharp, typescript, tsx, javascript, rust, java	all pass	all pass
TestParityGLRCapPressureTopLanguages: c, cpp, java (exhaustive)	pass	pass
TestParityCorpusFreshParse (exhaustive; docs only for c, java)	c, java pass; others skip	same
grammargen real corpus (no-error / sexpr / deep over 25)	cpp 19/17/17; c_lang 23/20/20; c_sharp 23/20/20; typescript 25/24/24; tsx 25/25/25; javascript 23/23/23; rust 25/24/24; java 23/23/23	cpp 17/14/14 (see below), others identical
full tree-sitter-cpp corpus vs C, 197 blocks, harness route (token source)	173 pass / 24 fail	174 / 23: "Templates with optional type parameters" fixed, nothing new fails
"Nested template calls", one block per line, vs C	5 of 10 fail	2 fail
the three regression inputs vs C (harness route)	2 fail	pass
julia fixture vs C	exact	exact
earthfile, fsharp fixtures vs C	pre-existing divergences	the same divergences as baseline
pascal fixture and its first gate edit vs C	match	match

Why cpp drops from 17 to 14 in the grammargen real corpus: the test admits only samples whose reference parse (gotreesitter with the shipped blob) is clean, and it stops after 25. This branch repairs the reference parse of three reflection.txt blocks, which C parses without errors. Those blocks become eligible and expose generated-grammar gaps that also exist on baseline, and they push out three exact matches. This is a sample-set artefact, not a regression.

Invariant gate on the earlier head b5d35445 (scripts/run_v1_invariant_sessions.sh)

Not re-run for this revision: the script requires Docker. On this head, the host counter ledger and the fixture-tree comparison above stand in for it.

206 languages × 2 routes, 72 edit steps each:

• 412/412 sessions pass, with passed=true and source_unchanged=true;
• no-edit reparses allocate nothing;
• deterministic counters (counters.json) are byte-identical to baseline (sha256 7b46fd40…, 0 differing fields). The first revision had 40 differing fields in fsharp, julia, earthfile and pascal.

Broad host tests on the earlier head b5d35445 (Windows, Go 1.27)

• go test ./grammars/... passes except grammars/lean TestPackagedBlobMatchesNativeGrammar, a blob SHA-256 mismatch that does not involve the parser. I did not check it against baseline.
• go test . has 8 failures. All of them are among the 14 that fail on v0.57.0 in the same checkout: incremental leaf-reuse and fast-rescue tests.

Performance

editor_latency (go), this head, host replica

CI's editor_latency (go) failed on 9a38de8 with +6.6 to +7.7 % per edit. The gate itself needs Docker, cgo and the C oracle, so I could not run it on the host. Instead I used a scratch replica of its Go backend (GoA/GoB):

• the same fixture, golang/go@d00c67f src/debug/elf/elf.go, sha256 ff2f811d…;
• the same three workloads and the same parse calls as cgo_harness/editor_latency;
• base and head run interleaved, GOMAXPROCS=1, on Windows (Ryzen 7 9800X3D).

These are host numbers, not CI's.

Deterministic counts:

• One one-byte edit pair calls ownLexModeToken 93,450 times.
• None of those calls finds a same-named rival. Each one returns in O(1) now, before the scan cache is walked.
• No call reaches a read.
• In a CPU profile of 8 s of one-byte edits, neither ownLexModeToken nor electionHadSameNameRival has a sample.

Timing, benchstat:

workload	main	head
one_byte (20 × 750 ms)	94.49 ms ± 8 %	98.49 ms ± 7 %	~ (p=0.096)
hundred_byte (20 × 750 ms)	218.8 ms ± 11 %	231.5 ms ± 10 %	~ (p=0.114)
typing (20 × 750 ms)	2.011 s ± 4 %	2.061 s ± 8 %	~ (p=0.242)
one_byte (30 × 20 ops)	93.88 ms ± 5 %	94.93 ms ± 5 %	~ (p=0.602)

B/op and allocs/op are unchanged. Host noise is about 5 to 10 %, so these runs cannot resolve a change of a few percent, and I do not call the cost fixed. CI's editor_latency (go) on this head decides. On cd88a46 that job was still running when I wrote this.

Earlier head b5d35445 (Docker, not re-run)

Measured with scripts/run_randomized_benchmarks.sh against baseline 934a39e:

• script defaults: GOMAXPROCS=1, 20 interleaved seeds, -benchtime=750ms, -benchmem;
• one container pinned to CPUs 6,7;
• medians, n=20, benchstat golang.org/x/perf@v0.0.0-20261009192801-be2c69fb417e.

benchmark	base	this branch	delta	B/op	allocs/op
GoParseFullDFA	7.204 ms ± 13%	6.903 ms ± 12%	~ (p=0.862)	1.146 KiB → 1.146 KiB	6 → 6
GoParseIncrementalNoEditDFA	7.040 ns ± 16%	6.379 ns ± 10%	~ (p=0.176)	0 → 0	0 → 0
GoParseIncrementalSingleByteEditDFA	61.90 µs ± 11%	62.14 µs ± 13%	~ (p=0.547)	393 → 393	5 → 5
CPPConditionClauseAmbiguityDFA/while_assign	18.98 µs ± 7%	18.81 µs ± 8%	~	1.133 KiB → same	6 → 6
…/while_equal	19.12 µs ± 5%	18.70 µs ± 8%	~	same	6 → 6
…/if_assign	19.44 µs ± 9%	18.82 µs ± 8%	~	same	6 → 6
…/for_assign	19.07 µs ± 9%	19.52 µs ± 9%	~	same	6 → 6

No significant change.

These benchmarks run the DFA route only and do not reach the C token source. So I also measured allocations per fresh parse through LangEntry.Parse("cpp"): 50 repetitions of each line, testing.AllocsPerRun, Windows host.

input ×50	token source without the split (as on baseline)	this branch
int f(int a) { return a >> 2; }	24, no error	24, no error
void g() { x = a >> b; y = c<d<e>>(1); }	345, no error	490, no error
std::vector<std::vector<int>> v;	237, HasError	2637, no error

• A plain shift costs nothing extra: only one state is live there and it reads >>.
• Where several states are live at a >>, liveLexModeReadsOneByte builds one Lexer per live state.
• The std::vector line: the correct template parse forks more than the old error parse did. The token-source route used to misparse it.

Remaining gaps

The std::-qualified inputs: still open, being worked on

The root cause is found. The fix needs an R3 decision.

Six of the 18 inputs still differ from C on this head:

• std::vector<std::shared_ptr<Foo>> v;
• std::map<std::string, std::vector<Foo>> m;
• std::vector<std::vector<Foo>> g();
• auto p = std::make_shared<std::vector<Foo>>();
• B<C::D<E, F>>::G field;
• K::L<M<N>> v = K::L<M<N>>{};

This is not a lexing problem. I compared the GLR trace with ccore's C-format parse log for std::vector<std::shared_ptr<Foo>> v;:

• After std::vector<std::, C has two versions in state 8983 at the same byte, the comparison reading and the template reading.
• C merges them into one stack node with two links. ts_stack_can_merge compares only status, state, position, error cost and external scanner state.
• At shared_ptr C reduces _scope_resolution over both links, which leaves two versions, 3596 and 7921. The template reading survives and later closes std::shared_ptr<Foo>.
• The legacy engine refuses the GSS merge because the two versions' dynamic precedence differs (gssMergeScoresRefuse). cpp's one-version-per-merge-key policy then keeps only one history.

The existing per-blob bit LegacyMergeAdmitDynamicPrecedence admits C's merge. elixir already sets it. With the bit on cpp's exact profile, on top of this head:

• all 18 inputs give C's trees on both routes (ccore);
• the corpus gives 197 / 175 equal blocks (main 190 / 173), and no block regresses;
• the cpp sample moves toward the core: 727 visible nodes instead of 732 (the core has 717), and 3 ERROR nodes instead of 6.

The bit also raises the counter ledger's cpp default new_nodes from 1171 to 1379 (+17.8 %). That is an R3 exception, which I cannot approve. So the bit and its tests sit on a separate local commit (3dfee8bd, not pushed), waiting for the maintainer's decision. The six new TestOwnLexModeReadMatchesLockedC rows on that commit have not been compiled or run: there is no cgo on the host. On main, the bit alone fixes one more of the 18 inputs, on the DFA route only. Both changes are needed.

Also

• The kept own reading covers the single-reduce path only (see above).
• 24 of the 198 tree-sitter-cpp corpus blocks still differ from ccore on the token-source route, and 3 on the DFA route. All of them also differ on main.

Checklist

• Focused package or unit tests ran for the touched behavior (host, this head)
• Affected parity validation in Docker: ran on b5d35445, not re-run for this head. On this head the oracle was ccore on the host.
• Documented anything intentionally left unvalidated in this PR
• Engine PR: deterministic counters before and after. On this head, the generated counter ledger (host) equals main's in all 412 rows.
• Invariant gate: 412/412 on b5d35445, not re-run for this head (Docker only)
• No language-name comparison in engine files (D6); one v1 lane only
• No unnecessary abstractions or speculative cleanup outside the PR's scope
• No new dependencies
• Reviewed my own diff end to end

[xidus90]

Regression: earthfile real sample goes from clean to has_error

go run ./cmd/perfcounterledger against main 0577027 flags one row for this PR:

earthfile default full: new_nodes rose from 4399 to 6844 (>2%)

This is a correctness regression, not only extra work. On the committed sample internal/benchfixtures/testdata/real/earthfile (1,894 bytes, plain DFA route):

	main	this PR	C (py-tree-sitter, tree-sitter-language-pack)
HasError()	false	true	false
tokens	492	478
new_nodes	4,399	6,844

With this PR, one ERROR covers bytes 1518–1893 (the smoke-test: target from line 56 to the end).

Where it goes wrong

The shared token streams first differ at byte 660. On main, the parser gets block_token1 (indent) and then the keyword COPY. With this PR, the indent is gone and COPY arrives as _string_base_alpha. That follows from earlier OWN-LEX substitutions. The first is at byte 59, WORKDIR (sym 45 → _immediate_string_base_alpha 133) on a stack in state 1619, which continues the previous line's string.

Over the whole file, OWN-LEX fires hundreds of times. Earthfile tells many tokens apart by lex state alone: _string_base_alpha/_immediate_string_base_alpha (130→133), . (60→49), - (61→50), [ (73→139), and others. Swapping them per stack while the rest of the loop stays in lockstep changes which versions survive, and the parse ends on an error path that C does not take.

What I tried (none is enough on its own)

1. Only take a reading with the same symbol name as the shared token, which is what the C++ >/> case needs. This removes WORKDIR→string, but .→. and -→- remain, and the sample still has an error.
2. Never take an immediate token after skipped whitespace. This has no effect: Language.ImmediateTokens is false for all of these earthfile symbols.
3. Probe from where the shared lexer started, before the skipped whitespace, and require the same span. This is C-faithful, because C lexes each version from the previous token's end. It removes the whitespace cases (./ after a space), but most substitutions have no skipped prefix, and the sample still has an error.

All three kept TestCppNestedTemplateCloseAngleMatchesC, the Apex test and the two TypeScript angle tests green.

Options

• Narrow the rule to what the PR targets: apply the own-lex-mode reading only when the shared election was decided by the same-name tie-break (compareSpecificTokenPreference) between two candidates the stacks disagree on. This is closer to the "alternative considered" in the description.
• Keep the per-stack reading but gate it on proof that the grammar's lex states agree on everything except the disputed symbol. That is harder, and I have not designed it.

This PR should not merge until one of these is settled. #1632, which is stacked on this PR, does not add a ledger change of its own, but it inherits this one.

Regression test to add with the fix (fails on this PR, passes on main):

func TestOwnLexModeReadingKeepsEarthfileSampleClean(t *testing.T) {
	src, err := os.ReadFile(filepath.Join("..", "internal", "benchfixtures", "testdata", "real", "earthfile"))
	if err != nil {
		t.Fatalf("read earthfile sample: %v", err)
	}
	parser := ts.NewParser(DetectLanguageByName("earthfile").Language())
	parser.SetTimeoutMicros(10_000_000)
	tree, err := parser.Parse(src)
	if err != nil {
		t.Fatalf("parse failed: %v", err)
	}
	if root := tree.RootNode(); root.HasError() || root.EndByte() != uint32(len(src)) {
		t.Fatalf("has_error=%v end=%d/%d", root.HasError(), root.EndByte(), len(src))
	}
}

[xidus90]

Fixed in cd88a46: the own-lex-mode reading now applies only when the union election scanned a same-named rival of the shared token (C++ >/>), and the reading has that name. The earthfile sample parses cleanly again, the counter ledger passes with no row changed (412 rows), and TestOwnLexModeReadingKeepsEarthfileSampleClean pins it. The description has the details.

[odvcencio]

Thank you for this work, and for finding the earthfile regression yourself before I got to it. The diagnosis is right: C lexes once per stack version, and one shared lookahead for all stacks is where this engine goes wrong. I checked the PR by running it. The runs below used your first revision (e49472a), except where I name another; I will repeat them on the head you call ready.

What I ran (main 0577027; "C" is the pinned runtime, 0.27.1)

Check	main	first revision
TestCppNestedTemplateCloseAngleMatchesC	fails, also on v0.57.0	passes
18 C++ inputs, node by node against C	6 equal	14 equal
tree-sitter-cpp corpus blocks (202) against C	194 equal	199 equal

No input is right on main and wrong with the fix. The four that stay different are std::-qualified, the common spelling in real code:

std::vector<std::shared_ptr<Foo>> v;
std::map<std::string, std::vector<Foo>> m;
std::vector<std::vector<Foo>> g();
auto p = std::make_shared<std::vector<Foo>>();

For the first one the engine still gives a binary_expression where C gives a declaration with a template_type. Your table lists B<C::D<E, F>>::G field; and K::L<M<N>> v = ... as open, so you may have these already. I mention them because a fix that leaves std::vector<std::shared_ptr<Foo>> wrong will not feel fixed to users.

For reference: internal/ccore, the port of the C core in this repository, gives C's result on all 18 inputs (trace, digest and records equal), because it lexes per version as C does.

One thing to change: keep the stack's own token until it is shifted. C keeps the lookahead through reductions: after a reduction it looks the same symbol up in the new state (ts_parser__advance, cgo_harness/internal/coracle/src/parser.c:1720-1745). Here the stack's reading lasts for one dispatch. stackRelexRestoreTok puts the shared token back, and when the loop returns to the stack after its reduction, the stack reads again with the new state's lex mode. The test below (built from your ownLexModeLanguage) shows it: Q takes its own x, reduces, and lands in a state where the two readings shift to different states. The engine shifts the shared reading and the root is rule_xa; by C's rule the root is rule_xb (I read the C code for that; I did not run C on this table). main gives rule_xa too, so this is not a regression. It is a path where a stack reduces on one symbol and then shifts another. On your first revision that shape occurred in the committed samples of earthfile, less, make, luau, groovy, nim and asm, and not in C++, Go, JavaScript, Python or TypeScript; I have not counted it on cd88a46.

Test (root package)

func keptTokenLanguage() *Language {
	return &Language{
		Name: "kept_token", SymbolCount: 7, TokenCount: 4, StateCount: 9, ProductionIDCount: 1,
		SymbolNames: []string{"end", "a", "x", "x", "n", "rule_xa", "rule_xb"},
		SymbolMetadata: []SymbolMetadata{
			{Name: "end"}, {Name: "a", Visible: true}, {Name: "x", Visible: true}, {Name: "x"},
			{Name: "n", Visible: true, Named: true}, {Name: "rule_xa", Visible: true, Named: true}, {Name: "rule_xb", Visible: true, Named: true},
		},
		LexStates: []LexState{
			{Default: -1, EOF: -1, Transitions: []LexTransition{{Lo: 'a', Hi: 'a', NextState: 2}, {Lo: 'x', Hi: 'x', NextState: 3}}},
			{Default: -1, EOF: -1, Transitions: []LexTransition{{Lo: 'x', Hi: 'x', NextState: 4}}},
			{AcceptToken: 1, Default: -1, EOF: -1},
			{AcceptToken: 2, Default: -1, EOF: -1},
			{AcceptToken: 3, Default: -1, EOF: -1},
		},
		LexModes: []LexMode{{LexState: 0}, {LexState: 1}, {LexState: 0}, {LexState: 0}, {LexState: 0}, {LexState: 0}, {LexState: 0}, {LexState: 0}, {LexState: 0}},
		ParseActions: []ParseActionEntry{
			{},
			{Actions: []ParseAction{{Type: ParseActionShift, State: 1}, {Type: ParseActionShift, State: 2}}}, // 1: `a` forks into Q and P
			{Actions: []ParseAction{{Type: ParseActionShift, State: 8}}},                                      // 2: P on the shared reading
			{Actions: []ParseAction{{Type: ParseActionReduce, Symbol: 4, ChildCount: 1}}},                     // 3: Q reduces on its own reading
			{Actions: []ParseAction{{Type: ParseActionShift, State: 4}}},                                      // 4: T on the shared reading
			{Actions: []ParseAction{{Type: ParseActionShift, State: 5}}},                                      // 5: T on Q's reading
			{Actions: []ParseAction{{Type: ParseActionReduce, Symbol: 5, ChildCount: 2}}},                     // 6
			{Actions: []ParseAction{{Type: ParseActionReduce, Symbol: 6, ChildCount: 2}}},                     // 7
			{Actions: []ParseAction{{Type: ParseActionShift, State: 3}}},                                      // 8: goto n -> T
			{Actions: []ParseAction{{Type: ParseActionShift, State: 6}}},                                      // 9: goto rule_xa
			{Actions: []ParseAction{{Type: ParseActionShift, State: 7}}},                                      // 10: goto rule_xb
			{Actions: []ParseAction{{Type: ParseActionAccept}}},                                               // 11
		},
		ParseTable: [][]uint16{
			{0, 1, 0, 0, 8, 9, 10},
			{0, 0, 0, 3, 0, 0, 0}, // Q (lex mode 1)
			{0, 0, 2, 0, 0, 0, 0}, // P
			{0, 0, 4, 5, 0, 0, 0}, // T
			{6, 0, 0, 0, 0, 0, 0},
			{7, 0, 0, 0, 0, 0, 0},
			{11, 0, 0, 0, 0, 0, 0},
			{11, 0, 0, 0, 0, 0, 0},
			{0, 0, 0, 0, 0, 0, 0},
		},
	}
}

func TestOwnTokenIsKeptThroughAReduction(t *testing.T) {
	lang := keptTokenLanguage()
	p := NewParser(lang)
	source := []byte("ax")
	ts := newDFATokenSourceDirect(NewLexer(lang.LexStates, source), lang, p.lookupActionIndex, nil, nil, nil)
	tree := p.parseInternal(source, ts, nil, nil, arenaClassFull, nil, 0, 0, 0, false)
	if tree == nil {
		t.Fatal("nil tree")
	}
	defer tree.Release()
	if got := tree.RootNode().Type(lang); got != "rule_xb" {
		t.Fatalf("root = %s, want rule_xb: Q reduced on its own reading and must shift it too", got)
	}
}

A pointer that may help with the cost. C decides in ts_parser__can_reuse_first_leaf (parser.c:470-503) when a token lexed for one version may be used by another: the two states have the same lex mode, or the state has no external lex state and its table entry is marked reusable. Skipping the per-stack read when the stack's lex mode equals the mode the shared token was lexed in is therefore exact, not a heuristic. The cost is visible: CI's editor_latency (go) failed on 9a38de8 with +6.6 to +7.7 percent per edit, next to the counter_ledger failure you already fixed.

Smaller. The replacement token does not carry the shared token's skipped-prefix flag. I counted no stack killed because of that on the samples of 204 languages, and your new start position may make the point moot.

A maintainer has to approve the workflow runs of each push, because this is your first contribution here. I approved cd88a46 and will approve the next ones as they come. Tell me when the branch is where you want it, and I will run the full set again on that head.

[xidus90]

Thank you for the careful review and for the test. I rebased onto main 0f3af76 (#1631), not the 0577027 you used. The PR body has a map from the old commit hashes to the new ones. The new head is a2030deb.

What ran this round. Docker parity, the invariant gate and the Docker benchmarks were not re-run on this head. Everything below ran on the host (Windows, Go 1.27, anchored -run). As the oracle I used internal/ccore through bindtest, with cpp's scanner registered by a scratch test that is not committed. On main this oracle gives 6/18 and 173 equal corpus blocks on the token-source route, the same counts that C gave.

1. Keep the stack's own token until it is shifted (dfccea66)

• A stack that dispatched on its own reading and reduced in place now goes on with that reading (goto retryAction) until it shifts. Before, stackRelexRestoreTok gave it the shared token back.
• TestOwnTokenIsKeptThroughAReduction and keptTokenLanguage() are in verbatim. The test gave rule_xa before the commit and gives rule_xb with it.
• Scope: the single-reduce path only. On the conflict path and in packed transactions, a stack still gets the shared token back after a reduction.

2. std::-qualified nested templates. The root cause is the stack merge, not lexing. The fix needs your decision, because it moves a ledger row.

• For std::vector<std::shared_ptr<Foo>> v; I compared the GLR trace with ccore's C-format log:
  • After std::vector<std::, C has two versions in state 8983 at the same byte. It merges them into one node with two links.
  • The reduction at shared_ptr pops both links, and the template reading survives.
  • The legacy engine refuses that GSS merge because the two versions' dynamic precedence differs. cpp's one-version-per-merge-key policy then keeps only one history.
• The existing per-blob bit LegacyMergeAdmitDynamicPrecedence, which elixir already sets, admits C's merge. With it on cpp's profile, on top of this head:
  • 18/18 equal on both routes;
  • tree-sitter-cpp corpus, 198 blocks: 197 / 175 equal (DFA / token source) against main's 190 / 173, with no block regressing;
  • the cpp sample comes closer to the core: 727 visible nodes instead of 732 (core: 717), and 3 ERROR nodes instead of 6.
• The bit also raises the ledger's cpp default new_nodes from 1171 to 1379 (+17.8 %). That is an R3 exception, which I cannot approve. So the bit and its tests are a separate commit that I have not pushed.
• Without the bit, this head gives 12/12 of 18 (DFA / token source). The rebased first revision gives 12/6 on my set; you counted 14 for it on yours. So my set, or the route, evidently differs from yours. Please check against your 18.
• My 18: the 10 lines of "Nested template calls", the three regression inputs, one method body with both typename … L<M<N>> lines, and your four std:: lines.
• My corpus extraction finds 198 blocks to your 202. It also normalizes CRLF, so crlf.txt tests nothing there. On the DFA route, main differs from the oracle in 8 blocks and the head in 3, which matches your 194 → 199. My token-source figures (173 → 174, or 175 with the bit) answer a different question.
• The six new rows in TestOwnLexModeReadMatchesLockedC on the separate commit have not been compiled or run: there is no cgo on the host.
• The std:: cases stay open in this PR, and I am still working on them. If you want the bit here, say so and I will push it with these numbers. Otherwise it becomes its own PR.

3. Cost (301e62df)

• ownLexModeToken now applies ts_parser__can_reuse_first_leaf before it reads, through the existing incr.FirstLeaf, with the election's winning state as the state the token was lexed in. The winner's scan is kept first in the scan cache, so neither struct grows.
• electionHadSameNameRival now returns false in O(1) for a token that did not win the last election.
• On the committed fixtures the rule skips 4 of 6 reads in cpp, 2 of 6 in earthfile and 45 of 68 in less, with no tree change.
• editor_latency's gate needs Docker and cgo, so I measured a host replica of its Go backend: the same elf.go fixture and the same three workloads.
  • One one-byte edit pair makes 93,450 calls. None finds a rival, and each now returns in O(1). An 8 s CPU profile has no samples in either function.
  • Timing, main → head:
    • one_byte: 94.5 → 98.5 ms ± 8 % (p=0.10); in a second run, 93.9 → 94.9 ms ± 5 % (p=0.60);
    • hundred_byte: p=0.11;
    • typing: p=0.24.
  • B/op and allocs/op are unchanged. Host noise cannot resolve a few percent, so I do not call the cost fixed. CI's editor_latency (go) on this head decides. On cd88a46 that job was still running when I wrote this.

4. Skipped-prefix flag (a2030deb)

• The new start position does not make this moot. A read that restarts after a whole skip token, as Lexer.next does, has no record of the gap. The reading now takes the flag and its start from the shared token.
• TestOwnLexModeTokenReadsFromStackPosition checks both on every accepted case. One of them failed before the commit.

Other host checks on this head

• The counter ledger passes, and the generated ledger equals main's in all 412 rows.
• All 204 committed fixtures give the same trees as on main, on the DFA and token-source routes.
• go vet, v1guard and v1layout against origin/main pass.

From my side, the head is ready for points 1, 3 and 4 and for your full run. Point 2 waits on your call about the bit.