cpp: Parser.Parse stops with no_stacks_alive and a truncated root on five real files that locked C parses to the end #1559

On main f015f68, Parser.Parse (the plain DFA-lexer route) stops with no_stacks_alive on five real C++ files. In every case the published root has one child, an ERROR that wraps the part of the file the parser reached. That part is 1.2% to 75.6% of the file. Locked C covers 100% of all five. melonds_GPU.cpp is the sharpest case: locked C and the registered token source both parse it with no error, and the plain route stops at 75.6%.

The registered token source (grammars.ParseFile, LangEntry.Parse) reaches the end of all five, but on four of them its tree is also mostly one ERROR. No existing issue covers any of this.

Lane: Legacy fixes (Q item) for a bounded change to the legacy loop or DFA lexer. Core (recover) if the owner decides to turn the C recovery port on for cpp.

Measurements

Main f015f68. Go 1.25.14, linux/amd64 (20 host CPUs, container limit 4 CPUs and 6 GiB), Docker image gotreesitter/cgo-harness:go1.25-local. Locked C: tree-sitter runtime 0.27.1, grammar tree-sitter-cpp at c0092228. cgo_harness at the same commit.

Plain route. gotreesitter.NewParser(lang).Parse(src). Line is 1-based. Column is a 0-based byte offset. "Root end" is the end byte of the returned root. The stop token is the lookahead that had no action. The retry ladder runs 5 or 9 passes; the table shows the pass that got farthest, which is the one published.

Truncated is true on all five. Three runs in one process gave the same digest each time.

Same files on the registry route and in locked C.

• Locked C covers all five completely and reports errors on four. It reports none on melonds_GPU.cpp.
• "Covers" on the registry route does not mean a useful tree. In four files one ERROR child spans 86% to 99.98% of the file. Locked C's tree for melonds_DSi.cpp has one small error. I did not compare any tree node by node.
• Locked C parses these files in 8.1 to 29.9 ms (median of 20 parses, each with a new Parser). The plain route takes 1.8 to 332 ms to reach the stop (three parses each).

With the C recovery port forced on (GOT_C_RECOVERY=cpp, plain route), all five end accepted and cover the file. The ERROR counts are 290, 120, 15, 129 and 28 in the table's order. melonds_GPU.cpp then has 15 ERROR nodes where locked C has none. parser_recover_c.go:624 gives that as the reason cpp stays opted out: forced recovery still differs on edited expressions.

With the packed version order off (Language.CompactPackedGSSVersionOrderCertified = false, which removes the C-order action-cell transaction from the path), the five plain-route stops do not change. The root ends are the same: 1,855, 25,322, 36,225, 43,702 and 1,721. So the transaction is not involved here, unlike in the hang draft (finding B).

The shipping Q variant qa-wave-scratch, applied to main through -overlay, changes nothing here: the five stops and root ends are the same.

Where the published plain pass first differs from the registry token stream. The index counts tokens in the pass. The first pass stops at token 82, 1,089, 265, 882 and 164. In basic_string.cpp and tsan_interceptors_posix.cpp every pass stops at that same token. In the other three the published pass is a later retry pass that gets much farther (5,618, 6,294 and 6,840 tokens).

The first pass of melonds_GPU.cpp is an exception. It sees exactly the registry tokens and still dies at token 265, the ) [3308,3309) of NonStupidBitField<128*1024/VRAMDirtyGranularity>() on line 102. The retry passes get past that line. So for this file part of the difference is in what the parser does with the same tokens, not only in what the lexer returns.

Where the published pass stops, plain route against registry route, token by token:

In all five files the final stop sits where the plain route returns a different token, or none, for text that the registry source lexes as an ordinary token. The parse then has no action, and with the C recovery port off for cpp it has no recovery either. Full token windows for each file are in results/tokens_around_stops.txt. This is my reading of the streams; I did not trace the lexer.

Small inputs. Delta debugging with the predicate "the plain route stops with no_stacks_alive while the registry route accepts and covers the input" shrinks each file to 4 to 9 bytes of text that is not C++:

They show the plain route stopping on a bare #, | or >>, or on string text next to an identifier. They are not valid C++, so they do not show the failure on valid code.

I also tried 18 hand-made snippets of valid code taken from the stop lines (each inside a function). 17 parse on both routes. One stops the plain route: void f(){ log("a: %" PRIx64 "\n", x); } (38 bytes). It ends no_stacks_alive on the plain route and accepted on the registry route. It is the melonds_DSi.cpp pattern (a macro name between two string literals). The stops in melonds_GPU.cpp, stl_tree.cpp, basic_string.cpp and tsan_interceptors_posix.cpp did not reproduce in isolation, so they need the earlier part of each file.

A possible link to #1558 (cpp hang). The first token differences in basic_string.cpp and stl_tree.cpp are a word the registry source returns as the keyword default and the plain route returns as an identifier. #1558 is a loop that flips a word between its keyword and identifier readings. Both may come from keyword handling in the DFA token source (promoteKeyword). I did not test that.

Reproduction

The inputs, harness and probe sources, and raw outputs are in issue-cpp-no-stacks-alive-f015f6802.tar.gz (SHA-256 in issue-cpp-no-stacks-alive-f015f6802.sha256 next to it). The five full source files are not attached, because they are third-party code under their own licenses; their names, sizes and SHA-256 values are in inputs.sha256. The 38-byte snippet above reproduces the melonds_DSi.cpp stop on its own.

1. Use a source copy of main f015f68 as the module github.com/odvcencio/gotreesitter.
2. Get the five files (not attached; see above) and put them in c/inputs/. Check them against inputs.sha256.
3. Build the program in repro/c_five_files/main.go: go build -tags grammar_subset,grammar_subset_cpp -o c_five_files ./c_five_files.
4. Run ./c_five_files c/inputs/*.cpp. Expected: for each file, the route=plain line shows stop=no_stacks_alive covers=false with the root end and stop token from the first table, and the route=registry line shows stop=accepted covers=true.
5. For locked C: cprobe/zz_oak_probe_test.go is a Go test added to the read-only cgo_harness module with go test -c -tags treesitter_c_parity -overlay ov/cprobe.json. Run the test binary from cgo_harness with OAK_LIST=c/c_list.tsv OAK_OUT=out.jsonl.
6. For the forced-recovery row: GOT_C_RECOVERY=cpp ./c_five_files c/inputs/*.cpp shows stop=accepted on the plain route.
7. For the token streams: oak probe -lang cpp -file c/inputs/<file>.cpp -route plain -log prints one token sym= start= end= line per token.
8. For the small inputs: oak ptest -list c/variants2.txt -timeout-ms 2000 prints stop reason and time for each snippet on both routes. The 38-byte snippet is the one with no_stacks_alive.

Lane

Legacy fixes (Q item) if the fix is a bounded change to how the legacy loop and DFA lexer handle a token that has no action. The design asks for no language-name test (D6). Counters must stay flat or improve (D10). The cpp ledger row uses a 2 KB sample, so it will not show a change.

Core (recover) if the choice is to enable the C recovery port for cpp. The recorded reason for the opt-out is at parser_recover_c.go:624.

Whether the plain route for cpp is in scope is the owner's decision. docs/api-guide.md says NewParser(lang).Parse always uses the DFA lexer and that the registered source "is not always the better lexer", so both routes are public.

Related issues

I searched issues and pull requests for: no_stacks_alive, basic_string.cpp, stl_tree.cpp, melonds, tsan_interceptors_posix, "cpp truncated", "cpp stop reason", "cpp recovery", "stops early cpp". Nothing covers these files. Closest:

• # field reports from downstream user (Windows GUI toolkit, pure-Go editor widget, multiple syntaxes) on latest release versions #454 (downstream field reports, open). Its C++ root-coverage item concerns a 137 KB generated fixture that was fixed in v0.54.0. These files are real code.
• parser: legacy top-level replay returns false-clean trees outside the C recovery port #1530 and javascript: production reports no error on two inputs where locked C reports one #1519 (false-clean trees from top-level replay). A different cause: those trees have no error where C reports one. These trees stop early.
• v1 Q2: budget retries by first-pass work #1306 (Q2, retry budget). It mentions one no_stacks_alive HTML fixture. It does not cover cpp.

What I did not verify

• Why the DFA lexer produces these tokens in these states. I compared token streams and did not trace the lexer.
• Other languages on the plain route.
• The compact route.
• Whether the files stop the same way on release v0.56.0.
• Node-by-node equality between the registry trees and locked C's.
• The licenses of the inputs before upload to the public evidence release. basic_string.cpp and stl_tree.cpp are libstdc++ headers (GPL-3 with the GCC runtime exception). melonds_*.cpp are melonDS (GPL-3). tsan_interceptors_posix.cpp is LLVM compiler-rt (Apache-2.0 with LLVM exception). I took this from the file headers and did not record upstream commits. The owner should decide whether to attach them.
• Every Go result here comes from an unmodified source copy of main f015f68.