[odvcencio]

What does this PR do?

Adds the package internal/lexbuild: a Go port of the lexer path of the tree-sitter generator, release v0.27.1 (grammar front end for tokens, NFA, lex table construction, token conflicts, state merging and minimization, lex modes, reusable flags). Nothing calls the package yet, so no parse changes, and go list -deps . does not list it. It also adds lexbuild to the race lane of .github/workflows/ci.yml that already lists lexexact.

Why this approach?

What the package does

It reads a grammar.json and the valid tokens of each parse state, and builds the lex states the v0.27.1 generator builds, in the record form that internal/lexexact reads from a generated parser.c. Each Go function names the generator function it ports (file.rs: fn name) and keeps its order of operations and its iteration orders where they reach the output. The licence of the ported code (MIT) is in internal/lexbuild/LICENSE, copied byte for byte from the upstream release. The package uses the standard library and internal/lexexact only, reads no environment variable, and builds for wasip1.

Contract

• Input: a grammar.json and one token set per parse state (the tokens that are valid lookaheads in it). The package does not compute the parse table; the tests take token sets from a reference parser.c.
• Output: lex states with the accept token, the step at end of input, the step on an ill-formed byte and the ranges of every state, the lex state of every parse state, and the reusable flag of every action entry.
• Refusals: anything the generator refuses is refused (the error names the rule). Constructs not ported yet return a NotPortedError that names them: the regex flags x and -u, script, Script_Extensions, Age and segmentation property classes.
• Size limits (a deliberate difference from the generator, which has none). The generator runs until memory is gone on some inputs of a few hundred bytes ([ab]*a[ab]{24} has 33,554,432 reachable lex states; a SEQ of 24 two-way CHOICEs flattens to 16,777,216 productions; a?{1000000} has 2,000,001 automaton states). What holds in this package, and no more:
  1. Caps on the shape of the input, checked first (ParseGrammar, Prepare, and the repetition counts of patterns). A grammar over a cap is refused with a *SizeError that names the cap.
  2. Budgets on the constructions and passes behind the caps, each checked before the allocation or step that would pass it, each returned as a *SizeError by Prepare, Analyze, BuildUnmerged, BuildLexTables or NewFirstLastSets.
  3. Memory held by a construction is bounded by the budgets. The audit of the package (every place where the result or the working state can grow faster than the grammar) multiplies each limit by an estimated size of one item (rule node 112 B, production step 112 B, automaton state 80 B, range 8 B) and adds the constructions that are held at the same time: about 1 GiB. That figure is computed from the limits and the estimates; it was not measured at the limits.
  4. Work: what is enforced is that each pass named in the budget table counts its work against its budget before it does that work and stops with a *SizeError. A pass without a budget is polynomial in the capped sizes. No total of work is claimed: earlier versions of this text gave one (3 x 10^10, then 6.05 x 10^10) and three reviews found work that it left out.
     The package is written for the repository's own grammars and was not otherwise hardened against hostile input. Each limit is more than ten times the largest value over the 204 bundled grammars that could be measured; move and wat were not measured (the generator writes no reference parser for them). doc.go lists the same values (TestDocListsTheLimits checks that it does), and the census test fails if a limit is not ten times above the measured maximum.

Cap on the shape of the input	Cap	Largest measured
bytes of grammar.json	8,388,608	772,230
rules	16,384	1,071
external symbols	2,048	167
precedence orderings	512	44
names in one precedence ordering	256	21
bytes in a name (rule, symbol, field, alias, precedence, external, reserved set)	1,024	53
bytes in a string literal	2,048	151
bytes in a pattern source	262,144	16,644
largest count of a repetition {n} or {n,m} in a pattern	1,024	64
nesting depth of rules	256	23
entries in a list of the grammar (supertypes, inline names, conflicts and each conflict, extras, reserved sets and each set, members of a rule)	16,384	284

Budget	Limit	Largest measured
distinct precedence pairs	2,048	135
steps of the pairing of tokens that share a parse state	4,294,967,296	417,990,405
rule nodes times dropped rules in the removal of unreachable rules	1,073,741,824	368,082
rule nodes compared while looking for equal rules	4,194,304	223,247
rule nodes evaluated while expanding repeats	524,288	45,669
steps of the indirect-recursion search	32,768	2,873
bytes of production keys retained while flattening rules	8,388,608	669,443
steps of the keyword identification	4,194,304	412,300
steps of the grouping of parse states into lex states	17,179,869,184	1,599,019,852
automaton states visited while closing state sets over splits	33,554,432	2,815,851
steps of the grouping of transitions into character sets	2,147,483,648	157,325,188
steps of the ordering of lex states	33,554,432	2,142,237
steps of rendering lex states (large character sets and plans)	134,217,728	9,113,452
transition evaluations needed to list the ranges of the lex states	67,108,864	4,190,534
character ranges joined for the starting characters of the tokens that follow each token	8,388,608	576,146
character ranges read into the classes of the patterns of a grammar	16,777,216	11,769
expansion steps in the check of inline rules	200,000	134
symbols copied in the check of inline rules	16,777,216	678
automaton states in the lexical grammar	262,144	17,081
character ranges summed over automaton states	4,194,304	15,079
choice paths explored while flattening rules	131,072	6,728
production steps retained while flattening rules	1,048,576	35,153
steps in one flattened production	4,096	21
cells of a token-by-token or rule-by-token matrix	16,777,216	595,152
state sets in one token-pair conflict search	131,072	5,925
automaton states retained by one token-pair conflict search	2,097,152	18,522
lex states	131,072	11,785
automaton states summed over lex states	8,388,608	273,142
character transitions summed over lex states	1,048,576	43,023
character ranges summed over lex-state transitions	8,388,608	331,296
automaton states summed over the transitions of one lex state	65,536	319
steps of the comparisons while minimizing a lex table	134,217,728	8,327,580
lex states summed over the single-token constructions	262,144	9,727

• Exported signatures: BuildUnmerged returns an error as a third result; NewFirstLastSets returns (*FirstLastSets, error); the field SizeError.Limit is an int64. Other exported signatures are unchanged, and the fourth review round (below) changes none.
• Rejected alternative: changing the existing grammargen lexer builder. Its design (one automaton per lex mode, widened valid sets, fixes after construction, no end-of-input edges) differs from the generator's, so a port gives exact equality to check against.

Correctness

• Focused package or unit tests ran for the touched behavior
• Affected parity validation ran in Docker, one language or grammar at a time
• If grammargen or parity logic changed, ran the smallest relevant focus target in Docker
• Documented anything intentionally left unvalidated in this PR
• Engine PR: before-and-after deterministic counters are in the PR body (work counters did not rise, reuse counters did not fall)
• Invariant gate ran for each language the PR can affect (incremental equals fresh, ERROR root implies HasError, root covers input, no-edit reparse allocates nothing)
• No language-name comparison in engine files (D6); one v1 lane only

Tests

Run on Linux (amd64 for the local checks; a Linux arm64 container, Go 1.26, network off, for the checks that need the reference files) on main 24984a9c, with this change applied:

• go test ./internal/lexbuild ./internal/lexexact: pass (103 top-level tests in lexbuild, none fail). Also with -race, and go vet with GOARCH=386 and GOOS=wasip1 GOARCH=wasm. The whole lexbuild test suite also passes with GOARCH=386 (the test binary runs on this machine): 97 top-level tests pass and the rest skip because they need the reference files.

• Record equality with the generator's parser.c for three small grammars: lex states 44/44, 89/89 and 22/22 equal; lex starts 32/32, 137/137 and 28/28; reusable flags 224/224, 2,077/2,077 and 117/117. State behaviour equal in 53/53, 100/100 and 26/26 state pairs.

• Census over the bundled grammars against the generator, release v0.27.1 (needs the reference files; skipped without them): 202 equal; 2 differ (ocaml, verilog: the generator cannot write their unmerged table, so the coincident-token input comes from the merged table); 2 not measured (move, wat: the generator writes no reference parser for them). Ten more grammars are equal only given that merged-table input (ada, c_sharp, cpp, crystal, julia, kotlin, nim, odin, ruby, scala). Reserved-word rows are checked as a subset of the port's sets, because the generator removes tokens when it merges parse states. No measured grammar reaches a size limit.

• Differential test against the generator binary (TestLexDifferential, needs the binary, skipped without it). It asserts: any grammar that ends as differ, port refuses only, generator refuses only or reference unreadable fails the test after it prints them, and a run that compares nothing fails. -diffreport is an explicit report-only mode whose output says "report only, nothing asserted". The census returns a structured category so that a size refusal reaches its own bucket, "port refuses: size limit". Result of the asserting form: 24,000 generated grammars (seeds 6000 x 5000, 20000 x 10000, 40000 x 6000, 60000 x 3000), four runs, all pass. Both accept and equal: 14,695. Both accept and differ: 0. Port refuses only: 0. Generator refuses only: 0. Reference unreadable: 0. Port refuses as not ported yet: 418. Port refuses: size limit: 567. Both refuse: 8,319. Generator timeout, not compared: 1. The 567 size refusals are generated grammars that the generator accepts and a cap or budget refuses (for example a repetition count of 4294967295 on an empty expression, which the generator reduces to nothing). The earlier run without the caps had 15,258 equal, 429 not ported and 8,312 both refuse.

• Size limit tests (size_test.go): one test for each cap and each budget lowers the limit and checks the *SizeError from each entry point (TestShapeCaps, TestLexSizeBudget, TestLexMinimizeBudget, TestTransitionIDsBudget, TestConflictSearchBudget, TestFlattenBudget, TestFlattenKeysDoNotCopyNames, TestPrecedencePairsBudget, TestFirstLastBudget, TestClosureWorkBudget, TestNFABudget, TestPatternRangesBudget, TestInlineBudget, TestSymbolCellsBudget, TestWorkBudgets). TestAddStatesMatchesGenerator compares the linear closure with the generator's scan-and-insert version on 300 random automata. The blowup inputs run with the default limits and are measured (peak heap, sampled every 2 ms): lex table 0.15 s and 48 MiB, conflict search 0.16 s and 28 MiB, flatten 0.04 s and 40 MiB, 10,000 precedence names 0.04 s and 5 MiB, a?{120000} 0.69 s and 88 MiB. The 5,631-byte input with a 4,096-byte field name is refused by the name cap, and by the key budget (0.04 s, 39 MiB) when the name cap is raised. TestDifferentialClassifiesSizeRefusal checks the bucket.

• Fourth review round (work that was quadratic inside the caps). Each fix has a test that compares the new function with a copy of the old one on random inputs (quadratic_test.go), and a test with the input that made the old one slow. Times are from a Linux arm64 container, one run each (directional); the "before" column is the code of the pull request head, the "after" column is this tree:

  input (inside every cap)	function	before	after
  PATTERN of one letter with 1,000,000 i flags (finding 1)	filterPatternFlags (was flags += "i")	Prepare 43.6 s	Prepare about 20 ms
  class of 65,500 separate code points, then {64} (262,006 bytes; finding 2)	charSetFromClass (was add_range from index 0 for each range)	Prepare 52.8 s; the scan makes 64 x 65,500 x 65,499 / 2 = 137,285,904,000 iterations (computed; the test counts 1,999,000 for a class of 2,000)	Prepare 36 ms
  same grammar, BuildUnmerged with token set {0} (finding 3)	removeIntersectionSteps (was removeIntersection with front removals)	38.6 s; 274,571,808,000 range elements copied (the review's figure, computed) while the counter charged 8,384,128 steps	about 0.1 s (the counter now charges the merge)
  262,000 letters in one pattern	hirConcatOf (joined a literal run by copying it for each letter)	Prepare 7.9 s	Prepare about 60 ms

  The equality tests: TestFilterPatternFlagsMatchesAppendLoop, TestCharSetFromClassMatchesRestartLoop (canonical, ordered and unordered range lists), TestRemoveIntersectionMatchesInPlaceVersion (20,000 random pairs, including identical sets; the two remainders and the intersection, in order), TestAddMatchesRangeByRangeAdd, TestUnionOfMatchesAddOneByOne, TestHirConcatOfMatchesPairwiseJoin, TestClassDifferenceMatchesRestartScan, TestRangeConditionMatchesRangeByRange, TestDoesIntersectMatchesMerge, TestTerminalIndicesMatchesBitLoop, TestGetFollowingTokensMatchesPairwiseAdd. The search for the same shape in the whole package found these further places, each with its bound at the caps in growth-audit.md (rows 12, 13, 18, 20, 22, 33 to 38 and 40): the join of literal runs, the difference of classes, the unions of starting and following characters, the listing of the ranges of a state (Table.Ranges) and the joins of follow tokens were above 10^10 steps at the caps and are fixed; three budgets were charged too little (transition grouping, rendering, minimization rounds) and now count what they run; the others are stated with their numbers. Two budgets are new (bRangesWork, bFollowChars), and three limits are higher because the new count measures more: transition grouping 536,870,912 to 2,147,483,648 (measured 31,040,624 before, 157,325,188 now), rendering 33,554,432 to 134,217,728 (2,203,471 and 8,909,711).

• Fifth review round (three findings and a search for the same shapes; the details and the numbers are in growth-audit.md).

  1. parseGroup converted the whole remaining pattern to a string for each group. Four patterns of strings.Repeat("()", 131000) plus one letter (1,048,185 bytes of grammar) converted 68,644,524,000 runes. It now converts the next three runes, the longest prefix it tests. TestManyGroupsInspectThreeRunesEach counts the runes it inspects: 1,571,996 (at most 3 for each of the 524,000 groups), against 68,644,524,000. TestGroupHeadMatchesWholeRemainder puts the whole-remainder version back through a test hook and compares the trees and error texts on 20,000 drawn patterns with groups, non-capturing groups, named groups, flag groups and look-around forms (4,916 patterns that parse and 35,084 that are refused; both versions agree on every one).
  2. The range-work guard multiplied in int: (2*80500+4)*(15001+1) is 2,415,382,008, which a 32-bit build wraps to -1,879,585,288 and passes. rangesCost and its sum are now int64, the operands are converted before the product, and the check goes through checkWork. The audit of every budget check and counter found four more products that could pass 2^31 - 1 at the caps and changed them: splitStateIDGroups, collectLargeCharacterSets, planState (the three through mulWork), getFollowingChars, and the reserved-set cells in flattenGrammar (now through cells). The other counters are checked after every addition and their limit plus one addition stays far below 2^31 (listed row by row in the audit). Regression tests: TestRangesCostDoesNotWrap, TestRangeWorkGuardRefusesTheReviewsGrammar (the review's grammar through BuildUnmerged: the guard counts 2,415,382,008 and refuses), TestMinimizationProductDoesNotWrap, TestFollowingCharsWorkDoesNotWrap, TestReservedCellsDoNotWrap, TestRenderingProductDoesNotWrap, TestMulWorkAtTheCaps. Each computes the old arithmetic in int32 and requires that it wraps, so none depends on the host. With the old arithmetic put back in a scratch copy, the 386 run fails the seven overflow tests: six report the failure and TestMinimizationProductDoesNotWrap runs into the test timeout, because the unchecked loop does not stop. On a 64-bit host the old arithmetic does not wrap, so only the 386 run and the int32 checks inside the tests show the defect.
  3. Every elapsed-time assertion in the package's tests is gone. The tests read operation counters (opcount.go: package-level, compiled in always, added to once after the loops of the function that does the work) and compare them with a bound that a linear version meets and the earlier version misses by at least a factor of 20 (inRange checks the factor): TestPatternFlagsAreLinear, TestClassOfManyRangesIsLinear, TestRenderingManyRangesIsLinear, TestLongLiteralIsLinear, TestClassDifferenceIsLinear, TestStartingCharsAreLinear, TestRangesOfStateAreLinearInPoints. The blowup tests (TestLexSizeBlowupRefused, TestConflictSearchBlowupRefused, TestFlattenBlowupRefused, TestFirstLastBudget, TestClosureWorkBudget) now require that no budget counted more than twice its limit, in place of a time limit. The 8-minute watchdog of TestLexCensus is removed. Elapsed times are printed with t.Logf. The 60-second limit on the generator process in the differential test stays: it bounds the reference program, and a timeout puts the grammar in its own bucket instead of failing the test.
  4. The search for the three shapes found one more place: parseCounted built its "unclosed" error, which quotes the whole pattern, for every counted repetition before it knew it needed one. A pattern of 60,000 a{1} quoted 240,000 bytes 60,000 times (1.4 x 10^10 bytes). The message is now made where it is returned (TestCountedRepetitionsBuildNoMessage: 0 bytes quoted for a pattern that parses, once for one that fails). The other conversions of a remainder (filterPatternFlags, hirConcatOf, the property-class name scan) were fixed earlier or read each rune once.
  5. doc.go and this text state only what is enforced (see Contract). TestDocListsTheLimits now also fails if doc.go lists a limit the code does not have, if it claims a total of work, or if it lacks the sentence that the package was not hardened against hostile input.

• Sixth review round (three findings and a search for the same two shapes; the numbers are in growth-audit.md).

  1. The advance map cloned the accumulated ruled-out set for each leading advance, and bRenderWork did not count the clones: five maps of 8,000 separate ranges copied 5 x 8,000 x 7,999 / 2 = 159,980,000 ranges. The leading sets are now joined once (leadingRuledOut: the same canonical union) and the pairs, their sort and the join are counted before they are made. TestAdvanceMapJoinsLeadingSetsOnce counts the ranges collected (40,000 for the review's input, run at 8,000 alternatives in the container, 20,000 in the default test at 4,000) and requires that bRenderWork counted at least as many. TestRuledOutJoinMatchesCloneLoop compares the plans with the clone-and-add loop on 300 random states, and the census and differential code compares the plan of every state of both tables with the plan made by the clone-and-add loop (no difference in 202 + 24,000 grammars).
  2. The work tests accepted a count of zero. Every counter test now requires a positive lower bound that the input guarantees (inRange, workRange). TestWorkChecksRejectEarlierImplementations runs the earlier implementation of nine functions through the existing hook or its Old copy (which counts its work in the same counter), on a smaller input, and requires that the same check fails: groups (groupHeadOf, 16,008,000 against a bound of 24,000), flag filter (200,010,000 against 80,000), literal join (200,009,999 against 80,000), class of single code points (7,998,000 against 32,000), class difference (4,501,500 against 24,000), range intersection (4,507,498 against 24,000), union of starting characters (4,504,499 against 48,000), range conditions (10,000,000 against 28,000) and the advance-map sets (ruledOutOf, 10,005,000 against 40,000). The current implementation passes each check at the same size. Two counters are allowed to read zero and say so in the test: the range-insertion count of TestStartingCharsAreLinear (it measures the absence of insertions in the middle of a vector, and the positive count of the same run is checked beside it) and the quoted-pattern bytes of TestCountedRepetitionsBuildNoMessage (absence; TestMessageCounterCountsEagerQuotes shows the counter reads 100 x the pattern when the message is made eagerly).
  3. The accept state of each token, and the two split placeholders (REPEAT and +), were appended to the automaton without a check against bNFAStates. A grammar of a pattern of 262,143 letters and the pattern () ended with 262,145 states. Each now calls checkSize(0) first. TestAcceptStateIsChecked runs that grammar, and the boundary on seven small patterns (the limit at the number of states builds, one less refuses) and the limit zero on ().
  4. The search: 18 places that copy or join inside a loop and 16 arrays that a budget limits (tables in growth-audit.md). Fixed besides the review's: the join of the main-token sets and the copy of the first large set in collectLargeCharacterSets were not counted; the pairs of the advance map were not counted. Three places are bounded by the caps and not counted (the extra terminals, the lifting of common prefixes, the set operations of a class); they are listed with their bounds. doc.go is unchanged: it states only what is enforced.

• Independent review rounds found defects in earlier versions; each fix has a test that fails without it (for example assertions kept until expansion, JSON reading and number range as the generator's reader does them, nesting limit enforced before class-set lowering).

Repository checks on the final tree (the CI staticcheck command with v0.6.1 and the package list of the workflow prints nothing): go run ./cmd/citestplan and -stale-run-names, go run ./cmd/v1guard, scripts/check_generated_ownership.sh and scripts/check_evidence_paths.sh pass; go run ./cmd/v1layout prints 680/680 root Go files (the count of this tree's base; this change adds no root file); go list -deps . lists nothing from lexbuild. Data tables copied from the Rust Unicode sources start with a plain comment, not a Code generated marker, because no generator in this repository owns them.

Logs of the runs in this section and the growth audit: pr1618-lexbuild-daba3.tar.gz (sha256 1b2c96a6d4b1de9dd05d9e2f8de41554fa0438e60a31fe5a9e76007096a4051a).

Limits and not verified

• Not ported: the constructs listed under Contract. They return an error; they do not build wrongly.
• The valid tokens of each parse state come from the reference in the tests; the package has not been run with tokens computed by a Go parse-table builder.
• Each pass that has a budget counts its work against it; a pass without a budget is polynomial in the capped sizes; no total of work is claimed (see the tables and growth-audit.md). The walk for FIRST and LAST sets is the generator's (2 x rules x productions, 9.2 x 10^9 at the caps, not reduced) and the pairing of tokens depends on the parse states of the caller. The memory figure in the contract is computed from the limits and estimated unit sizes (rule node 112 B, production step 112 B, automaton state 80 B, range 8 B); it was not measured at the limits, and the large inputs of the table above were run only as far as that table says. Listing the ranges of a state costs up to 60 steps for each counted transition evaluation (three binary searches); that factor is estimated. The sampled heap peaks in the blowup tests are memory checks, not time checks; a sample can only under-measure. Go's encoding/json and the Unicode tables were not audited. A full run of the package at the caps of a 32-bit build was not made.
• The bound on Analyze and BuildLexTables takes the parse states that the caller passes as an input: the pairing of tokens that share a parse state and the grouping of parse states are budgeted, but the number of states is the caller's.
• The Unicode alphabetic table follows the Rust standard library the generator was built with (Go's unicode package is Unicode 15.0.0; the test reports 9,656 differing code points against the Rust tables of Unicode 17.0.0 and does not use Go's tables).
• The differential test covers the pattern and grammar.json forms its generator draws; other forms are covered by the census and the unit tests only. The generator's own limits (time, memory) are not modelled: a grammar the generator cannot build is not compared.
• The caps and budgets are chosen from the 204 bundled grammars that could be measured (not move and wat). A grammar that needs more than a limit is refused, not built wrongly. A pattern such as (){4294967295} that the generator accepts is refused by the cap on repetition counts.
• Parity validation in Docker, the engine counters and the invariant gate are not run: nothing calls this package, so no parse result changes.
• Tests that need reference files read them from the folder in LEXBUILD_REF, and from the binary in LEXBUILD_GEN; without them these tests skip. The census and the differential of the sixth round ran in a Linux arm64 container on the tree as pushed to the branch plus this change; the budget maxima of the census differ from the previous round in one budget only (rendering: 8,909,711 to 9,113,452, because the join of the main-token sets is now counted).

Performance

• If perf-relevant, compared before and after with scripts/run_randomized_benchmarks.sh (GOMAXPROCS=1, -count=1 per process, 20 shuffle seeds, -benchtime=750ms, -benchmem); a fixed-order run is not evidence
• Included before and after numbers, or explicitly said perf is not the goal of this PR
• If large-grammar behavior changed, checked bounded RSS, timeout, or OOM behavior under Docker

Performance is not the goal of this PR; no benchmark comparison is claimed.

Maintainability

• No unnecessary abstractions or speculative cleanup outside the PR's scope
• Generated files or harness changes are only here because they are required by the change
• No new dependencies without justification

The diff adds one package and one name in a CI package filter. The Unicode tables and the small golden files are inputs of the port and its tests.

Self-review

Review your own diff before requesting review. Walk through it as if you're seeing it for the first time, and keep any self-review comments concise and focused on the non-obvious parts.

• Reviewed my own diff end to end
• Left comments on notable sections or the crux of the solution
• Called out anything I'm unsure about or want a second opinion on

The limits above are the open points. The diff is about 19,500 lines, most of it Unicode tables and tests.

Due diligence

• Read the code you're changing before changing it
• Checked that similar patterns exist elsewhere in the codebase and stayed consistent
• If touching the parser or lexer: validated against diverse affected grammars
• If adding a grammar or scanner: verified against upstream C output