[odvcencio]

What does this PR do?

Registers the bash and html external-scanner ports with the scanner registry, so that the C-faithful core may bind them. With the scanner mode on, each port follows its pinned C scanner (tree-sitter-bash src/scanner.c, commit a06c2e4; tree-sitter-html src/scanner.c and src/tag.h, commit 73a3947). With the mode off, each port makes the same decisions as the port on main. The core serves no public parse, so no default behaviour changes.

Why this approach?

Both ports differed from C in many calls of the differential test (200,000 generated calls, seed 1): bash in 4,329 ordinary and 865 valid-symbol-mask calls, html in 11,828 and 21,860. The core is checked against C, so it needs the C behaviour. The default engine uses the same ports and must not change, so the C behaviour sits behind the scanner mode that internal/scanmode defines. A registration means: the core may bind this scanner and switch the mode on for its calls. Nothing else reads the mode.

The rule for the mode off: the port's Scan, Serialize and Deserialize equal main's. Each *_scanner.go keeps main's code and gains the mode flag in its payload, three dispatch lines and one scanmode.RegisterScanner call. The C-faithful code is new and lives in bash_scanner_c.go and html_scanner_c.go. The cost is duplicated scanning logic (about 1,100 lines for bash, 440 for html); the gain is that "equal to main" can be read and tested.

The reference of the mode is the one stated in internal/scanmode: the pinned C scanner compiled with signed char for a little-endian target.

bash. The Go port of bash was a re-implementation, not a transliteration: in the differential test it differed from C in 4,329 ordinary calls and 865 valid-symbol-mask calls of 200,000 (seed 1, made-up saved states off). The cause of the largest group, 1,908 calls where the port reads one byte more than C before it returns, is a structural one. In C a section that ends with return false ends the scan (for example scanner.c:879-884, the extglob section, when a ) is followed by white space). In the port each section is if scanX() { return true }, so a section that returns false falls through to the later sections, and the next one (bshScanBraceStart, grammars/runtime/bash_scanner.go:1120 on the base) reads the next byte. One fix in one place could not remove it, so the mode uses a line-by-line port of C's scan in a new file, and the existing port stays as the default path.

The bash scanner depends on the reference in one place: the heredoc delimiter is an array of char that is compared with the lookahead as an int (scanner.c:233), and its saved state writes a 32-bit length with memcpy. The mode uses signed comparison and a little-endian length on every target.

html. The Go port of html uses the tag table of html_tags.go, which the html, vue, angular, svelte, blade and astro ports share. That table numbers the tags in another order than tag.h, lacks some of its entries (for example COMMAND, ISINDEX, NEXTID and AUDIO) and has another content model for <p>. In the differential test the port differed from C in 11,828 ordinary calls and 21,860 valid-symbol-mask calls of 200,000 (seed 1). Because the saved state holds the tag numbers, the state bytes differ too.
Changing the shared table would change the default path of all six ports. This change does not touch it. The C tag table of html is added beside it, in a new file that only the html port uses, and the html scan, serialize and deserialize pick it when the mode is on. html_tags.go and the other five ports are byte for byte as on main. The cost is a second tag table; the gain is that the default path of six ports is unchanged and "equal to main" can be tested.

The html scanner does not depend on either: tag names are upper-cased ASCII, and the saved state holds two 16-bit counts, which the mode writes little-endian on every target.

What this changes

1. grammars/runtime/bash_scanner_c.go, html_scanner_c.go (new): the C-faithful scans, saved-state codecs and, for html, the C tag table and tag stack.
2. grammars/runtime/bash_scanner.go, html_scanner.go: the mode flag in the payload, three dispatch lines, one registration call each. Nothing else. html_tags.go is not changed.
3. grammars/runtime/{bash,html}_registration_test.go: the registration is present, the declared payload type is what Create returns, the mode is off in a new payload and can be set on and off, and the port binds through the real registration behind 0, 1 and 2 order adapters.
4. grammars/runtime/{bash,html}_scanner_c_parity_test.go: 8 bash tests and 5 html tests without cgo, each naming the C line it mirrors, run with the mode on, with the mode-off behaviour of main pinned beside it. Each fails without its fix: with the dispatch lines disabled, all 13 fail (negative/dispatch-disabled.log in the asset).
5. internal/ccore/bind/bindall/bindall_test.go: the list of registered languages gains bash and html.

No engine file changes and no root file is added. The CI matrix and cgo_harness/docker/ccore_equality.sh are unchanged.

bash: differences from main that the mode removes

difference	C	main's port	test
a section that returns false ends the scan	scanner.c:879-884 and every return false	falls through to the next section; reads one byte more	TestBashExtglobReadsNoByteBeyondC
end of input	lexer->eof where C says so (:431), lookahead == 0 elsewhere	any NUL byte counts as the end	TestBashNulIsNotTheEndLikeC
white space and letters	iswspace, iswalpha, iswalnum, iswdigit of the C locale (ASCII)	unicode.IsSpace and friends	TestBashWhiteSpaceIsTheCLocale
OPENING_PAREN, ESAC	declared (:11-41), never produced	produced	TestBashNeverProducesOpeningParenOrEsac
heredoc delimiter comparison	signed char against int (:233)	unsigned	TestBashDelimiterIsSignedLikeC
advance_word	appends to the delimiter (:164-190, :210-224)	replaces it	TestBashAdvanceWordAppendsLikeC
size limit of the saved state	TREE_SITTER_SERIALIZATION_BUFFER_SIZE 1,024 (:101)	the length of the buffer	TestBashSerializeStopsAtC1024
deserialize of a state with fewer heredocs	the array keeps its older entries (:121-157)	shortens the array	TestBashDeserializeKeepsExtraHeredocsLikeC

html: differences from main that the mode removes

difference	C	main's port	test
tag numbers and saved state	tag.h:5-166: 126 names, END_OF_VOID_TAGS 23, CUSTOM 126; counts then one byte per tag (scanner.c:36-72)	another numbering and table	TestHTMLTagTableAndStateFollowC
content model of <p>	the 26 tags of tag.h:271-276 end a paragraph	another list (for example <ul> ends it)	TestHTMLParagraphContentModelFollowsC
/> with no open tag	returns true; the C lexer starts a scan with result symbol 0, the external START_TAG_NAME (scanner.c:293-304)	returns no token	TestHTMLSelfClosingWithNoOpenTagReturnsStartTagNameLikeC
white space before a token	iswspace of the C locale (scanner.c:322)	space, tab, line feed, carriage return only	TestHTMLWhiteSpaceIsTheCLocale
size limit of the saved state	1,024 bytes; a tag that does not fit ends the list, and the tags left are read back as END_ tags (scanner.c:36-110)	the length of the buffer; fails closed	TestHTMLSerializeLimitIs1024LikeC

Correctness

• Focused package or unit tests ran for the touched behavior
• Affected parity validation ran in Docker, one language or grammar at a time
• If grammargen or parity logic changed, ran the smallest relevant focus target in Docker
• Documented anything intentionally left unvalidated in this PR
• Engine PR: before-and-after deterministic counters are in the PR body (work counters did not rise, reuse counters did not fall)
• Invariant gate ran for each language the PR can affect (incremental equals fresh, ERROR root implies HasError, root covers input, no-edit reparse allocates nothing)
• No language-name comparison in engine files (D6); one v1 lane only

Gate results, both architectures

The logs, summary lines, signature files, reproducers and the copy of the equality script behind this section and the next ones are in the asset pr-scanner-bash-html-33d0a.tar.gz (sha256 d16941644433b9382fe28c13aab20ba73890bfe588171b9d2690daf5b45f2260) of the evidence release; its README maps each folder to a check.

Seed 1, 200,000 generated calls, mode on, one grammar per process, no known-differences file. "Gate" is SCANDIFF_MODE=on SCANDIFF_BATCH=20000 SCANDIFF_HOSTILE=<off for bash, all for html> cgo_harness/docker/scandiff.sh <grammar> (exit 0 required), with the pinned C scanners and the repository's C runtime. "Equality" is cgo_harness/docker/ccore_equality.sh <grammar>. Each cell is compared / differing. The gate runs were made on the tree of #1625 plus this change; the files of this change are identical in the tree on main 0f3af76e.

grammar	run	architecture	ordinary	mask	midchar	C crashes	skipped	exit
bash	gate	linux/arm64	165,704 / 0	32,937 / 0	0 / 0	0	0	0
bash	gate	linux/amd64	165,704 / 0	32,937 / 0	0 / 0	0	0	0
bash	one order adapter	linux/arm64	165,704 / 0	32,937 / 0	0 / 0	0	0	0
bash	two order adapters	linux/arm64	165,704 / 0	32,937 / 0	0 / 0	0	0	0
bash	made-up states on, report mode	linux/arm64	166,239 / 0	31,630 / 0	1 / 0	687	0	0
html	gate	linux/arm64	87,786 / 0	102,676 / 0	76 / 0	0	0	0
html	gate	linux/amd64	87,786 / 0	102,676 / 0	76 / 0	0	0	0
html	one order adapter	linux/arm64	89,685 / 0	103,950 / 0	23 / 0	0	0	0
html	two order adapters	linux/arm64	89,685 / 0	103,950 / 0	23 / 0	0	0	0

Equality is failed=0 for both grammars on both architectures (bash: 73 trace, 76 digest, 76 record lines; html: 73, 74, 74); the incremental count is 0 because the harness has edit-session sets only for go, json, python, typescript and tsx. Range probes (a diagnostic, not in the gate): bash 1,360 / 0, html 6,223 / 0 in the gate runs.

bash and made-up saved states. The gate runs with made-up saved states off. The C scanner of bash reads past a damaged heredoc stack and crashes on made-up states (687 C crashes in the run with them on); a parse cannot produce such a state. The run with them on is in report mode: it found 0 differing in 166,239 ordinary, 31,630 mask and 49 damaged-state calls among the calls where C did not crash.

For comparison, main's port in the same harness (report mode, arm64): bash ordinary 4,329 differing of 165,704 and mask 865 of 32,937; html ordinary 11,828 of 87,786 and mask 21,860 of 102,676.

Linux amd64 is the CI image (Go 1.25.14, gcc 12.2, char signed). On Linux arm64 (Go 1.26.3, gcc 14.2) char is unsigned by default, so the C oracle of the arm64 runs was built with -fsigned-char through a cc and a c++ script at the front of PATH of the run; the repository's harness is not changed.

arm	linux/arm64	linux/amd64
gate, mode on, direct (both grammars)	yes	yes
equality script (both grammars)	yes	yes
bash with made-up states on (report mode)	yes	no
gate behind one and two order adapters	yes	no
mode forced off, and main's port (signature sets)	yes	no
harness self-tests (scandiff.sh --self-test, TestScandiffSelf*)	yes	no
unit tests of grammars/runtime, internal/ccore/bind/bindall	yes	yes
direct mode-off comparison with main's file (below)	bash, html	html; bash

What a gate PASS covers: gate mode rejects any difference in ordinary calls, in valid-symbol-mask calls and in calls that start inside a character. It does not decide on calls with damaged saved states or on included-range probes. A PASS does not show that a port equals its C scanner on every input.

How the mode-off rule was checked

First, the differential test ran twice over the same 200,000 calls on Linux arm64, in report mode: with the mode forced off, and with main's port in place of the registered one. Equal sets of difference signatures mean the two Go scanners gave the same result on every call where either differs from C.

grammar	signatures, mode off	signatures, main's port	sets equal
bash	7,436	7,436	yes
html	47,289	47,309	no: 20 only with main's port

The 20 html signatures are all in the category of made-up (damaged) saved states. Both html runs report the same counts of differing ordinary (11,828), mask (21,860), midchar (0 of 76) and range-probe (743) calls, and differ only in the damaged-state diagnostic (776 against 790). This is not traced (see the open point below).

Second, a direct comparison with no C involved: each port with the mode off against a renamed copy of main's file of the port. The test generates cases from a seed: random sources built from the literals of the file, white space of several kinds, control bytes, non-ASCII letters, invalid and cut-off UTF-8; every byte value 0 to 255 inserted at every position of 150 short bases; the empty input; offsets including the end; valid-symbol masks; and three kinds of start state (fresh, made-up states of 0 to 4,096 bytes including 1,023, 1,024 and 1,025, and states the scanner wrote itself). Each case compares the return of Scan, the token, the cursor and the farthest byte examined, the payload after the scan and after each Deserialize, and Serialize into buffers of 0 to 4,096 bytes. A panic in one scanner and not in the other counts as a difference. For bash, a made-up state whose heredoc length field is above 1 MiB is not compared (see the defect below).

grammar	architecture	cases	Scan calls compared	Serialize calls compared	Deserialize calls compared	skipped made-up states	differing
bash	linux/arm64	536,849	972,572	11,926,413	217,846	50,563	0
html	linux/arm64	530,449	1,060,898	13,128,705	265,047	0	0
html	linux/amd64	530,449	1,060,898	13,128,705	265,047	0	0
bash	linux/amd64	536,849	972,572	11,926,413	217,846	50,563	0

Controls: with the dispatch line of Scan forced to the C-faithful path while the mode is off, the same test (4,000 random cases and 12 bases per port) reports 1,939 differing comparisons for bash (21,409 cases) and 20,147 for html (20,129 cases), both on arm64. The comparison is a sample. It is not in the diff because it needs main's file as a copy; its source, runner and logs are in the asset.

A defect in the present engine

Main's bash Deserialize allocates a 32-bit length read from the saved state without checking it against the buffer (bash_scanner.go, make([]byte, delimLen)). With the mode on, the port clamps the length to the bytes given. With the mode off, the port is main's and still allocates. Only a corrupt saved state reaches it; a parse does not produce one. The direct comparison hit it (a test process ran out of memory), which is why that comparison skips such states.

Open point: html damaged-state signatures

In the signature check, 20 html signatures (47,289 with the mode off, 47,309 with main's port) appear only with main's port, all in the damaged-saved-state category. The direct comparison, which includes made-up states of 0 to 4,096 bytes, finds no difference between the port with the mode off and main's file, so I suspect the harness (the history of the previous call, or bytes the C scanner reads beyond the state), but I did not trace it. Separately, the calls with damaged saved states that differ from C in the html gate (diagnostic, outside the gate) count 42 of 3,240 on arm64 and 57 of 3,240 on amd64, and vary between runs of the same seed.

Local checks

On Linux amd64 (this host, Go 1.25.1), on the tree of this PR (a copy of main 0f3af76e with the diff applied), each check as its own job through the single-job Go slot of the host:

• go test -count=1 ./grammars/runtime ./internal/ccore/bind/bindall: exit 0.
• the same with -race (-p 1): exit 0.
• the new bash and html tests with -race -count=20: exit 0.
• the draft-correctness command (go test . ./grammars ./grammars/runtime ./roottest/bench ./roottest/parse -count=1 -timeout 25m): exit 0.
• one job for the static checks (exit of the job: exit 0): gofmt -l on the changed Go files: no output; go vet ./grammars/runtime ./internal/scanmode ./internal/ccore/bind/bindall: no output; the pinned staticcheck 0.6.1 with CI's package list: no output; go run ./cmd/v1layout -base main: 650 of 650 root files; go run ./cmd/v1guard -base main: 481 language-name uses match their baselines; go run ./cmd/citestplan: ok; -stale-run-names: no stale references; scripts/check_evidence_paths.sh: ok; scripts/check_generated_ownership.sh: ok.
• On Linux arm64 (container, Go 1.26.3), on the tree of add(grammars/runtime): gate eight scanner ports behind a C-faithful scan mode #1625 plus the changes: go test ./grammars/runtime ./internal/ccore/bind/bindall: ok; go vet: ok.
• git apply --check of the diff on a fresh copy of main 0f3af76e: ok.

Not validated

• One seed (1). Architectures other than amd64 and arm64. Other locales.
• bash calls with made-up saved states against C: the C scanner crashes on 687 of them.
• Incremental parsing equals fresh parsing for bash and html: the harness has no edit-session sets for them. The invariant gate was not run.
• The race build of the root package was not run; CI runs it.
• Gate arms behind adapters, the signature arms and the harness self-tests on amd64.
• The gate runs used the tree of add(grammars/runtime): gate eight scanner ports behind a C-faithful scan mode #1625 plus this change, not main 0f3af76e; the files of this change are identical, and the diff applies to main (git apply --check).
• The runs are bounded samples of generated calls, not proofs.
• main's html tag-name scan accepts ., C's does not; the mode follows C, and no unit test pins it.

Performance

• If perf-relevant, compared before and after with scripts/run_randomized_benchmarks.sh (GOMAXPROCS=1, -count=1 per process, 20 shuffle seeds, -benchtime=750ms, -benchmem); a fixed-order run is not evidence
• Included before and after numbers, or explicitly said perf is not the goal of this PR
• If large-grammar behavior changed, checked bounded RSS, timeout, or OOM behavior under Docker

Performance is not the goal of this PR. With the mode off nothing changes for the existing engines. The counter ledger (GOMAXPROCS=1 GOWORK=off go run ./cmd/perfcounterledger) printed counter ledger passed: 412 rows, 206 languages on this tree and counter ledger passed: 412 rows, 206 languages on main 0f3af76e; the outputs are byte-identical; the run on main left the tree clean (git status), and the run on this tree changed no file beyond the diff.

Maintainability

• No unnecessary abstractions or speculative cleanup outside the PR's scope
• Generated files or harness changes are only here because they are required by the change
• No new dependencies without justification

Self-review

Review your own diff before requesting review. Walk through it as if you're seeing it for the first time, and keep any self-review comments concise and focused on the non-obvious parts.

• Reviewed my own diff end to end
• Left comments on notable sections or the crux of the solution
• Called out anything I'm unsure about or want a second opinion on

The points to review first are bash_scanner_c.go against scanner.c, section by section (the labels regex, extglob_pattern, expansion_word and brace_start of C become the methods that bshScanC calls in order), and html_scanner_c.go against scanner.c and tag.h (the table htmlCTagNames must equal TAG_TYPES_BY_TAG_NAME in order; the tag type is the index, plus one from the entry after the last void tag, because the enum has END_OF_VOID_TAGS in between). The empty-stack /> case of html returns a token with the symbol that C's lexer starts with.

Due diligence

• Read the code you're changing before changing it
• Checked that similar patterns exist elsewhere in the codebase and stayed consistent
• If touching the parser or lexer: validated against diverse affected grammars
• If adding a grammar or scanner: verified against upstream C output