[odvcencio]

What does this PR do?

It follows #1592 (the package internal/ccore, a Go port of the C tree-sitter runtime 0.27.1) and #1595 (root adapters and equality tests against pinned C). It replaces the core's append-only arena by a paged record store with reference counts, free lists, a tree cell with a finalizer and an edit allocator. Nothing in the library calls the package: no parse route exists, so no default behaviour changes. This revision also lowers the cost of a fresh parse on the store (allocations per parse from 48 to 7 for 1 MiB of Go; see Performance) and corrects the benchmark harness that measured it.

Why this approach?

With an arena, a long editing session keeps every record it ever made; the store lets records of released trees be reused, as C does with its reference counts.

Design

• The tests of the linear reduction (from main) are adapted to the store's parser construction: each parser gets a store of its own in place of the arena's clear, and the comparison adds the reference words; kidSpan returns an empty list for a node without children, so such a node can be the first node of a store.

• Records, child lists and scanner-state bytes live in fixed-size pages (4,096 records per page; child lists of 1 to 8 handles, then powers of two; lists longer than a page sit behind a stable pointer; state bytes in 64 KiB pages). Free records and child lists go on free lists, sorted by address in batches so that a parse allocates in ascending order.

• API changes: Parse and ParseWith(src, *Tree) return a *Tree; Tree.Edit edits in place; Tree.Copy and Tree.Release give shared ownership; a tree dropped without Release is released by a finalizer. The parser keeps no tree: Digest, Records, ArenaDigest, Walk, RootEnd and DigestOf take the tree and read through a read context of their own, so they never write the parser's state. A stopped parse returns no tree.

• Counting is not done during the parse (C counts during the parse; the cost of that is not archived here). A publication walk counts when an incremental tree is published; the release of a tree is queued. Tree.Release drains on the caller's goroutine. The finalizer of a dropped tree only appends its entry and counts the cell down; if that was the last cell it ends the store itself when the lease is free, otherwise it starts one goroutine that tries the drain. The protocol, step by step, is at the head of internal/ccore/lifetime.go.

• A fresh parse neither logs allocations nor publishes, and it does no reachability sweep. Records that the finished tree does not reach (a replaced root, discarded error leaves) and the storage they own stay in the store until the whole store ends: releasing the first tree, editing it and reparsing does not reclaim them. Later incremental parses sweep what they allocate and the tree does not reach, so a new set of such records does not build up on every reparse. The retained share can rise as the document shrinks. The measured shares are below; TestCcoreDigest asserts at most 10 percent over each CI input set.

• A cached heap token is handed out once; later hand-outs are copies of the record (with its state bytes), so a leaf sits in one slot of a tree. Nothing in the parser stops a new internal node from being placed in two slots, and C's compression tests ownership with no size condition. So before the balance step every parse that finishes a tree walks it for new records reached more than once (findMulti); the balance step treats them as shared (C's ref_count > 1 test). A fresh parse has no publication walk, so it adds the extra owners to the reference words (countFreshShared); an incremental parse's publication walk counts them itself. Committed tests: TestBalanceRefusesANodeReachedTwiceThroughTheRealEntry (the five-node graph, with zero-width and one-byte leaves), TestFreshTreeCountsARecordItOwnsTwice (through publication and release) and TestFreshParseCountsANodeTheTreeReachesTwice (through Parse).

• Edit against Release of the same tree: a running Edit holds one count in the cell count and marks its cell; a Release on another goroutine neither ends the store nor queues the release of the root the edit reads; the edit queues the releases at its commit. A panic inside Edit lowers the count, clears the mark and queues a deferred release.

• A parse takes the old tree's released flag, root and page-table headers together under the short mutex after its initial drain, so an Edit of that tree on another goroutine cannot give it a root that its headers do not cover.

• One rule for internal failures. A panic or a limit refusal inside a publication walk, a sweep or a drain poisons the store and is swallowed there. ParseWith turns it into (nil, stop) with a named reason; a drain in Release, in a finalizer's goroutine or in the exit of a parse returns quietly, so nothing in the store can terminate the process. A poisoned store frees nothing piecemeal again: its drains discard their queue, a parse of one of its trees (also one that was waiting for its lease when it was poisoned) parses afresh in a new store, and its pages go to the pool when its last tree is released or dropped; the end check runs after a failed drain too. An Edit that fails after it began to allocate, up to the point where its commit has published the new root and queued the operations, poisons the store and re-throws to its caller, which exists (Edit has no result); the commit sets its done flag last and unlocks with a defer. A waiting parse whose old tree is released while it waits is rejected ("old tree released") on every round, also when the store was poisoned meanwhile.

• Limits, per store: the record index (2^28 - 1), the reference count (2^28 - 1), the child pages whose start offset must stay below the large-list tag (2^19 pages) and the scanner-state pages (2^16 pages) are checked before anything is handed out, by the parse allocator and by the edit allocator; the shared page-growth limits (record, child and state pages) are checked and the page claimed in one section of the short mutex, so a parse and an edit of separate trees cannot both pass one limit; the cursor within a page and a parse's free lists belong to the holder of the lease, an edit's cursor to the edited tree (one Edit at a time), and an edit's pops from the shared free lists run under the short mutex; a record that owns storage is claimed first (the slot is taken and left empty) and filled after its storage exists, so a refused record strands nothing and no other allocator can take its slot in between; an index from a free list is checked against the record limit like a bumped one; the first child page is under the child-page limit. A refusal stops the parse with a named reason, or poisons the store when it is in publication, a drain or an edit.

• Three integrity counters are kept for the whole process: sharedInTree (a new record met in two slots during publication), doubleFree, badRetain (a retain that reaches a free record).

• A test hook point parse-wait fires in ParseWith after the check of the old tree and before the lease is taken, so a test can wait for a parse to reach the lease instead of sleeping.

• internal/ccore/bind/bindtest follows the new API; internal/ccore/bind is unchanged.

• docs/v1-design.md gains one paragraph under "Invariant gate": its four checks apply to the public parse API, and the core's own ParseWith hands out a new tree handle on every call, with its allocation count pinned by a test.

Fresh-parse cost (this revision)

• The walk for records reached twice (findMulti) still runs for every finished tree. A fresh parse now keeps its marks in a bit set with one bit per record slot (findMultiBits) instead of in the flags word of each record and a list of visited records; the walk, its order and the counts it gives are unchanged, and the walk with flag marks stays for incremental parses and for a store of more than 2^24 slots. marks_internal_test.go compares the two on hand-built graphs, cycles and 300 random graphs, and the refusal over the bound.
• The three page tables (recs, refs, kids) start as slices over arrays inside the store core (four slots each) and, for a larger parse, are sized from the number of pages the parser's last fresh parse took (or from the source length for the first parse) instead of growing by doubling.
• The stack that compress uses in the balance step stays on the parser instead of being built again on every parse.
• An ended store drops its pages everywhere: the pages go to the pool, and the page tables, the inline page-table arrays of the store core (recsIn, refsIn, kidsIn, cleared after the pages are handed to the pool, not before) and the lease holder's lists (rel, walk, taken, sortTmp, freedRecs, freedEditRecs, spans, freedEditSt) are cleared, so a parser or a released tree that stays reachable and keeps the core reachable keeps no page and no list. TestReleasedTreeHoldsNoPage checks the inline arrays, the lists and the parser.
• The benchmark harness changes are in their own section under Performance.

Reading a tree from several goroutines

Many goroutines may read one Tree value at the same time (Reader, Root, the accessors of a view, Digest and the other readers). Edit and Release need exclusive use of that Tree value: no goroutine may read it while another edits or releases it. A goroutine that wants to read while another edits takes its own Copy and reads that. A view returned by Reader keeps the tree's cell reachable, also when the caller copies it by value, and a view, the handles read through it and every slice its accessors returned are valid until the owning tree is edited or released; a caller that needs them longer reads an unedited Copy. No lock is taken on the read path.

Correctness

How to check

• go test ./internal/ccore/... -count=1 and -race; go test -race -count=20 ./internal/ccore/. The store tests are store_test.go (19 test functions), storefix_test.go (30), storeg_test.go (8), storeg_internal_test.go (3), storeh_test.go (5) and storeh_internal_test.go (9) and noedit_alloc_test.go (1): 75 in all; none needs cgo.
• bash cgo_harness/docker/ccore_equality.sh <json|python|c|typescript> in the parity container: the equality tests plus the store tests CI runs (TestCcoreStoreChain for python, 300 edits, 31 steps compared with C; TestCcoreKeyword for typescript, 241 steps compared with C). A raised store counter fails TestCcoreDigest, TestCcoreRecords, TestCcoreIncremental, TestCcoreKeyword and TestCcoreStoreChain. TestCcoreDigest and TestCcoreIncremental also fail when a heap record is reached twice in a finished tree, and TestCcoreDigest fails when more than 10 percent of the records of the fresh parses of a CI input set (json, python, c, typescript) are unreachable. The race and stopped-parse gates accept only the stop-hook panic, only in a round that enables it.

Evidence (10 and 11 November 2026, Linux arm64 containers on a Mac, pinned C 0.27.1; the host is shared, so timing is directional)

All figures below are for the head of this branch after the review fixes, on main 24984a9c, unless a line names another base. Raw files for every group of numbers are in the asset store-evidence-37f19e5e.tar.gz of the evidence release; its README.md names the revision and the command of each group (folders timing, allocations, counters, equality, chains-and-gates, mutants, unit-and-race). The container runs (Linux arm64 on a shared Mac) used the pinned C 0.27.1 oracle; the host was shared, so timing is directional. Earlier figures from the base 685ef801 (the equality, balance, chain and memory numbers) are unchanged on the head before this revision. This revision changes findMulti for fresh parses, the page tables and the balance scratch (see Design); what ran on the final tree is listed under "Checks on the final tree" below. The other results in this section (counters, the 10,000-edit chains, the stopped and race gates, the three mutants) ran on the head before this revision and were not repeated on the final tree unless the list below says so.

• Linear reduction (added in main with reduce_linear.go) against the store: go test ./internal/ccore/ with the cross-check of the fast reduction switched on for every parser (so that each fast reduction is compared with the general walk while the store is in use) passes, also under -race: 2,646,395 fast reductions were cross-checked in that run. The equality tests with the check on (digest for json, go, python and typescript; trace for the same four; five incremental sessions; a 2,000-edit Python chain) cross-checked 10,004,563 more and all pass. The fast path creates its record through newNode, the call of the general path, so it follows the store's protocol: the claim before the child list, the limit checks and the allocation log (subtree.go:542-549, called from reduce_linear.go:54 and parser.go:195); it adds no retain or release, and findMulti and countFreshShared run on the finished tree whichever path built it (api.go:130, subtree.go:774); the trailing extras use the same buffer and the same order of putBuf and push as the general path (reduce_linear.go:53-71, parser.go:196-221). Main's TestLinearReductionMatchesGeneral found one defect of the store, in the general path: a node without children as the first node of a store indexed a child page that did not exist yet (kidSpan now returns an empty list for length 0). Four deliberate defects of the fast path (extras counted, subtrees not reversed, dynamic precedence dropped, fragile flag ignored) each fail the adapted test (it compares the records below the watermark, their reference words, the child list of each record, every head field, the stack graph and the free list). (asset folder unit-and-race, files unit-check*.log, and chains-and-gates).

• Deterministic counters of the existing engines: GOMAXPROCS=1 GOWORK=off go run ./cmd/perfcounterledger prints counter ledger passed: 412 rows, 206 languages on main 685ef801 and on the combined tree (the committed ledger file is identical in both and was not written). Work counters of the core, main against the combined tree, over the 52 equality inputs of the json, go, python, typescript and c sets: steps, tokens, maximum versions and multi-token counts are equal in all 52 (0 differences); only arena_bytes differs, as intended (for the 1 MiB Go input 70,804,256 on main against 57,794,512 with the store). Allocations per fresh parse of the 1 MiB inputs (3 iterations): Go 17 allocations and 185,594 B on main against 48 and 200,410 B with the store; JSON 10 and 8,186 B against 36 and 13,786 B. The additional allocations are the store's pages. (asset folder counters, files ledger-*.log and counter-diff.txt). Allocations per fresh parse of the 1 MiB inputs are in the asset folder allocations.

• Equality through the binding: 4,325 of 4,326 result lines equal to C (trace 1,197 of 1,198, with one known difference in a call of the shipped bash scanner port; digest 1,283 of 1,283; records 1,233 of 1,233; incremental steps 509 of 509; three one-byte edits at 1 MiB and a 100-step chain equal). The CI script passes for json, python, c and typescript on this head (failed=0 each; asset folder equality/ci). (asset folder equality).

• Balance step: over 1,792 finished trees (fresh parses of the equality inputs and edit-session steps) the parses made 6,081 internal nodes of byte size zero, every tree went through the pre-balance walk (1,792 walks), and the balance step refused 0 rewrites. The five-node graph that needs a refusal is a regression test that enters through the real balance step: the rewrite is refused, and an unchecked control rewrites the shared node. No heap record is reached twice in any of the 1,792 trees, zero-width ones included (checked on the finished tree). What the walk costs at 1 MiB is in the Performance section. (asset folder equality).

• Records left in the store by a fresh parse (no sweep): 8.0 percent for the 1 MiB Go file (645,061 allocated, 593,456 reachable), 0.001 percent for the 1 MiB JSON file, 1.1 percent for the Python CI sample (python_large_indent.py), 2.7 percent for the TypeScript CI sample. Over each CI input set the share is 0.001 percent (json), 2.1 (python), 0.15 (c) and 4.6 (typescript); the limit asserted is 10 percent. (asset folder allocations, files unreachable-*.log).

• Memory after 10,000 edits (a chain that keeps the source valid, tree edited in place and released after each reparse, against C's own chain; 41 sampled steps compared). Acceptance limit for this change: the store holds at most 1.5 times the bytes of a fresh store. Go file of 1 MiB: 59,244,544 bytes against 57,769,984, ratio 1.026. Generated Python file of 70 KB: 8,404,992 against 5,865,472, ratio 1.433. 41 of 41 sampled steps equal C in each chain; the three counters are 0 (rerun on this head; asset folder chains-and-gates).

• Other store checks: 1,000 stopped or panicking parses (843 stopped, 60 finished, 97 injected panics, each the exact stop-hook panic): no tree returned, store bytes 59,277,312 before and after, lease free, and a recovery parse with the interrupted parser and store equals C. Race run on a 140 KB Go and a 70 KB Python document, 300 rounds, two repeats each, with concurrent Digest calls: no race report, old trees unchanged, lease free; 257 to 277 completed writes and 36,752 and 34,010 reader walks (Python; Go 17,780 and 16,488) in the two runs of each; the run fails on any panic that is not the injected stop-hook panic. Two identical edit sequences give identical handles (401 handles, 3 runs). Heap keyword set: 241 of 241 steps equal C. Three deliberately broken builds of the incremental code fail 63, 72 and 72 Python session steps (trace comparison) and none in TypeScript or TSX (rerun on this head; asset folder mutants).

• Tests that fail without their fix (each run against a copy with the fix removed; the logs are kept outside the repository): readers swapping the parser's store (data race under -race), the root read through stale page-table headers, a copied view not keeping the cell reachable, a released tree keeping its pages, a panic that does not poison the store, a failure re-thrown from a drain, a waiting parse that does not recheck poisoning, a record that is not reserved before its child list, the edit allocator without its limit check (both checks removed), the balance step without the twice-reached check (through the real entry), the child-page, scanner-state-page and record limits checked outside the short mutex (three race tests), the first child page not checked, a record slot taken before its storage exists, a free-list index not checked, the edit latch armed late or released early (three variants), the release recheck of a waiting parse, a fresh tree whose twice-owned record is not counted (two tests), the watermark at page boundaries, and a waiting parse whose tree ends the store while it holds the lease (TestParseWithWhenStoreEndsWhileWaiting: it waits for the hook parse-wait before it releases the old tree, so it does not depend on scheduling; without the afterParse call it times out, with it the test passes 200 times under -race), and a replay of the goldens that returns no tree after acceptance (TestReduceLinearCheck now requires a tree for every golden that finishes; with a mutant that drops the finished root the old form passed and the new form fails). Not every new test is tied to one fix: TestParseAndEditAtTheChildPageLimitRace is a stress test of a parse and an edit at the child-page limit and passed against the child-page mutant. Gate mutants: a quiet exit-drain failure and a publication panic that becomes a quiet stop each make the stopped-parse gate and the race gate fail. go test -race -count=20 ./internal/ccore/ passes in the container (568.801 s in the archived log, Linux arm64, on the head before this revision) and on an amd64 host (423.200 s, the line ok github.com/odvcencio/gotreesitter/internal/ccore 423.200s of linux-amd64-checks.txt, attached to the evidence store-evidence-retention-8920a.tar.gz (sha256 0d860b0e0e3d63bba5b42bcbc97f94e6db82211f7bc812fa747c02f018c4011d)); the 380 s of an earlier revision of this text has no log and is withdrawn. Raw logs of the container runs: the asset store-evidence-37f19e5e.tar.gz of the evidence release, folders mutants and unit-and-race.

• Limit tests (8 in storeg_test.go, 6 more in storeh_*): record index in a fresh parse, reference count in publication (after the setup's queue is drained; the test asserts that the publication walk ran), reference count in the exit drain, record index in the edit allocator, child pages, scanner-state pages, a refused record stranding no child list, and limits lowered on one store not affecting another; a parse and an edit of separate trees racing for the last child page, state page or record; the first child page; a child-page limit of zero; a free-list index above the limit; and the record count at page boundaries.

• Counters: 0 in the 85 incremental, digest, records, keyword, stopped, race and chain lines. With a counter forced to 1, the three tests that assert counters fail with "store counters must be 0: sharedInTree 1, doubleFree 0, badRetain 0".

• Local checks: gofmt; go vet . ./internal/ccore/...; go test ./internal/ccore/... (also -race); go test ./cmd/citestplan ./cmd/v1layout ./cmd/v1guard; go run ./cmd/v1layout (680 of 680 root Go files, on a tree built on current main); go run ./cmd/v1guard; go run ./cmd/citestplan and -stale-run-names; go list -deps . shows neither ccoretest nor bindtest. Root cross builds: linux/amd64, windows/amd64, darwin/amd64, darwin/arm64, js/wasm, wasip1/wasm, linux/386. Staticcheck 0.6.1 with the CI checks (SA4000,SA4003,SA5011,SA4006,SA4009,SA1019) reports nothing on ./internal/ccore/....

Checks on the final tree

The checks below ran on the tree that this revision produces (PR head a15b7239 plus fix-vs-pr-head.diff; the diff also applies on d2fd2bb1, which differs from a15b7239 only in docs/v1-design.md).

check	result
gofmt -l on the changed files (Linux amd64 host)	no files listed (run on the same diff, earlier; no exit code recorded)
go vet ./internal/ccore/... (Linux amd64 host)	no output (run on the same diff, earlier; no exit code recorded)
staticcheck 0.6.1 with CI's checks and package list (Linux amd64 host)	no output (run on the same diff, earlier; no exit code recorded)
go run ./cmd/v1guard (Linux amd64 host)	R6 guardrails line, no finding (run on the same diff, earlier; no exit code recorded)
go run ./cmd/v1layout (Linux amd64 host)	L0 guardrails: 680/680 root Go files (run on the same diff, earlier; no exit code recorded)
go test ./internal/ccore/... -count=1 (Linux amd64 host)	4 packages ok (run on the same diff, earlier; no exit code recorded)
go test -race -count=3 ./internal/ccore/... (Linux amd64 host)	not run
go test -race -count=20 ./internal/ccore/ (Linux amd64 host)	ok, 423.200 s (run on the same diff, earlier; no exit code recorded)
bash cgo_harness/docker/ccore_equality.sh json (arm64 container)	SUMMARY language=json trace_lines=191 digest_lines=192 record_lines=192 incremental_lines=73 failed=0
bash cgo_harness/docker/ccore_equality.sh python (arm64 container)	SUMMARY language=python trace_lines=82 digest_lines=109 record_lines=109 incremental_lines=73 failed=0
bash cgo_harness/docker/ccore_equality.sh c (arm64 container)	SUMMARY language=c trace_lines=77 digest_lines=80 record_lines=80 incremental_lines=0 failed=0
bash cgo_harness/docker/ccore_equality.sh typescript (arm64 container)	SUMMARY language=typescript trace_lines=73 digest_lines=78 record_lines=78 incremental_lines=73 failed=0
1,000-edit chain, 1 MiB Go, TestCcoreBigChain (CCORE_HEAVY=1) (arm64 container)	PASS
300-edit Go chain, TestCcoreStoreChain (arm64 container)	PASS
TestCcoreKeyword, typescript (arm64 container)	PASS
TestCcoreStopped (arm64 container)	PASS
TestCcoreHandles (arm64 container)	PASS
incremental mutant m1 (typescript, tsx, python edit sessions; must fail)	not run (the job reached its time limit)
incremental mutant m2 (typescript, tsx, python edit sessions; must fail)	not run (the job reached its time limit)
incremental mutant m3 (typescript, tsx, python edit sessions; must fail)	not run (the job reached its time limit)

Not run on the final tree: the 10,000-edit chains, the container race runs (go test -race -count=20 and the race gate of the equality harness), the work and reuse counters against main (46 rows), the three mutants of the incremental code (the container job reached its time limit before them), and the equality inputs of the other languages. The change touches the walk for records reached twice, the page tables and the balance scratch; the equality script above is the check of the parse results through them, and the store's unit tests (with the new marks_internal_test.go) are the check of the walk.

Checks of the retention fix

The first revision left the inline page-table arrays of the store core populated after the store ended; a reachable parser or released tree then kept the pages alive after the pool dropped them. The fix and its checks (Linux amd64 host, one check per run; logs in the evidence folder store-evidence-retention-8920a.tar.gz (sha256 0d860b0e0e3d63bba5b42bcbc97f94e6db82211f7bc812fa747c02f018c4011d)):

check	result
TestReleasedTreeHoldsNoPage without the fix (must fail)	FAIL: storefix_test.go:1120: the core of an ended store still holds 3 page pointers in its inline page tables
`go test ./internal/ccore/ -run 'TestReleased	TestRelease
go test ./internal/ccore/... -count=1 with the fix	exit 0, ok (1.843s, 0.056s, 2.303s, 0.004s)
go vet ./internal/ccore/...	exit 0
staticcheck 0.6.1 with CI's checks and package list	exit 0
go test -race -count=3 ./internal/ccore/...	exit 0, ok (61.797s, 1.157s, 24.307s, 1.046s)
gofmt -l on the changed files	no files listed

What else keeps memory after Release or after the store ends, checked in the code and by the test: the page tables and inline arrays (cleared), the pending and spare operation queues (set to nil at the end), the free-list heads (numbers), the lease holder's lists (cleared at the end; before this revision they stayed with the core), the parser's view of the page tables (cleared at the end of every parse), the parser's scratch bit set for the walk (reachBits: bits, no pointer, at most 2 MiB), the compress stack kept on the parser (compressTmp: handles, as long as the deepest balance, no pointer to a page), and the size hint (two numbers).

Counters

counter	result
sharedInTree	0 in the 85 incremental, digest, records, keyword, stopped, race and chain lines
doubleFree	0 everywhere
badRetain	0 everywhere
work counters of the existing engines	identical on main 685ef801 and on the tree with this change (counter ledger: 412 rows, 206 languages, counter ledger passed on both; not rerun on 24984a9c, where the diff touches no file that the engines use); nothing calls the package

Core work and reuse counters, main 24984a9c against this head, same inputs and same C oracle (asset folder counters). The counters are the ones the equality harness prints: reused subtrees and reused bytes (reuse; they must not fall by more than 2 percent), external-scanner calls, tokens lexed and reductions (work; they must not rise by more than 2 percent), and steps and tokens of the fresh parses. Every row changes by 0.00 percent, so no row meets the 2 percent threshold; 46 rows checked, 0 outside it.

session (steps)	reused subtrees, main / head	reused bytes, main / head	external-scanner calls (lex_external), main / head	steps that differ
json-session (73)	3,482 / 3,482	137,098 / 137,098	0 / 0	0
python-session (73)	5,331 / 5,331	92,576 / 92,576	20,369 / 20,369	0
typescript-session (73)	2,243 / 2,243	108,724 / 108,724	9,308 / 9,308	0
tsx-session (73)	2,798 / 2,798	147,606 / 147,606	2,030 / 2,030	0
go-session (73)	7,717 / 7,717	106,388 / 106,388	0 / 0	0
go-generic (144)	1,536 / 1,536	4,080 / 4,080	0 / 0	0

one-byte edit in the 1 MiB Go file	reused, main / head	reused bytes, main / head	tokens lexed, main / head	reductions, main / head
mid_524340	12,914 / 12,914	1,048,584 / 1,048,584	2 / 2	12,914 / 12,914
start	12,907 / 12,907	1,047,973 / 1,047,973	297 / 297	13,596 / 13,596
end	12,907 / 12,907	1,047,925 / 1,047,925	297 / 297	13,596 / 13,596

fresh parses of the equality inputs	inputs	steps, main / head	tokens, main / head
json	2	316,070 / 316,070	182,171 / 182,171
python	28	101,687 / 101,687	42,326 / 42,326
c	4	39,807 / 39,807	16,176 / 16,176
typescript	6	7,827 / 7,827	3,283 / 3,283
go	13	1,416,026 / 1,416,026	596,504 / 596,504

The two 10,000-edit chains (Go 1 MiB, Python 70 KB) exist only for the store (TestCcoreStoreChain), so there is no main arm for them. On this head the core's reuse counters equal C's in all 41 sampled steps of each chain (reused subtrees, tokens lexed and reductions: 0 fields differ; Python sums over the 41 steps 246,134 reused, 286,154 lexed, 634,052 reductions; Go 529,204, 81 and 529,145), and main equals C on the sessions above. A 100-step chain on the 1 MiB Go file exists on both sides: 100 of 100 steps equal a fresh parse and 2 of 2 checks equal C on main and on this head; the memory held at the end is 317,953,904 B on main against 61,095,936 B on this head.
The paired editor-latency job is not triggered by this PR: the scope rule answers false (only internal/ccore, its test harness and documentation change).

Risk

• The invariant gate checks the public parse API; this PR adds a paragraph to docs/v1-design.md ("Invariant gate") that says so and names the test below. No public parse reaches this package, so the gate has no language to run for here. The package's own figure: ParseWith on an unedited tree allocates 2 objects per call (the number includes the Release of the result), for a 45-byte input and for a 1 MiB input alike (TestNoEditReparseAllocations, testing.AllocsPerRun, bound 2, also under -race; asset folder allocations). An allocation profile of 1,000 reparses shows exactly two sites: the new Tree value (api.go) and its cell, which carries the finalizer (tree.go, newCell). The release queue and the mark list of findMulti allocate nothing in steady state. Before this revision the count was 3: an empty drain dropped the spare queue buffer, so the next Release allocated one (lifetime.go; the drain now keeps the buffer, with no change to the order of the protocol's steps; the test fails at the old count). Removing the last two means returning the same tree when nothing changed, which changes who owns the root and who releases it; this PR does not change the ownership protocol.

• A fresh parse leaves up to 8 percent of its records in the store for the store's life (1 MiB Go file; see the measured figures above); the stored size after 10,000 edits is 1.026 and 1.433 times a fresh store.

• Performance against main (Performance section): allocations per fresh parse are 7 for 1 MiB of Go and of JSON (arena 17 and 10); the edit rows with the release counted (+0.7 to +7.3 percent), their 99th percentile and the resident memory of a fresh parse are inside their limits. Fresh-parse time and Go/C are inside the 10 percent limit when GODEBUG=disablethp=1 is set and not shown to be inside it in the default environment of the benchmark container (JSON Go/C +10.5 and +11.1 percent; Go time rows +8.8 to +11.4 percent), where every process takes one of two speed levels and this revision draws the slow one in more Go processes than the head before it. The cause of the two levels is not found.

• A logger or scanner callback must not parse the same document or call StoreInfo or ArenaDigest: the lease is not re-entrant.

• Edit does not drain (except when a Release overlapped it). Edits with no parse, release or drop in between grow the queue and the edit pages.

• Memory with trees dropped without Release (finalizer only) was not measured.

• Reader contract (above): a reader that runs while its own Tree value is edited or released is not protected, and a view and the slices it returned are valid only until the owning tree is edited or released. A released tree gives an empty store whose accessors that index a page panic; accessors that decode an inline handle, and ExtState(0), do not.

• A handle is reused after its record is released. The counters do not see a stale handle once its record slot is reused.

• A poisoned store keeps its pages while any of its trees lives, and the records it leaked stay in it until then. A failure inside Edit is re-thrown to the caller of Edit after the store is poisoned.

• The store keeps pages in a pool after a store ends (at most 128 MiB; pages idle for 30 s are dropped).

• No package calls internal/ccore yet; the risk to users is zero today.

Not verified

• GitHub CI: the two CI store tests and the script's self-test run there for the first time.

• Native Linux amd64 and macOS runs of the equality and gate tests; they ran in Linux arm64 containers. Timing is directional.

• The invariant gate (incremental equals fresh, an ERROR root implies HasError, the root covers the input, a reparse without edits allocates nothing) was not run: no public parse reaches the package, so there is no language to run it for, and the box in the checklist stays unticked. The package's own reparse of an unedited tree allocates 2 objects, pinned by TestNoEditReparseAllocations (above).

• The core's reuse counters over the two 10,000-edit chains were compared with C's, not with main (main has no such chain).

• Timing was measured on a shared host in an arm64 container (directional), with a loop that applies the flags of the AGENTS.md procedure to three revisions and not with scripts/run_randomized_benchmarks.sh itself. Native Linux amd64 and macOS timings were not measured. Each iteration of BenchmarkCcoreBigEditCycle adds two observations of the core, and a process runs 21 to 34 iterations per workload (360 process outputs in the evidence folder), so a process has 42 to 68 observations. The function that reports the 99th percentile returns the largest observation for fewer than 100 observations, so the "99th percentile" in this text is the largest observation of one process, and the tables give the median of that maximum over 20 seeds. A tail claim with a real 99th percentile needs at least 100 observations per process; none was collected.

• The cause of the two speed levels per process, and whether they appear on a host that is not a Linux VM on Apple silicon.

• Peak resident memory of the 1,000-edit chain on this revision (the fresh-parse figure is measured).

• The default-environment fresh-parse row of the Performance section is above its limit or not shown to be inside it (see there); an exception for it is the repository owner's to give.

• The checks that the "Checks on the final tree" subsection lists as not run.

• Resident memory was measured only at 1 MiB and as the whole process, with no empty-process baseline; larger inputs and out-of-memory behaviour were not checked.

• Memory that only a finalizer frees (trees dropped without Release) was not measured.

• A mutation campaign over the store code; only three deliberately broken builds of the incremental code and the mutants named above were run.

• Whether any grammar other than TypeScript and TSX hands out a cached heap token twice, and whether any shipped input makes the balance step compress through a record with two parents (the pre-balance walk runs for every tree that is balanced, whatever the size of its nodes; the double-reach check runs on the finished tree).

• Races or stopped parses on a grammar with a TypeScript-like zero-width token (the race and stopped runs use Go and Python).

• In the container, go test ./internal/ccore/... with the private Go table of the harness fails TestEveryRegisteredLanguageBinds (that table registers the Go scanner, which this test of main expects to be unregistered); the same package passes on an amd64 host with the shipped tables, and the test does not touch this change.

• go vet of cgo_harness still reports copy-of-lock notes in older files (none in ccore_* files).

• Focused package or unit tests ran for the touched behavior

• Affected parity validation ran in Docker, one language or grammar at a time

• If grammargen or parity logic changed, ran the smallest relevant focus target in Docker

• Documented anything intentionally left unvalidated in this PR

• Engine PR: before-and-after deterministic counters are in the PR body (work counters did not rise, reuse counters did not fall)

• Invariant gate ran for each language the PR can affect (incremental equals fresh, ERROR root implies HasError, root covers input, no-edit reparse allocates nothing)

• No language-name comparison in engine files (D6); one v1 lane only

Performance

The harness correction

The benchmark campaigns reported in the earlier revisions of this PR ran their child workloads in a fixed order in every process: BenchmarkCcore and BenchmarkCcoreBigEdit are single top-level benchmarks, and the shuffle of scripts/run_randomized_benchmarks.sh only reorders top-level benchmarks. They were not randomized comparisons and are withdrawn; none of their numbers is used below. This revision changes cgo_harness:

• The child workloads of BenchmarkCcore and BenchmarkCcoreBigEdit run through internal/benchorder in an order set by GTS_BENCHMARK_SEED (the driver sets it once per process). The order is the same on every revision for one seed and differs between seeds. In the runs below the three revisions emitted the same order of workloads for each of the 20 seeds, and the 20 seeds gave 20 different orders for the fresh-parse workloads (order-check.txt in the evidence folder).
• A new arm paired of BenchmarkCcore times the core and C in Go-C-C-Go cycles in one process (each parse timed alone) and reports go-ns/op, c-ns/op and go-over-c; the Go/C ratio below is this metric. The arms spike and c stay as they were.
• A new BenchmarkCcoreBigEditCycle times edit plus reparse of the three one-byte edits in the 1 MiB Go file in Go-C-C-Go cycles, per iteration. It reports the mean and the 99th percentile of the time per iteration, with and without the release of the old trees (the arena has nothing to release; C frees inside Close, which is counted with its run), and go-over-c. A process runs 21 to 34 iterations per workload and each adds two core observations, so there are 42 to 68 observations; for fewer than 100 observations the reported 99th percentile is the largest observation of the process, and the table gives the median of that maximum over the 20 seeds.
• The edits are in cgo_harness/ccore_c_test.go, cgo_harness/ccore_incremental_big_c_test.go and the new cgo_harness/ccore_bench_seed_c_test.go. The same edits were applied to main 24984a9c for the arena arm (the release calls go through helpers that do nothing on a revision without a release).

To run it: CCORE_LANG=go CCORE_ARMS=spike,paired GTS_BENCHMARK_SEED=<n> GOMAXPROCS=1 <test binary> -test.run '^$' -test.bench '^BenchmarkCcore$' -test.benchtime 750ms -test.benchmem (build with go test -c -tags 'cgo treesitter_c_parity' in cgo_harness); CCORE_LANG=json for JSON; -test.bench '^BenchmarkCcoreBigEdit$/_core$' and -test.bench '^BenchmarkCcoreBigEditCycle$' for the edits.

Conditions of the runs

Three revisions: main 24984a9c (the arena), this PR before this revision (a15b7239), and this revision. One process per seed and revision, GOMAXPROCS=1, -count=1, -benchtime=750ms, -benchmem, 20 seeds (1 to 20), the same seed for the three revisions, the revisions in rotating order per seed; the loop applies the flags of the AGENTS.md procedure but is not scripts/run_randomized_benchmarks.sh itself (that script runs two revisions; this loop runs three). One Linux arm64 container (4 CPUs, 8 GiB) on a shared Mac, Go 1.26.3, pinned C 0.27.1, the private Go table of the harness. Comparisons are medians of 20 processes; benchstat output for every table is in the evidence folder. Timing on this host is directional. The raw output of every process, the benchstat text, the order check and the container sampling are in the evidence folder (store-evidence-fresh-06b63.tar.gz, sha256 cc1ba80ca76df400f6382c9c807304f31f6b70ad38e0e7aa0c391adaa0e69c0f).

Two environments, because of the finding below: default (nothing set) and stable (GODEBUG=disablethp=1). Other containers on the Mac while the runs went: 0 to 2 at the sample taken a minute after the start (docker ps and docker stats before, during and after each run are in the evidence folder); the load average of the Mac at the start was 4 to 17.

A finding: two speed levels per process

In the default environment the time of a fresh parse takes one of two levels per process, for the arena as well as for the store. The level is fixed for the life of the process and drawn again for each process. Fresh parse of 1 MiB Go, 20 processes per revision (slow level: 108 ms or more):

revision	processes at the fast level	median, fast	processes at the slow level	median, slow
arena (main)	17	101.8 ms	3	113.8 ms
PR head before this revision	15	97.2 ms	5	114.2 ms
this revision	6	98.0 ms	14	112.8 ms

The levels are the same for the three revisions (fast about 97 to 102 ms, slow about 113 ms); what differs is how many processes draw the slow level. The same holds at 137 KiB (fast 12.4 to 13.3 ms, slow 14.8 to 15.0 ms; slow in 3, 6 and 14 of 20 processes) and 32 KiB. For the 2 KiB JSON sample the levels are 47.5 and 58 us; slow in 0 (arena), 13 (before) and 9 (this revision) of 20 processes. For the 1 MiB JSON file the levels are 24.3 and 26.8 to 30 ms; slow in 4, 13 and 11 of 20 processes. A 20-seed median of this sample therefore moves by 10 to 20 percent with the number of processes that drew the slow level. With GODEBUG=disablethp=1 or with GOGC=off the step is gone for all three revisions (spread 1 to 2 percent; probe runs in the evidence folder). The cause is not found. A CPU profile of a slow and a fast process of the same binary differs in the time of the record write; a plain Go loop that writes 72-byte records into pages is not bimodal on this host; the heap's AnonHugePages does not differ between the levels; allocating pages in slabs of 16 did not change it; removing the finalizer did not change it.

Ratchet rows

Limit: 10 percent above the arena's 20-seed median (allocations per KiB, Go/C, incremental latency) and peak resident memory not above the arena's. Each value is the median over 20 seeds; the first number under "after" is the default environment, the second the stable one.

row	arena	before this revision	after this revision	limit	verdict
allocations per fresh parse, 1 MiB Go	17	48	7	18	met
allocations per fresh parse, 1 MiB JSON	10	36	7	11	met
allocations per fresh parse, 137 KiB Go / 32 KiB Go / JSON sample	12 / 9 / 5	35 / 26 / 12	7 / 7 / 4	13 / 9 / 5	met
Go/C, fresh Go 32 KiB / 137 KiB / 1 MiB, default environment	.524 / .513 / .507	.539 / .491 / .493	.575 / .543 / .507 (+9.7 / +5.7 / -0.1 percent)	+10 percent	met by 0.3 points at 32 KiB; at the line
Go/C, fresh JSON sample / 1 MiB, default environment	.553 / .558	.694 / .696	.615 / .616 (+11.1 / +10.5 percent)	+10 percent	not met (above by 0.5 to 1.1 points; before this revision +25 percent)
Go/C, fresh Go 32 KiB / 137 KiB / 1 MiB, stable environment	.551 / .560 / .575	.550 / .550 / .579	.547 / .553 / .569 (-0.6 / -1.4 / -1.1 percent)	+10 percent	met
Go/C, fresh JSON sample / 1 MiB, stable environment	.587 / .615	.592 / .613	.589 / .604 (+0.4 / -1.7 percent)	+10 percent	met
fresh-parse time (not a ratchet row; the Go/C row uses the C arm of the same process), default environment	see the tables below		Go +11.4 / +9.2 / +8.8, JSON +11.6 / +10.9 percent		at or above the line; the spread of the slow level (above)
edit and reparse with the release of the old trees counted, 1 MiB Go, default environment	7.15 / 6.77 / 7.19 ms (end / middle / start)	7.34 / 7.11 / 7.59	7.59 / 7.12 / 7.72 (+6.1 / +5.2 / +7.3 percent)	+10 percent	met
edit and reparse with the release counted, stable environment	6.93 / 6.79 / 6.98 ms	7.23 / 7.01 / 7.21	7.14 / 6.84 / 7.19 (+2.9 / +0.7 / +3.1 percent)	+10 percent	met
largest observation per process of edit and reparse with the release counted (the reported "99th percentile" of 42 to 68 observations; end / middle / start; default, then stable)	17.6 / 19.0 / 16.1, then 23.5 / 23.8 / 24.8 ms	8.9 / 8.8 / 9.0, then 9.0 / 8.8 / 9.1	9.2 / 8.6 / 9.3, then 9.1 / 8.7 / 9.0	+10 percent	met (about half of the arena's: the arena's collector pauses)
peak resident memory, fresh parse of 1 MiB Go (whole process, ru_maxrss)	749,172 / 750,176 / 753,824 KiB	569,056 / 579,308 / 615,916 KiB	579,528 / 589,540 / 599,208 KiB	not above the arena	met (-21 percent, median)
peak resident memory, 1,000-edit chain	565,560 / 608,900 / 604,892 KiB	565,836 / 587,080 / 566,236 KiB (from the earlier revision of this PR, same procedure)	not measured on this revision	not above the arena	not verified for this revision

Reading the table: allocations, the edit rows, the per-process maximum of the edit time (not a true 99th percentile, see above) and the resident memory are inside their limits in both environments. Go/C is inside its limit in the stable environment for every input. In the default environment it is inside for Go (32 KiB at the line) and above the line for both JSON inputs by about one point, and the time of the fresh Go rows is 8.8 to 11.4 percent above the arena's median; the step between the two levels (about 15 percent) is larger than these differences, and this revision draws the slow level in more processes than the head before it for Go (14 against 5 of 20 at 1 MiB) and in fewer for JSON (9 against 13). I did not find why. The row is open for the default environment; the stable environment is the measurement in which the difference between the revisions is smaller than the noise of the run.

Fresh parse, tables

Median time per fresh parse in ms (spike rows; paired rows are one Go parse and two C parses and are not shown), medians of 20 processes.

Default environment:

Default environment, Go:

workload	arena	PR head	this change	head vs arena	this change vs arena
go_generated_137K_spike	13.358	12.837	14.584	-3.9%	+9.2%
go_generated_1M_spike	103.277	98.208	112.400	-4.9%	+8.8%
go_generated_32K_spike	3.119	3.009	3.474	-3.5%	+11.4%

Default environment, JSON (ms):

workload	arena	PR head	this change	head vs arena	this change vs arena
json_1M_spike	24.508	30.121	27.171	+22.9%	+10.9%
json_sample_spike (0.047 / 0.059 / 0.053 ms; 47.5 / 58.5 / 53.0 us)	0.047	0.059	0.053	+23.3%	+11.6%

Stable environment (GODEBUG=disablethp=1), Go:

workload	arena	PR head	this change	head vs arena	this change vs arena
go_generated_137K_spike	12.662	12.686	12.585	+0.2%	-0.6%
go_generated_1M_spike	97.518	100.083	99.226	+2.6%	+1.8%
go_generated_32K_spike	2.936	2.979	2.979	+1.5%	+1.4%

Stable environment, JSON (ms; the sample is 0.0469 / 0.0482 / 0.0481 ms):

workload	arena	PR head	this change	head vs arena	this change vs arena
json_1M_spike	24.812	25.032	24.698	+0.9%	-0.5%
json_sample_spike	0.047	0.048	0.048	+2.9%	+2.7%

Go time over C time, from the paired arm (Go-C-C-Go cycles in one process); default environment, then stable:

workload	arena	PR head	this change	head vs arena	this change vs arena
go_generated_137K_paired	0.513	0.491	0.543	-4.4%	+5.7%
go_generated_1M_paired	0.507	0.493	0.507	-2.7%	-0.1%
go_generated_32K_paired	0.524	0.539	0.575	+2.9%	+9.7%

workload	arena	PR head	this change	head vs arena	this change vs arena
json_1M_paired	0.558	0.696	0.616	+24.8%	+10.5%
json_sample_paired	0.553	0.694	0.615	+25.5%	+11.1%

workload	arena	PR head	this change	head vs arena	this change vs arena
go_generated_137K_paired	0.560	0.550	0.553	-1.9%	-1.4%
go_generated_1M_paired	0.575	0.579	0.569	+0.7%	-1.1%
go_generated_32K_paired	0.551	0.550	0.547	-0.1%	-0.6%

workload	arena	PR head	this change	head vs arena	this change vs arena
json_1M_paired	0.615	0.613	0.604	-0.3%	-1.7%
json_sample_paired	0.587	0.592	0.589	+1.0%	+0.4%

Allocations per fresh parse (the spike rows are the parse alone; the paired rows include C's binding allocations):

workload	arena	PR head	this change	head vs arena	this change vs arena
go_generated_137K_spike	12	35	7	+191.7%	-41.7%
go_generated_1M_spike	17	48	7	+182.4%	-58.8%
go_generated_32K_spike	9	26	7	+188.9%	-22.2%

workload	arena	PR head	this change	head vs arena	this change vs arena
json_1M_spike	10	36	7	+260.0%	-30.0%
json_sample_spike	5	12	4	+140.0%	-20.0%

Bytes per fresh parse, 1 MiB Go: 185,592 (arena), 200,408 (head), 8,312 (this revision); the page tables, the store core, the cell and the tree are what remains; the pages come from the pool.

Edit and reparse, tables

One-byte edit and reparse in the 1 MiB Go file, ms per iteration. CcoreBigEdit (edit plus reparse, release outside the timed region), default environment:

workload	arena	PR head	this change	head vs arena	this change vs arena
edit end_core	5.955	6.129	6.131	+2.9%	+2.9%
edit mid_524340_core	5.626	5.749	6.044	+2.2%	+7.4%
edit start_core	5.793	6.442	6.180	+11.2%	+6.7%

CcoreBigEditCycle, with the release of the old trees counted (the median of the mean over 20 seeds), default environment, then stable:

workload	arena	PR head	this change	head vs arena	this change vs arena
edit end	7.153	7.338	7.587	+2.6%	+6.1%
edit mid_524340	6.771	7.108	7.120	+5.0%	+5.2%
edit start	7.193	7.586	7.721	+5.5%	+7.3%

workload	arena	PR head	this change	head vs arena	this change vs arena
edit end	6.933	7.230	7.135	+4.3%	+2.9%
edit mid_524340	6.793	7.007	6.842	+3.2%	+0.7%
edit start	6.979	7.208	7.193	+3.3%	+3.1%

Largest observation per process of the same (reported as the 99th percentile; 42 to 68 observations), default, then stable:

workload	arena	PR head	this change	head vs arena	this change vs arena
edit end	17.586	8.938	9.199	-49.2%	-47.7%
edit mid_524340	19.005	8.757	8.641	-53.9%	-54.5%
edit start	16.094	8.954	9.289	-44.4%	-42.3%

workload	arena	PR head	this change	head vs arena	this change vs arena
edit end	23.524	9.017	9.097	-61.7%	-61.3%
edit mid_524340	23.755	8.791	8.674	-63.0%	-63.5%
edit start	24.843	9.133	9.033	-63.2%	-63.6%

The arena's 99th percentile is two to three times its mean because it allocates about 15 MiB per iteration (the copy of the tree) and the collector pauses show in the tail; the store allocates 1 to 2 MiB.

Peak memory

Maximum resident set size of a process (resource.getrusage of the child), three repetitions each, same container, fresh parse of the 1 MiB Go input (BenchmarkCcore, go_generated_1M_spike, 20 iterations): arena 749,172 / 750,176 / 753,824 KiB; the head before this revision 569,056 / 579,308 / 615,916; this revision 579,528 / 589,540 / 599,208 (-21 percent against the arena, by medians). These are whole-process figures with no empty-process baseline, as before. The 1,000-edit chain was not measured on this revision (the earlier figures above are for the head before it).

• If perf-relevant, compared before and after with scripts/run_randomized_benchmarks.sh (GOMAXPROCS=1, -count=1 per process, 20 shuffle seeds, -benchtime=750ms, -benchmem); a fixed-order run is not evidence
• Included before and after numbers, or explicitly said perf is not the goal of this PR
• If large-grammar behavior changed, checked bounded RSS, timeout, or OOM behavior under Docker

Maintainability

• No unnecessary abstractions or speculative cleanup outside the PR's scope
• Generated files or harness changes are only here because they are required by the change
• No new dependencies without justification

Self-review

Review your own diff before requesting review. Walk through it as if you're seeing it for the first time, and keep any self-review comments concise and focused on the non-obvious parts.

• Reviewed my own diff end to end
• Left comments on notable sections or the crux of the solution
• Called out anything I'm unsure about or want a second opinion on

Due diligence

• Read the code you're changing before changing it
• Checked that similar patterns exist elsewhere in the codebase and stayed consistent
• If touching the parser or lexer: validated against diverse affected grammars
• If adding a grammar or scanner: verified against upstream C output