Skip to content

Chunk pack: combine a carried run once, when emitted - #902

Open
frankmcsherry wants to merge 1 commit into
master-nextfrom
chunk-pack-once
Open

frankmcsherry wants to merge 1 commit into
master-nextfrom
chunk-pack-once

Conversation

@frankmcsherry

Copy link
Copy Markdown
Member

pack (the maximal-packing helper behind Chunk::settle) grew its carry with combine(&mut acc, next). That is linear only when appending leaves acc in place, as Vec does. Corgi's combine rebuilds its columns, so a carry absorbing many small chunks recopied itself every time. Over scc's churn rounds that was 30.7M carried rows recopied to append 2.3M new ones (counted).

Now the carry is a run of chunks, and combine(run) is called once, when the run is emitted. Between calls the run goes back to the input uncombined. The three implementations change to match: vec concatenates, columnar melds each trie onto the first, and corgi calls its existing concat. The peel-an-oversized-chunk path is folded into the loop, so pack_absorb goes. 46 lines added, 71 removed.

One trade: a run handed back between calls is re-walked by the next call. That costs one deque pop and push per chunk, not a copy of its rows.

Churn, ms/round, corgi, 3 runs each (loads unchanged):

case master-next this
count 3.265–3.278 3.187–3.190
count2 (churn 10k) 22.87–22.90 22.38–22.44
min 2.574–2.578 2.523–2.540
reach 69.61–69.78 68.95–68.99
scc 33.19–33.35 32.87–33.11

Validation: DD lib tests plus the int_proxy, columnar, reduce_oracle, trace and scc tests pass, as do all interactive tests. AoC 2023 passes 33/33 on both vec and corgi. columnar_spill spills the same 5425 chunks at the same rate.

🤖 Generated with Claude Code

`pack` grew its carry by `combine(&mut acc, next)`, which is linear only if
appending leaves `acc` in place. Corgi's combine rebuilds its columns, so a
carry absorbing many small chunks recopied itself each time: over scc's churn
rounds it recopied 30.7M carried rows to append 2.3M new ones. The carry is
now a run of chunks, combined once when emitted; between calls the run goes
back to the input uncombined.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ABLpEcZUPN1oAUY2usfkx

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant