fix: icebug disk can now read zstd + dictionary RLE compressed parquet - #821
Merged
Conversation
ScanRelTable::getNextTuplesInternal pulls bound nodes in batches of DEFAULT_VECTOR_CAPACITY and IceDiskRelTable::initScanState re-initialized the parquet scan for every batch, which reset the scan to row group 0 each time. The indices file was therefore re-scanned once per bound-node batch (O(batches x edges)) and a full-materialization query effectively never terminated (e.g. wiki-Talk: ~1169 batches x 5M edges). Add range-limited scan support to ParquetReader (initializeScan skipRows and numRows; rowsToSkip/rowsRemaining state) and use it in the FWD CSR path to scan only the rows [indptr[minNode], indptr[maxNode+1]). Queued skips are applied via StructColumnReader::skip()/applyPendingSkips(), which also now sizes the dummy definition/repetition buffers to avoid writing into unallocated storage.
scanInternal() returned true after switching to the next row group without updating the result's selection vector size. Callers that scan multiple row groups in a single initializeScan() (ice-disk rel-table scans, indptr loading, and the dump harness) therefore saw the previous chunk's stale selSize and re-processed the previous chunk's data: values diverged at every row-group boundary and the scan over-read (2,394,386 -> 2,398,482 rows, i.e. 2x2048 extra rows for the indptr file). Set the selection size to 0 on the row-group switch path so callers correctly treat the switch as producing no rows. Also drop the debug fprintf statements added while investigating.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes: #820