Skip to content

Declare dataset partition counts as unknown - #202

Merged
quinnj merged 1 commit into
masterfrom
maintenance/parquet-partition-iterator
Oct 4, 2026
Merged

quinnj merged 1 commit into
masterfrom
maintenance/parquet-partition-iterator

Conversation

@quinnj

@quinnj quinnj commented Oct 4, 2026 •

Copy link
Copy Markdown
Member

Calling collect(Tables.partitions(dataset)) fails with a MethodError for length(::Parquet.DatasetPartitions). This iterator walks and filters files lazily, but inherits Julia's default HasLength contract without implementing length.

Declare its size as unknown, as specified by the Julia iteration interface. Keep the existing file traversal and filtering behavior. Add public collection tests through both Dataset and read_parquet, covering all, one, and no selected data files, exclusion of metadata files, and decoded row values. The separate table partition iterator retains its existing length contract.

Validation: the unchanged reader passes six manual-iteration controls and fails six collection controls on both actual Julia 1.3.1 and 1.13.1. All 30 focused checks pass after this fix. The complete suites pass with four threads and bounds checking: 4,092 checks on Julia 1.3.1 and 4,078 on Julia 1.13.1. An independent agent privately reviewed the complete source/test diff and found no blocker. git diff --check passes.

Hosted validation at 8d621a9b1304822a638a403d629dbda6184e4f42: all nine Julia 1.3/current/nightly jobs on Linux, macOS, and Windows and both coverage checks pass. Each uploader log confirms acceptance for this repository and commit; all nine reports are processed at 86.62%.

Merged validation at 426c4a7ba8c3c51d0c424c0eb3cec8f00ad913aa: all nine merge jobs pass, with all nine correct-repository uploads accepted and processed at 86.62%. The merged tree matches the tested tree.

Co-authored by Codex

AI disclosure: This work was prepared with assistance from OpenAI Codex.

Allow collect to iterate filtered dataset partitions instead of requiring an undefined length. Thirty public collection controls pass on actual Julia1.3.1/1.13.1, covering both dataset constructors and all, one or no selected files.

Co-authored by Codex

AI disclosure: This work was prepared with assistance from OpenAI Codex.
@codecov

codecov Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.62%. Comparing base (7824e2e) to head (8d621a9).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #202   +/-   ##
=======================================
  Coverage   86.61%   86.62%           
=======================================
  Files          10       10           
  Lines        1599     1600    +1     
=======================================
+ Hits         1385     1386    +1     
  Misses        214      214           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@quinnj
quinnj merged commit 426c4a7 into master Oct 4, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant