Skip to content

Preserve typed columns for empty datasets - #204

Merged
quinnj merged 1 commit into
masterfrom
maintenance/parquet-empty-dataset
Oct 4, 2026
Merged

quinnj merged 1 commit into
masterfrom
maintenance/parquet-empty-dataset

Conversation

@quinnj

@quinnj quinnj commented Oct 4, 2026 •

Copy link
Copy Markdown
Member

Reading a dataset whose filter excludes every file, or a directory containing only schema metadata, currently throws BoundsError when accessing a column or converting through Tables.columntable or Tables.rowtable. The schema is valid, but the loader never initializes its columns.

Initialize an empty vector for each declared column when no partitions are selected, matching the single-file reader's existing behavior. Preserve the discovered schema and logical-type overrides, and document that a schema must still be available from metadata or an existing data file.

Validation: the unchanged implementation passes 21 controls and errors on 30 public empty-dataset operations on both actual Julia 1.3.1 and 1.13.1. The fix passes all 51 controls plus 420 permanent checks on both versions, covering both constructors, fallback and metadata schemas, metadata-only directories, all column names and types, row/column conversion, unused callbacks, and reading again after close. Both full suites pass with four threads and bounds checking: 4,588 checks on Julia 1.3.1 and 4,574 on Julia 1.13.1. An independent agent privately reviewed the complete diff and schema/iteration/loading/close paths and found no blocker. git diff --check passes. No package dependencies, exports, or version changes.

Hosted validation at aebbf1274af9d2f57d4274acd27ea9e01ff2fce7: all nine Julia 1.3/current/nightly jobs on Linux, macOS, and Windows and both coverage checks pass. Each uploader log confirms acceptance for this repository and commit; all nine reports are processed at 86.84%. No coverage threshold was changed.

Normally merged as c67ff831c2afc5ab412da1bd475b13fef3e0906e. The merged tree matches the fully tested PR tree. All nine exact-merge native jobs pass, and all nine correctly addressed coverage reports are accepted and processed at 86.84%.

Co-authored by Codex

AI disclosure: This work was prepared with assistance from OpenAI Codex.

When no partitions are selected, initialize each dataset column from the
discovered schema, matching the single-file reader's empty-column behavior.
Cover metadata-only datasets, filters excluding every file, logical overrides,
table conversions, and reading again after close.

AI disclosure: This work was prepared with assistance from OpenAI Codex.
@codecov

codecov Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.84%. Comparing base (d4f28a6) to head (aebbf12).

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #204      +/-   ##
==========================================
+ Coverage   86.81%   86.84%   +0.03%     
==========================================
  Files          10       10              
  Lines        1600     1604       +4     
==========================================
+ Hits         1389     1393       +4     
  Misses        211      211              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@quinnj
quinnj merged commit c67ff83 into master Oct 4, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant