fix: project_by_schema now reorders fields inside List<Struct> types #5703
+442
−10
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously, project_by_schema only recursively handled direct Struct fields. List<Struct>, LargeList<Struct>, and FixedSizeList<Struct> types fell through to the default case which cloned them without reordering inner struct fields. This caused Arrow validation errors when reading fragments where fields were stored out of order (scrambled `fields` array in DataFile metadata) combined with schema evolution requiring null-filling. Fixes lance-format#5702 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…ance-format#5702) Adds test data and integration test that reproduces the original bug: - Fragment 0: List<Struct<a, b, c>> with all fields + "extra" column - Fragment 1: List<Struct<c, b>> with reordered/missing inner struct fields This combination of out-of-order field storage + schema evolution inside the List<Struct> triggers project_by_schema to reorder fields. Before the fix, this would fail with: "Incorrect datatype for StructArray field expected List(Struct(...)) got List(Struct(...))" Also adds a direct unit test in dataset_schema_evolution.rs that tests the project_by_schema function with misordered List<Struct> fields. Fixes lance-format#5702 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
wjones127
commented
Jan 12, 2026
wjones127
commented
Jan 12, 2026
wjones127
commented
Jan 12, 2026
- Move test data to version-specific directory (v1.0.1) - Replace prints with assertions in datagen.py - Add assertion that scanning fails with issue lance-format#5702 error - Remove redundant test from dataset_schema_evolution.rs Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Regenerated the test data using pylance 1.0.1 as intended. Also fixed assertion to match actual error message format. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Xuanwo
approved these changes
Jan 13, 2026
Collaborator
Xuanwo
left a comment
Xuanwo
left a comment
There was a problem hiding this comment.
Thank you for working on this!
jackye1995
pushed a commit
to jackye1995/lance
that referenced
this pull request
Jan 21, 2026
…ance-format#5703) ## Summary - `project_by_schema` previously only handled field reordering for top-level columns and direct `Struct` fields, but not for fields nested inside `List<Struct>`, `LargeList<Struct>`, or `FixedSizeList<Struct>` types - Added `project_array` helper that recursively handles these nested list types - Added unit tests and integration test with checked-in test data that reproduces the original issue ## Test plan - [x] Unit tests in `lance-arrow` verify `project_by_schema` correctly reorders fields inside `List<Struct>` - [x] Integration test in `dataset_migrations.rs` reads test data that triggers the bug scenario - [x] Test data created with fragment 0 having `List<Struct<a,b,c>>` and fragment 1 having reordered/missing inner struct fields - [x] Verified that without the fix, reading fails with "Incorrect datatype for StructArray field expected List(Struct(...)) got List(Struct(...))" Fixes lance-format#5702 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
project_by_schemapreviously only handled field reordering for top-level columns and directStructfields, but not for fields nested insideList<Struct>,LargeList<Struct>, orFixedSizeList<Struct>typesproject_arrayhelper that recursively handles these nested list typesTest plan
lance-arrowverifyproject_by_schemacorrectly reorders fields insideList<Struct>dataset_migrations.rsreads test data that triggers the bug scenarioList<Struct<a,b,c>>and fragment 1 having reordered/missing inner struct fieldsFixes #5702
🤖 Generated with Claude Code