Skip to content

Conversation

@yingjianwu98
Copy link
Contributor

@yingjianwu98 yingjianwu98 commented Oct 17, 2025

In order to compress complex all null, we need to add additional parameters in the proto so we know what compression are used for definition level and repetition level and the number of values accordingly.

resolve #4885

@github-actions github-actions bot added the enhancement New feature or request label Oct 17, 2025
@yingjianwu98
Copy link
Contributor Author

yingjianwu98 commented Oct 17, 2025

Have this initial implementation ready for review, and if this is on the right track, I will work on changes to make it backward compatible.

@codecov-commenter
Copy link

Codecov Report

❌ Patch coverage is 86.22449% with 27 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.74%. Comparing base (cd4a91e) to head (2cdd916).
⚠️ Report is 22 commits behind head on main.

Files with missing lines Patch % Lines
.../lance-encoding/src/encodings/logical/primitive.rs 85.32% 15 Missing and 12 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4990      +/-   ##
==========================================
+ Coverage   81.72%   81.74%   +0.01%     
==========================================
  Files         334      340       +6     
  Lines      133358   137163    +3805     
  Branches   133358   137163    +3805     
==========================================
+ Hits       108986   112119    +3133     
- Misses      20720    21312     +592     
- Partials     3652     3732      +80     
Flag Coverage Δ
unittests 81.74% <86.22%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Xuanwo
Copy link
Collaborator

Xuanwo commented Jan 14, 2026

Thank you for working on this, and sorry for the late review. I believe this PR can continue the work from #5641. Are you still interested in working on it? If you don't have time, I can continue the work base on this PR.

@yingjianwu98
Copy link
Contributor Author

@Xuanwo

Thanks for the review!
I don't want to block your development but If this is not too urgent, I can work on a PR for review this this week if that's ok.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Rep/Def levels should be compressed in the all null page layout

3 participants