Skip to content

GH-48277: [C++][Parquet] unpack with shuffle algorithm #47994

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
AntoinePrv wants to merge 87 commits into
base: main
Choose a base branch
from
+2,023 −45,857
Open

GH-48277: [C++][Parquet] unpack with shuffle algorithm #47994

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
e617103
Add SSE4.2 implementation
AntoinePrv Oct 27, 2025
0879e3d
Add unpack uint8_t benchmark
AntoinePrv Oct 28, 2025
19c3b1c
Add bool unpack benchmark
AntoinePrv Oct 30, 2025
21590e6
Bias benchmarks toward small scale
AntoinePrv Nov 25, 2025
605353b
Add Kernel plan builder
AntoinePrv Oct 20, 2025
3eaa18d
Add simd kernel
AntoinePrv Oct 24, 2025
05310ad
Handle rshifts on SSE2
AntoinePrv Oct 27, 2025
9fb8630
Use new kernel when possible in generated 128 code
AntoinePrv Oct 27, 2025
6ea2a12
Refactor array to xsimd::batch_constant
AntoinePrv Oct 27, 2025
432d41b
Refactor right shift
AntoinePrv Oct 27, 2025
53044dc
Add oversized plan
AntoinePrv Oct 28, 2025
5faa493
Add oversized kernel
AntoinePrv Oct 28, 2025
d747743
Rename kernels
AntoinePrv Oct 28, 2025
a60fe4c
Add simd kernel dispatch
AntoinePrv Oct 28, 2025
1dc5029
Call Simd kernel directly
AntoinePrv Oct 28, 2025
0a2c3ed
Fix SIMD level None
AntoinePrv Oct 29, 2025
111f8a0
Initialize swizzles to -1
AntoinePrv Oct 29, 2025
ef197ab
Doc
AntoinePrv Oct 29, 2025
f38d591
Improve test error message
AntoinePrv Oct 29, 2025
c01b02b
Use new kernel in avx2
AntoinePrv Oct 28, 2025
adb67a3
AVX2 swizzle fallback
AntoinePrv Oct 29, 2025
a0e81a7
Remove dead code
AntoinePrv Oct 30, 2025
0d7af17
Simplify Large masks
AntoinePrv Oct 30, 2025
cb8621c
Remove bpacking 256 generated file
AntoinePrv Oct 30, 2025
d18ff10
Remove uint8_t fallback
AntoinePrv Oct 30, 2025
9b6e71a
Add boolean simd implementation
AntoinePrv Oct 30, 2025
b0e67d8
Use std::is_base_of for arch detection
AntoinePrv Oct 30, 2025
7411345
Improve swizzle
AntoinePrv Nov 17, 2025
2fe50e5
Only use lshift hack when available
AntoinePrv Nov 17, 2025
d3b3511
Fix return type
AntoinePrv Nov 17, 2025
ddfe4f0
Fix shift included size
AntoinePrv Nov 18, 2025
bbeeb24
Add Avx2 uint16_t shift fallback
AntoinePrv Nov 19, 2025
377f201
Refactor make_mult
AntoinePrv Nov 19, 2025
3c414a3
Add Avx2 lshift unint8_t fallback
AntoinePrv Nov 19, 2025
ebb58ce
Refactor right shift excess
AntoinePrv Nov 19, 2025
05ef1c2
Refactor make_mult
AntoinePrv Nov 20, 2025
09100ed
Add SSE var shift uint8_t fallback to uint16_t
AntoinePrv Nov 20, 2025
b4718dc
Implement size reading reduction
AntoinePrv Nov 20, 2025
6ee9057
Add fallback Avx2 right shift
AntoinePrv Nov 24, 2025
3dc3d89
Refactor static dispatch
AntoinePrv Nov 26, 2025
5ba3f2d
Forward oversized to larger uint when possible
AntoinePrv Nov 26, 2025
fe4904f
Add arch detection functions
AntoinePrv Nov 26, 2025
bceb26b
Refactor traits usage
AntoinePrv Nov 26, 2025
26495e5
Forward x86_64 unpack64 to unpack32
AntoinePrv Nov 26, 2025
25c24a2
Simplify template usage
AntoinePrv Nov 26, 2025
c7da4f7
Reorganize and doc
AntoinePrv Nov 26, 2025
4e2d568
Refactor KernelDispatch and remove Oversized dispatch
AntoinePrv Nov 26, 2025
6230da8
Forward large unpack8 to unpack16 on SSE2
AntoinePrv Nov 26, 2025
8c7fbdf
Use fallback right shift on large uint8_t avx2
AntoinePrv Nov 26, 2025
880e826
Fix enable_if
AntoinePrv Nov 27, 2025
e4395ec
Add missing header
AntoinePrv Nov 27, 2025
352b4f8
fmt
AntoinePrv Nov 27, 2025
c290839
Add SSE4.2 to dynamic dispatch
AntoinePrv Nov 27, 2025
f711f27
Rename bpacking_simd_impl > bpacking_simd_kernel
AntoinePrv Nov 27, 2025
3a6bebd
Restore modifications to simd_codegen
AntoinePrv Nov 27, 2025
b37669e
Reduce reading size and declare bytes read
AntoinePrv Nov 27, 2025
0e2e5cb
Add kBytesRead to scalar code
AntoinePrv Nov 27, 2025
b180f04
Add kBytesRead to simd 512 generated code
AntoinePrv Nov 27, 2025
92253aa
Prevent overreading
AntoinePrv Nov 27, 2025
60c185f
Fix pessimit overeading guard
AntoinePrv Nov 28, 2025
c5da75e
Fix overreading guard comparison
AntoinePrv Dec 1, 2025
dcab234
Add UnpackOptions and max_read_bytes
AntoinePrv Dec 1, 2025
c5e3ba2
Use C++20 NTTP
AntoinePrv Jan 7, 2026
e6e24fc
xsimd 14.0 compatibility
AntoinePrv Jan 8, 2026
d9fdecb
fmt
AntoinePrv Jan 21, 2026
42c43f2
C++20 NTTP options
AntoinePrv Jan 23, 2026
8ad49c2
Homogenous wording
AntoinePrv Jan 23, 2026
8432902
Remove xsimd backward compatibility
AntoinePrv Feb 3, 2026
7734ce8
Apply doc fixes from code review
AntoinePrv Feb 6, 2026
eed06b1
Documentation and code improvements
AntoinePrv Feb 6, 2026
752bd4f
Move utilities into bpacking sub ns
AntoinePrv Feb 9, 2026
8ff5322
Refactor plan builders
AntoinePrv Feb 9, 2026
925d7a0
Move utilities
AntoinePrv Feb 9, 2026
a12d9af
Kernel documentation
AntoinePrv Feb 9, 2026
f793933
adjust_bytes_per_read doc
AntoinePrv Feb 9, 2026
77f17bd
Fewer typename
AntoinePrv Feb 9, 2026
2e879f3
Add documentation
AntoinePrv Feb 9, 2026
8ee302d
Fix bounds in plan builders
AntoinePrv Feb 10, 2026
46fc753
Change names
AntoinePrv Feb 10, 2026
2a8d86d
Add extra comments
AntoinePrv Feb 11, 2026
c05d2ab
Fix comments
AntoinePrv Feb 12, 2026
f5b9aec
Old compiler compatibility
AntoinePrv Feb 19, 2026
2ca9e05
Remove all workflow
AntoinePrv Feb 19, 2026
6ab408c
disable SIMD
AntoinePrv Feb 19, 2026
12f9d66
Try SSE4_2 SIMD level
AntoinePrv Feb 19, 2026
421ce25
Try SSE4_2 compile, AVX2 runtime SIMD
AntoinePrv Feb 19, 2026
bb99583
Try avx512
AntoinePrv Feb 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 0 additions & 84 deletions .github/workflows/archery.yml
View file
Open in desktop

This file was deleted.

79 changes: 0 additions & 79 deletions .github/workflows/check_labels.yml
View file
Open in desktop

This file was deleted.

75 changes: 0 additions & 75 deletions .github/workflows/comment_bot.yml
View file
Open in desktop

This file was deleted.

Loading
Loading