Compare→bitmask SIMD lowering: where custom SIMD actually helps

[joseph-isaacs]

Summary

This branch investigates where hand-written SIMD genuinely beats compiler
auto-vectorization for compare → bitmask lowerings, and lands the one place
it clearly does plus a safety cleanup. It is intentionally draft — the
headline is the analysis, validated by the compare_lowering benchmark (and,
I expect, by the CodSpeed run on this PR).

What's here

1. pack_nonzero_bytes (already on the branch): runtime-dispatched
   (avx512 → avx2 → scalar) SIMD pack of u8 != 0 into a bitmask, wired into
   the BitBuffer/BitBufferMut From<&[u8]> / From<&[bool]> paths.
2. between safety cleanup: route primitive/decimal between through a new
   safe collect_bool_slice (drops an unsafe { *slice.get_unchecked(idx) }).
   This is perf-neutral, not a speedup — see below.
3. compare_lowering bench: best portable scalar vs runtime-dispatched SIMD
   for three lowerings (u8 != 0, i32 > 5, 5 < i32 < 10), all writing the
   same u64 bitmask, with a verify() correctness gate.

The finding: it depends entirely on build target and working set

Vortex ships baseline x86-64 (SSE2) by default (no target-cpu in
.cargo/config.toml); CodSpeed builds with +avx2; the wall-clock bench CI
uses -C target-cpu=native (AVX-512). Whether scalar matches SIMD changes
across all three.

Wall-clock (taskset -c 0, median µs), all writing a u64 bitmask:

u8 != 0 — SIMD always wins, no scalar matches

target	best scalar @16ki	SIMD @16ki	best scalar @1mi	SIMD @1mi
SSE2	swar 2.5	0.13 (19×)	swar 159	29 (5.4×)
AVX2	swar 1.8	0.13 (14×)	swar 117	30 (3.9×)
native	swar 1.4	0.13 (11×)	swar 87	30 (2.9×)

The compiler can't synthesize the byte→bit pack; only a SIMD movemask/vptestmb
does it well. (The "best scalar" here is a carry-free SWAR — the textbook
haszero trick is wrong for an exact per-byte mask due to inter-byte borrow.)

i32 > 5 and 5 < i32 < 10 — scalar matches SIMD only with AVX-512

target	scalar @16ki	SIMD @16ki	scalar @1mi	SIMD @1mi
SSE2	12.2	1.1 (11×)	783	164 (4.8×)
AVX2	3.0	1.1 (2.7×)	197	165 (1.2×)
native	1.6	1.1 (1.4×)	161	161 (≈1.0×)

Compare→bitmask needs mask registers (AVX-512 vpcmpd→k) to vectorize well.
At SSE2/AVX2 the compiler leaves it scalar, so dispatched SIMD wins 2.7–11× in
cache. At native the compiler auto-vectorizes and scalar matches SIMD; at
DRAM sizes the op is memory-bandwidth-bound and the two converge regardless.

Honest caveats

• The between rewrite is a safety change (removes unsafe), not a speedup;
  an earlier "win" I reported was a divan harness artifact (it pessimizes the
  index-closure form; production calls these directly and vectorizes).
• CodSpeed runs in simulation/instruction-count mode, which ignores memory
  bandwidth. So I expect it to show SIMD doing far fewer instructions than scalar
  for all three ops at all sizes — including the i32 DRAM cases where
  wall-clock is at parity. That corroborates "SIMD is more work-efficient" but
  does not contradict the wall-clock memory-bound parity above.

Tests / checks

• cargo nextest run -p vortex-array between (30 passed)
• compare_lowering verify() gate passes (all SIMD/SWAR variants match scalar)
• cargo clippy -p vortex-buffer --bench compare_lowering clean
• cargo +nightly fmt -p vortex-buffer

Benchmarks run locally on one pinned core at SSE2 / +avx2 / target-cpu=native.

---

Generated by Claude Code

[codspeed-hq]

Merging this PR will degrade performance by 16.61%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

❌ 2 regressed benchmarks
✅ 1511 untouched benchmarks
🆕 62 new benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	varbinview_zip_block_mask	2.9 ms	3.7 ms	-21.59%
❌	Simulation	varbinview_zip_fragmented_mask	6.1 ms	6.9 ms	-11.31%
🆕	Simulation	between_scalar_pack[1048576]	N/A	2.6 ms	N/A
🆕	Simulation	between_scalar_pack[16384]	N/A	41.2 µs	N/A
🆕	Simulation	between_simd_bench[1048576]	N/A	2.3 ms	N/A
🆕	Simulation	between_simd_bench[16384]	N/A	36.3 µs	N/A
🆕	Simulation	gt_scalar_pack[1048576]	N/A	2.5 ms	N/A
🆕	Simulation	gt_scalar_pack[16384]	N/A	39.7 µs	N/A
🆕	Simulation	gt_simd_bench[1048576]	N/A	2.2 ms	N/A
🆕	Simulation	gt_simd_bench[16384]	N/A	34.9 µs	N/A
🆕	Simulation	u8_scalar_pack[1048576]	N/A	979.9 µs	N/A
🆕	Simulation	u8_scalar_pack[16384]	N/A	16.1 µs	N/A
🆕	Simulation	u8_scalar_swar[1048576]	N/A	934.2 µs	N/A
🆕	Simulation	u8_scalar_swar[16384]	N/A	15.2 µs	N/A
🆕	Simulation	u8_simd[1048576]	N/A	610.9 µs	N/A
🆕	Simulation	u8_simd[16384]	N/A	10 µs	N/A
🆕	Simulation	between_bitbuffer_new[1024]	N/A	5.7 µs	N/A
🆕	Simulation	between_bitbuffer_new[1048576]	N/A	2.8 ms	N/A
🆕	Simulation	between_bitbuffer_new[16384]	N/A	47 µs	N/A
🆕	Simulation	between_bitbuffer_new[262144]	N/A	692.4 µs	N/A
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing claude/vector-bitpack-lowering-yKOkC (5d476ef) with develop (e06d80b)}