Make ScalarFn array validity lazy when the function defines a validity expression

[joseph-isaacs]

Summary

PR 4 of a 4-PR stack (stacked on #8335) — the payoff: lazy validity for ScalarFn arrays.

Previously ValidityVTable<ScalarFn> always eagerly executed the validity expression via the legacy session. Now, when the scalar function provides a validity expression over its inputs, the expression is converted into a lazy ScalarFn array DAG instead:

• Literal nodes → ConstantArray
• ArrayExpr leaves → unwrap to the child array they hold
• interior nodes → lazy ScalarFn arrays via Array::<ScalarFn>::try_new
• constant results are folded back into AllValid/AllInvalid via child_to_validity

[codspeed-hq]

Merging this PR will degrade performance by 17.42%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

❌ 5 regressed benchmarks
✅ 1527 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	baseline_eq[4, 65536]	186.4 µs	238.8 µs	-21.93%
❌	Simulation	baseline_lt[16, 65536]	219 µs	276.4 µs	-20.78%
❌	Simulation	baseline_lt[4, 65536]	202.2 µs	254 µs	-20.39%
❌	Simulation	bitwise_not_vortex_buffer_mut[128]	215.3 ns	244.4 ns	-11.93%
❌	Simulation	baseline_eq[16, 65536]	231.3 µs	261.2 µs	-11.45%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing claude/cool-bardeen-l8jlsy-4-lazy-scalarfn-validity (f9b0a16) with claude/cool-bardeen-l8jlsy-3-definitely-all-invalid (a13f0ab)}

[joseph-isaacs]

Investigated the CodSpeed varbinview_zip_* regressions — they are a codegen-layout artifact, not a real cost of this change:

1. The changed code never runs in these benchmarks. Instrumenting ScalarFn::validity with a counter shows 0 calls during varbinview_zip_block_mask/varbinview_zip_fragmented_mask (vs ~47k calls in the bool-consistency tests, confirming the instrumentation works). The benches go straight through the dedicated ZipKernel for VarBinView and never query a ScalarFn array's validity.

2. Bisecting the two files in this PR reproduces the full ~19% delta locally only when arrays/scalar_fn/vtable/validity.rs changes — the file whose code is never executed here. The scalar_fn/erased.rs change alone shows no delta.

3. codegen-units=1 erases the regression entirely (branch 3: 505µs median vs branch 3 + this PR's files: 494µs). With default codegen units, adding code to that module shifts LLVM's CGU partitioning and changes inlining in the unrelated hot zip loop, which CodSpeed's instruction-counting simulation faithfully reports.

The same flavor of noise shows in the rest of the stack: the bitwise_not_vortex_buffer_mut and chunked_*_canonical_into benchmarks flip between ±20–47% on adjacent PRs that don't touch vortex-buffer or the chunked builders at all.

I can't acknowledge regressions on the CodSpeed dashboard from this session — that needs someone with dashboard access.

https://claude.ai/code/session_01VPQ7dfZtijfrsjAipwXvEj

---

Generated by Claude Code

[robert3005]

do you want to remove the TODO?