docs(adr): ADR 0010 — final design, kernel follow-up, drop of() factory

Updates after PR #36 (array interface refactor) and the lazy-ALP PoC
in PR #35 settled the shape:

- Phase 1 swaps the earlier "open interface + package-private
  BufferedDoubleArray + static of()" sketch for the chosen design:
  non-sealed interface + public MaterializedXxxArray record, no
  static factory on the interface, ~115 construction sites updated
  in PR #36.
- Adds a "Possible follow-up: kernel-based MaterializedXxxArray"
  subsection that explores generalising the per-encoding eager loop
  into a kernel constructor, with the trade-offs (loss of
  encoding-specific branch-split tricks, per-row interface dispatch
  risk).
- Phase 2 drops the hasFilter() gate that the early draft proposed.
  Measurement showed lazy is +9.5% on the no-filter full fold, so
  gating is unnecessary and was rejected. The fused chain detector
  description is folded into Phase 2.
- Phase 3 explicitly notes that compute pushdown lives as direct
  methods on the encoding-specific concrete (sumWhereGt) rather than
  a Kernel SPI; the SPI is deferred until a second encoding ships
  its lazy variant.
- Consequences section rewritten to match: public Materialized*
  types, no factory, method-on-concrete pushdown.
- Removes a duplicate Phase 2 — compute pushdown stub that earlier
  edits had left behind.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Author: dfa1

docs/adr/0010-lazy-decode.md

@@ -100,8 +100,9 @@ like a regression on the only number we measure.
   nothing. (TODO.)
 
 Keep the existing `javaReadClose` (full fold) as the **negative test**.
-No phase may regress it — the no-filter path stays bit-for-bit eager
-(see the API gate below).
+Initial fear was that lazy-by-default would regress it; PoC measurement
+showed the opposite (+9.5% on the OHLC chain), so the "gate on
+`hasFilter()`" idea is dropped — see Phase 2.
 
 #### Phase 0 baseline (10M rows, OHLC, `close` column)
 
@@ -116,96 +117,132 @@ Two observations:
 1. Java loses 3.5–4× to JNI at low selectivity. The whole loss is eager
    decode of rejected rows.
 2. Java *already* wins at 100% selectivity — JNI's per-batch Arrow
-   marshalling costs more than Java's tight fold. **This is the reason
-   to gate lazy on `hasFilter()` instead of making it the default.**
+   marshalling costs more than Java's tight fold.
 
-### API gate — eager unless a filter is present
+Initial reading was "gate lazy behind `hasFilter()` so the full-fold
+path stays eager and the filter path switches to lazy." Measurement
+rejected the gate (see below — lazy is faster on both paths).
+
+### API gate (rejected by PoC measurement)
+
+Early draft proposed gating lazy decode behind
+`ScanOptions.hasFilter()` so the no-filter path stayed bit-for-bit
+eager:
 
 ```
 ScanOptions.hasFilter() == false  →  eager path (today), zero change
 ScanOptions.hasFilter() == true   →  lazy + compute pushdown
 ```
 
-Consequences of the gate:
+The motivation was protecting `javaReadClose` (the README full-fold
+bench) from any regression caused by virtual dispatch on the lazy
+variant.
 
-- `javaReadClose` (no filter) is **untouched**. No `DoubleArray`
-  polymorphism, no virtual call, no patch-bitmap allocation. Eliminates
-  the "negative consequence" that worried earlier drafts.
-- `javaFilterClose` switches to the pushdown path. The user's loop does
-  not change: the chunk it receives is already **compacted** to matching
-  rows, so the per-row `if (v > threshold)` check goes away.
-
-```java
-// Today — user pays the per-row predicate check
-for (long i = 0; i < close.length(); i++) {
-    double v = close.getDouble(i);
-    if (v > threshold) sum += v;
-}
-
-// After phase 2 — chunk is pre-filtered, length = matched rows only
-for (long i = 0; i < close.length(); i++) {
-    sum += close.getDouble(i);
-}
-```
-
-The filter is applied inside `ScanIterator.next()` *before* the chunk is
-returned. `close.length()` reports the matched row count for that chunk.
-Empty chunks are skipped inside `next()`.
+**PoC measurement rejected this gate.** Lazy decode is *strictly
+faster* than eager on full fold (+9.5% on `javaReadClose`) because
+the materialisation write/read intermediate buffer disappears — the
+lazy variant returns the encoded segment directly and applies the
+transform on access; the fused variant unpacks bitpacked → double in
+one pass. Net halving of memory traffic on the OHLC chain. The gate
+is dropped; lazy is the default whenever the chain pattern matches.
 
 ### Phase 1 — Array hierarchy refactor (no behavior change)
 
 Today each primitive Array is a `public final class` with a
 `MemorySegment` buffer field. Lazy variants cannot extend it. Convert
-every numeric Array to an **open interface**; keep the current concrete
-behavior as a package-private default record exposed only via a static
-factory:
+every numeric and bool Array to a **non-sealed interface** and move
+the current behaviour into a **public** `MaterializedXxxArray` record.
+No static factory on the interface — encoders construct
+`new MaterializedXxxArray(...)` directly, keeping the interface a
+pure contract:
 
 ```java
-public interface DoubleArray extends Array {
+public non-sealed interface DoubleArray extends Array {
     double getDouble(long i);
     void forEachDouble(DoubleConsumer c);
     double fold(double identity, DoubleBinaryOperator op);
-
-    /// Default impl backed by a materialized MemorySegment.
-    /// Clients never reference the concrete type.
-    static DoubleArray of(DType dtype, long length, MemorySegment buffer) {
-        return new BufferedDoubleArray(dtype, length, buffer);
-    }
 }
 
-// package-private — name does not appear in the public API
-record BufferedDoubleArray(DType dtype, long length, MemorySegment buffer)
-        implements DoubleArray {
-    public double getDouble(long i) {
-        return buffer.getAtIndex(PTypeIO.LE_DOUBLE, i);
-    }
-    // forEachDouble, fold — same body as today's DoubleArray
+public final class MaterializedDoubleArray implements DoubleArray {
+    public MaterializedDoubleArray(DType dtype, long length, MemorySegment buffer) { ... }
+    @Override public double getDouble(long i) { ... }       // existing body
+    @Override public void forEachDouble(DoubleConsumer c) { ... }
+    @Override public double fold(double identity, DoubleBinaryOperator op) { ... }
 }
 ```
 
-Scope: `DoubleArray`, `FloatArray`, `LongArray`, `IntArray`,
-`ShortArray`, `ByteArray`. Defer `BoolArray`, `VarBinArray`,
-`VarBinViewArray`, `MaskedArray` — different shapes, no lazy candidate
-encoding for them yet.
+Scope: `BoolArray`, `ByteArray`, `ShortArray`, `IntArray`, `LongArray`,
+`Float16Array`, `FloatArray`, `DoubleArray`. Defer `VarBinArray`,
+`VarBinViewArray`, `NullArray`, `EmptyArray` — no lazy candidate
+encoding for them yet. Container types (`StructArray`, `MaskedArray`,
+`ListArray`, etc.) stay `final class` — not lazy candidates.
 
-Rewrite ~69 `new DoubleArray(...)` (and sibling) call sites in
-`reader.decode.*` and `ScanIterator` to use `DoubleArray.of(...)`.
-Pure textual change; the factory returns the same buffer-backed
-implementation as before.
+Rewrite all `new XxxArray(...)` call sites (~115 across reader,
+writer, integration, performance, core tests) to
+`new MaterializedXxxArray(...)`. Pure textual change.
+`ArraySegments` pattern matches resolve to the concrete
+`MaterializedXxxArray` so its `buffer()` accessor stays
+package-private.
 
 **Behavior unchanged.** Every existing test, integration test, and
-benchmark sees `BufferedDoubleArray` everywhere via the interface. JIT
-call sites stay monomorphic until a second impl is introduced. Run
-`./mvnw verify` and `RustVsJavaReadBenchmark` to confirm zero
-regression before moving on.
+benchmark sees `MaterializedXxxArray` everywhere via the interface.
+JIT call sites stay monomorphic until a second impl is introduced.
 
 **Not sealed.** Custom encoding-specific concretes (phase 2+) just
-`implements DoubleArray` — no permits edit, no exhaustive switch
+`implements XxxArray` — no permits edit, no exhaustive switch
 required in callers. Kernels use `instanceof` + `default` fallback so
 unknown impls degrade to the generic per-row path automatically. This
 keeps third-party encodings on equal footing with built-in ones.
 
-### Phase 2 — Lazy ALP variant + filter gate
+**No `static of(...)` on the interface.** An earlier draft added a
+factory returning a (then package-private) buffer-backed default.
+That coupled the interface to its concrete and proved unused once
+decoders were rewritten to call `new MaterializedXxxArray(...)`
+directly. The interface stays a pure contract.
+
+#### Possible follow-up: kernel-based MaterializedXxxArray
+
+Every encoding's eager decode is the same pattern: allocate a buffer,
+loop over rows, write `kernel(i)`. Generalising would let the
+constructor own the loop:
+
+```java
+public MaterializedDoubleArray(DType dtype, long length,
+                                LongToDoubleFunction kernel, Arena arena) {
+    MemorySegment dst = arena.allocate(length * 8, 8);
+    for (long i = 0; i < length; i++) {
+        dst.setAtIndex(LE_DOUBLE, i, kernel.applyAsDouble(i));
+    }
+    // store dst
+}
+```
+
+Then `AlpEncodingDecoder.decodeF64` shrinks to ~4 lines:
+
+```java
+return new MaterializedDoubleArray(dtype, n,
+        i -> (double) src.getAtIndex(LE_LONG, i) * scale, arena);
+```
+
+Trade-offs:
+
+- **Win:** dedup. Every eager decoder collapses to one expression +
+  one constructor call.
+- **Cost — per-row `invokeinterface`.** C2 inlines if the kernel
+  call site is monomorphic (one decoder = one kernel impl). Bimorphic
+  risk if multiple decoders alias the loop body.
+- **Lost flexibility.** Encoding-specific tricks — `i % srcCap`
+  branch-split, broadcast-cap branches, in-place writes when source
+  is writable — don't fit a pure kernel. Forcing them into the kernel
+  means a per-row `cap` check (banned by the hot-loop rule); keeping
+  them out means the kernel constructor only covers the boring case.
+
+Add the kernel constructor *alongside* the existing buffer
+constructor. Decoders that want the loop dedup opt in; decoders with
+quirky paths keep their own loop. Tracked as a follow-up; not part of
+this phase.
+
+### Phase 2 — Lazy ALP + fused chain
 
 Add the first lazy implementation:
 
@@ -238,46 +275,77 @@ Patches index: two options for O(1) lookup:
 
 Use the bitmap. Predictable per-access cost.
 
-**Filter gate** in `AlpEncodingDecoder.decode`:
+**No filter gate.** PoC measurement showed lazy is *strictly faster*
+than the eager path even on full-fold workloads (+10.6% on
+`javaReadClose`), because the materialisation write/read intermediate
+buffer is skipped entirely. The earlier "gate lazy behind
+`hasFilter()`" idea is dropped in the final design — lazy is the
+default whenever the chain pattern matches:
 
 ```java
-return ctx.hasFilter()
-    ? new AlpDoubleArray(dtype, n, encoded, scale, patches)   // lazy
-    : MaterializedDoubleArray.of(...);                         // today's eager path
+return new AlpDoubleArray(dtype, n, encoded, scale, patches);   // always lazy
+// fallback to: new MaterializedDoubleArray(...) when not lazy-eligible
+// (read-only source, broadcast source, patched chunk)
 ```
 
-`DecodeContext` gains an `hasFilter()` hint propagated from
-`ScanOptions`. No `RowFilter` parsing inside the decoder — only the
-boolean signal.
+No `DecodeContext.hasFilter()` plumbing is needed — the gate is gone.
+
+#### Fused chain detection
+
+When the ALP child layout is `FoR(Bitpacked)` (or `Bitpacked` directly
+when the writer dropped FoR for `ref==0`), the decoder skips the
+intermediate FoR/ALP buffers entirely and returns a
+`FusedAlpForBitpackedDoubleArray` that holds the raw packed buffer +
+`(bitWidth, offset, ref, scale)`. `sumWhereGt` unpacks each row,
+applies `+ref` and the threshold compare inline, decodes
+`(double)(val+ref)*scale` only for matches. The full-fold path
+(`getDouble`/`fold`/`forEachDouble`) lazily materialises through one
+pass that writes doubles directly from the bitpacked unpack —
+halving the memory traffic vs the old eager chain (one decode pass
+instead of bitpacked→FoR→ALP).
+
+`AlpEncodingDecoder` walks the `ArrayNode` tree to detect the chain
+(no decoding cost to peek). Falls back to `AlpDoubleArray` for the
+bare-ALP case, `MaterializedDoubleArray` for patched chunks.
 
 ### Phase 3 — compute pushdown
 
-`ScanIterator` routes `ScanOptions.rowFilter()` through a kernel SPI
-before falling back to materialization. Initial kernels:
+PoC chose **direct methods on the encoding-specific concrete** over a
+dedicated Kernel SPI. `AlpDoubleArray.sumWhereGt(threshold)` and
+`FusedAlpForBitpackedDoubleArray.sumWhereGt(threshold)` are public
+methods; callers (today: the filter bench; tomorrow: `ScanIterator`)
+pattern-match on the concrete type and call the right method:
+
+```java
+DoubleArray col = chunk.column("close");
+double sum = switch (col) {
+    case FusedAlpForBitpackedDoubleArray fused -> fused.sumWhereGt(threshold);
+    case AlpDoubleArray alp                   -> alp.sumWhereGt(threshold);
+    default                                    -> Filters.scalarSumGt(col, threshold);
+};
+```
+
+The Kernel SPI from earlier drafts is **deferred** until a second
+encoding (FoR, ZigZag, AlpRd) ships its own lazy variant and the
+combinatorial explosion of `(kernel × encoding)` becomes visible.
+Until then, method-on-concrete is simpler to read, faster to inline
+(no SPI dispatch), and lets each encoding expose the operators that
+make sense for its math without an upstream interface lock-in.
+
+When the SPI eventually lands, candidates are:
 
 - `CompareKernel`: `compare(arr, scalar, op) → BoolArray`. For
   `AlpDoubleArray`, encode the scalar to the int domain
-  (`enc = round(scalar / scale)`) and compare ints. For
+  (`enc = floor(scalar / scale)`) and compare ints. For
   `ForLongArray` (when it lands), subtract the reference and compare
   ints. Falls back to materialization when the scalar does not
   round-trip through the encoding.
 - `BetweenKernel`: same approach for two scalars.
 - `TakeKernel`: `take(arr, indices)` — decode only the requested
   indices. Unblocks the take/slice/projection wins from phase 0.
-- `SumKernel`, `MinKernel`, `MaxKernel`: deferred.
-  `sum(AlpDoubleArray) = sum(int) * scale + patch_correction` is
-  straightforward but not on the critical path.
-
-Pattern-match dispatch in `ScanIterator` with a `default` fallback —
-no exhaustiveness required, so unknown future impls degrade gracefully:
-
-```java
-DoubleArray col = chunk.column("close");
-BoolArray sel = switch (col) {
-    case AlpDoubleArray alp -> alp.compareGt(threshold, arena);
-    default                 -> Filters.scalarGt(col, threshold); // generic via getDouble
-};
-```
+- `SumKernel`, `MinKernel`, `MaxKernel`: `sum(AlpDoubleArray) =
+  sum(int) * scale + patch_correction` is straightforward but not on
+  the critical path.
 
 For multi-column filters: `AND` evaluates kernels in column order,
 intersecting selection vectors; `OR` unions them. Columns referenced
@@ -286,36 +354,20 @@ and are not delivered to the consumer.
 
 ### Future — extend the lazy family
 
-Once ALP proves the shape, add (no API change, just new permits):
+Once ALP proves the shape (which it has, on PR #35), add the same
+pattern per encoding. No interface change — each new variant is just
+another `implements DoubleArray` (or `LongArray`, `IntArray`):
 
 - `AlpRdDoubleArray` — same idea for ALP-RD
-- `ForLongArray`, `ForIntArray` — Frame-of-Reference, in-place lazy
-- `ZigZagLongArray`, `ZigZagIntArray` — XOR/shift on access
-- Composed: `AlpForBitpackedDoubleArray` fuses three transforms into
-  one expression evaluated per access
-
-### Phase 2 — compute pushdown
-
-`ScanIterator` routes `ScanOptions.rowFilter()` through a kernel SPI
-before falling back to materialization. Initial kernels:
-
-- `CompareKernel`: `compare(arr, scalar, op) → BoolArray`. For ALP,
-  encode the scalar to the int domain (`enc = round(scalar / scale)`)
-  and compare ints. For FoR, subtract the reference and compare ints.
-  Falls back to materialization when the scalar does not round-trip
-  through the encoding (e.g. ALP threshold that is not representable as
-  `int * 10^(f-e)` exactly).
-- `BetweenKernel`: same approach for two scalars.
-- `TakeKernel`: `take(arr, indices)` — decode only the requested
-  indices. Unblocks the take/slice/projection wins from phase 0.
-- `SumKernel`, `MinKernel`, `MaxKernel`: deferred. `sum(ALP) =
-  sum(int) * scale + patch_correction` is straightforward but not on the
-  critical path.
-
-For multi-column filters: `AND` evaluates kernels in column order,
-intersecting selection vectors; `OR` unions them. Columns referenced
-only by the filter (not by projection) are decoded just enough to test
-and are not delivered to the consumer.
+- `ForLongArray`, `ForIntArray` — Frame-of-Reference, lazy
+- `ZigZagLongArray`, `ZigZagIntArray` — XOR/shift on access (order
+  not preserved, so no pushdown — but lazy still skips the
+  materialisation pass)
+- Composed: `FusedAlpForBitpackedDoubleArray` (already exists) fuses
+  three transforms; analogous fused classes for other common chains
+- Extended fusion: handle bitpacked patches inside the fused kernel,
+  closing the remaining 64% of OHLC chunks that today fall back to
+  `AlpDoubleArray`
 
 ## Consequences
 
@@ -335,18 +387,24 @@ and are not delivered to the consumer.
 
 ### Negative
 
-- **API surface grows minimally.** Every numeric `*Array` becomes an
-  open interface; the default buffer-backed impl is package-private,
-  accessed via `*Array.of(...)`. Downstream consumers that constructed
-  `new DoubleArray(...)` directly must switch to the factory; consumers
-  that only *receive* arrays from `Chunk.column(...)` see no change.
-  Encoding-specific concretes (`AlpDoubleArray`, etc.) become first-class
-  public types that kernels can pattern-match against.
+- **API surface grows.** Every numeric `*Array` becomes a
+  `non-sealed interface`. The default buffer-backed impl becomes a
+  public `MaterializedXxxArray` record. Downstream consumers that
+  constructed `new DoubleArray(...)` directly must switch to
+  `new MaterializedDoubleArray(...)`; consumers that only *receive*
+  arrays from `Chunk.column(...)` see no change. Encoding-specific
+  concretes (`AlpDoubleArray`, `FusedAlpForBitpackedDoubleArray`,
+  etc.) are first-class public types that callers pattern-match
+  against. The interface stays a pure contract — no `static of(...)`
+  factory, no permits, no encoder coupling.
 - **Patch lookup is per-access in the lazy path.** Today patches are
   applied once at decode time. Lazy needs an index structure (cost
   above) and pays per-row. Only the filter path triggers this.
-- **Kernel SPI is a non-trivial design.** Initial scope must be small:
-  compare, between, take. Sum/min/max can wait.
+- **Compute pushdown lives as methods on the encoding-specific
+  concrete.** No Kernel SPI yet — `AlpDoubleArray.sumWhereGt(...)`,
+  etc., are direct public methods. Cleaner to inline, looser to
+  extend. SPI lands when a second encoding's lazy variant proves the
+  shape repeats.
 - **Filter semantics change.** Today `RowFilter` is a zone-map prune
   hint; the consumer still re-checks every row. After phase 3 the chunk
   returned by `next()` is already filtered. This is a breaking change