Commit a4af152
fix(reader): make primitive vortex.dict decode lazy
vortex.dict decoded asymmetrically. Strings went to VarBinArray.ofDict
and stayed lazy; primitives allocated rowCount * elemSize and scattered
codes into it. So an I64 dict column over a 256-entry pool wrote 8 bytes
per row to represent data whose compressed form is one code byte per row
plus a 2 KB pool — discarding the dictionary's entire point at decode.
The asymmetry was also between decoders, not just types:
DictLayoutDecoder.buildLazyDictPrimitive already built DictLongArray /
DictIntArray / DictShortArray / DictByteArray / DictDoubleArray /
DictFloatArray for the same dictionary expressed as a layout. This path
simply predates that work. It now uses the same carriers, so the two
agree.
The class javadoc claimed the broadcast semantics made lazy wrapping
non-trivial. They do not: an undersized codes or values buffer (the
ConstantEncoding fan-out) is fanned out by the Materialized* accessors'
own `i % elementCount`, which is where the expandXxx scatter loops got
it too. Kept and now pinned by decode-level tests, replacing the ones
that drove expandU8/U16/U32 directly — those covered a generic
element-width path the comment itself noted was unreachable through
decode(), and the helpers are deleted.
Two guarantees the eager path provided had to be re-established rather
than inherited, since a lazy carrier resolves codes at scan time:
- Codes past the pool. expand() got this from a boundary catch around
its scatter loop; lazily it would surface as a raw
IndexOutOfBoundsException far from the malformed file (ADR 0003). Now
one non-allocating pass at decode accumulates the maximum branchlessly
and compares once — strictly cheaper than the allocate-and-scatter it
replaces. It also now fires ahead of the pool-validity guard, so the
error blames the code rather than the validity bitmap it overran
second; one test's expectation moves accordingly.
- Codes ptype. expand() enforced U8/U16/U32 by dispatching on it. The
carriers accept any narrow-int array, so a signed or U64 codes child
no writer emits would have been accepted; restored as an explicit
check.
The zero-element-child guard stays: the broadcast wrap divides by
elementCount, so an empty child is still an ArithmeticException waiting
at scan time rather than one that disappeared with the expansion.
DictFilter pattern-matches DictLongArray/DictIntArray with ByteArray
codes and returns null for anything else, so encoding-level dicts newly
reaching its fast path degrade to the generic lane rather than
misbehaving; its tests pass unchanged.
docs/compatibility.md already listed vortex.dict as Lazy — true of the
layout path only. No edit needed; it is now true of both.
Closes #336
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>1 parent 8ff823e commit a4af152
3 files changed
Lines changed: 241 additions & 334 deletions
File tree
- reader/src
- main/java/io/github/dfa1/vortex/reader/decode
- test/java/io/github/dfa1/vortex/reader/decode
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
11 | 11 | | |
12 | 12 | | |
13 | 13 | | |
| 14 | + | |
14 | 15 | | |
15 | 16 | | |
16 | 17 | | |
| |||
0 commit comments