Skip to content

Commit a4af152

Browse files
dfa1claude
andcommitted
fix(reader): make primitive vortex.dict decode lazy
vortex.dict decoded asymmetrically. Strings went to VarBinArray.ofDict and stayed lazy; primitives allocated rowCount * elemSize and scattered codes into it. So an I64 dict column over a 256-entry pool wrote 8 bytes per row to represent data whose compressed form is one code byte per row plus a 2 KB pool — discarding the dictionary's entire point at decode. The asymmetry was also between decoders, not just types: DictLayoutDecoder.buildLazyDictPrimitive already built DictLongArray / DictIntArray / DictShortArray / DictByteArray / DictDoubleArray / DictFloatArray for the same dictionary expressed as a layout. This path simply predates that work. It now uses the same carriers, so the two agree. The class javadoc claimed the broadcast semantics made lazy wrapping non-trivial. They do not: an undersized codes or values buffer (the ConstantEncoding fan-out) is fanned out by the Materialized* accessors' own `i % elementCount`, which is where the expandXxx scatter loops got it too. Kept and now pinned by decode-level tests, replacing the ones that drove expandU8/U16/U32 directly — those covered a generic element-width path the comment itself noted was unreachable through decode(), and the helpers are deleted. Two guarantees the eager path provided had to be re-established rather than inherited, since a lazy carrier resolves codes at scan time: - Codes past the pool. expand() got this from a boundary catch around its scatter loop; lazily it would surface as a raw IndexOutOfBoundsException far from the malformed file (ADR 0003). Now one non-allocating pass at decode accumulates the maximum branchlessly and compares once — strictly cheaper than the allocate-and-scatter it replaces. It also now fires ahead of the pool-validity guard, so the error blames the code rather than the validity bitmap it overran second; one test's expectation moves accordingly. - Codes ptype. expand() enforced U8/U16/U32 by dispatching on it. The carriers accept any narrow-int array, so a signed or U64 codes child no writer emits would have been accepted; restored as an explicit check. The zero-element-child guard stays: the broadcast wrap divides by elementCount, so an empty child is still an ArithmeticException waiting at scan time rather than one that disappeared with the expansion. DictFilter pattern-matches DictLongArray/DictIntArray with ByteArray codes and returns null for anything else, so encoding-level dicts newly reaching its fast path degrade to the generic lane rather than misbehaving; its tests pass unchanged. docs/compatibility.md already listed vortex.dict as Lazy — true of the layout path only. No edit needed; it is now true of both. Closes #336 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 8ff823e commit a4af152

3 files changed

Lines changed: 241 additions & 334 deletions

File tree

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
1111

1212
- A run-end-encoded Utf8/Binary column (`vortex.runend`) no longer expands every run into a fully materialized buffer on decode; rows now resolve through the runs lazily, removing an unbounded `sum(runLength * valueLength)` allocation that a crafted file could drive to `OutOfMemoryError`. ([#334](https://github.com/dfa1/vortex-java/issues/334))
1313
- A `vortex.sequence` column no longer materializes `base + i * multiplier` into a full buffer on decode; rows are computed on access, so the encoding allocates nothing regardless of row count — closing an `OutOfMemoryError` risk from a metadata-only encoding whose row count no buffer bounds. ([#335](https://github.com/dfa1/vortex-java/issues/335))
14+
- A primitive `vortex.dict` column decoded through the encoding path no longer expands its codes into an `n * elemSize` buffer; it now returns the same lazy `DictXxxArray` carriers the layout path already used, so a dict column keeps the dictionary's memory benefit however it is reached. ([#336](https://github.com/dfa1/vortex-java/issues/336))
1415

1516
## [0.13.1] — 2026-08-06
1617

0 commit comments

Comments
 (0)