Skip to content

Commit 8ff823e

Browse files
dfa1claude
andcommitted
fix(reader): make vortex.sequence decode lazy
vortex.sequence is a closed-form encoding — A[i] = base + i * multiplier, entirely in proto3 metadata, no buffers and no children. Every one of SequenceEncodingDecoder's four paths nevertheless allocated n * elemBytes and filled it. Because the encoding is metadata-only, n was bounded by nothing on disk. A buffer-backed encoding can at least be sanity-checked against its mmapped segment; here a few bytes of metadata plus a large declared row count named an arbitrary allocation, making the cheapest column in the file the easiest one to turn into an OutOfMemoryError (ADR 0003/0004). Adds the LazySequenceXxxArray family (Long/Int/Short/Byte/Float/Double/ Float16) mirroring LazyConstantXxxArray, and the decoder collapses to metadata parsing. Value semantics are preserved exactly rather than tidied, so decoded output stays bit-identical: - integers compute in long and narrow on read, so they still wrap; - fold zero-extends for U8/U16, matching Materialized{Byte,Short}Array; - F32 keeps single-precision arithmetic — a double accumulator would sharpen steps that are not exactly representable; - F16 round-trips through floatToFloat16 on every read, as the eager path did by storing half and widening on access. Float16Array declares no default limited/materialize, so the F16 carrier supplies both; materialize is now the only place the encoding can allocate, and only on demand. Also removes the per-element `switch (pt)` that sat inside decodeInteger's loop — the non-uniform body CLAUDE.md's hot-loop rule prohibits. It disappears with the loop itself. docs/compatibility.md listed vortex.sequence as "Zero-copy — synthetic (no data)", which was not true of the code; this makes the row accurate. The decoder had no unit coverage at all. Adds SequenceEncodingDecoderTest (all eight integer ptypes plus F32/F64/F16, asserting the concrete lazy type since correct values alone also pass on the eager path) and LazySequenceArrayTest (accessors, fold, unsigned widening, truncation, half-precision round-trip). Closes #335 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 9fcb189 commit 8ff823e

12 files changed

Lines changed: 861 additions & 75 deletions

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
1010
### Fixed
1111

1212
- A run-end-encoded Utf8/Binary column (`vortex.runend`) no longer expands every run into a fully materialized buffer on decode; rows now resolve through the runs lazily, removing an unbounded `sum(runLength * valueLength)` allocation that a crafted file could drive to `OutOfMemoryError`. ([#334](https://github.com/dfa1/vortex-java/issues/334))
13+
- A `vortex.sequence` column no longer materializes `base + i * multiplier` into a full buffer on decode; rows are computed on access, so the encoding allocates nothing regardless of row count — closing an `OutOfMemoryError` risk from a metadata-only encoding whose row count no buffer bounds. ([#335](https://github.com/dfa1/vortex-java/issues/335))
1314

1415
## [0.13.1] — 2026-08-06
1516

docs/compatibility.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -145,7 +145,7 @@ decoder falls into one of three shapes:
145145
| `vortex.alprd` | Lazy | Lazy | `LazyAlpRdDoubleArray`/`LazyAlpRdFloatArray` — left/right + patches on access |
146146
| `vortex.dict` | Lazy | Lazy | `DictXxxArray` (numeric) + `VarBinDictArray` (string), ADR 0012 |
147147
| `vortex.sparse` | Lazy | Lazy | `LazySparseXxxArray` (primitive + bool); Utf8/Binary stays Materialized, ADR 0015 |
148-
| `vortex.sequence` | Zero-copy | Zero-copy | synthetic (no data) |
148+
| `vortex.sequence` | Lazy | Lazy | `LazySequenceXxxArray`; `base + i * multiplier` per access, no buffer, ADR 0015 |
149149
| `vortex.struct` | Zero-copy | Zero-copy | `StructArray` wraps fields |
150150
| `vortex.chunked` | Lazy | Lazy | `ChunkedXxxArray` (primitive/Bool) + `VarBinChunkedArray` (Utf8/Binary), ADR 0012 |
151151
| `vortex.fsst` | Materialized | Materialized | symbol-table decompression |
Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
package io.github.dfa1.vortex.reader.array;
2+
3+
import io.github.dfa1.vortex.core.model.DType;
4+
import io.github.dfa1.vortex.core.model.PType;
5+
6+
import java.util.Objects;
7+
import java.util.function.LongBinaryOperator;
8+
9+
/// Metadata-only [ByteArray] for `vortex.sequence` columns: `A[i] = base + i * multiplier`.
10+
///
11+
/// The encoding carries no buffers at all — base and multiplier live in proto3 metadata — so
12+
/// every row is computable in O(1) and no allocation is needed regardless of row count.
13+
///
14+
/// The sum is computed in `long` and narrowed, matching the eager decode this replaces.
15+
/// [#fold(long, LongBinaryOperator)] zero-extends for U8 columns, like the buffer-backed
16+
/// `MaterializedByteArray`.
17+
///
18+
/// @param dtype logical primitive type (I8 / U8)
19+
/// @param length total logical row count
20+
/// @param base value at row 0
21+
/// @param multiplier step added per row
22+
public record LazySequenceByteArray(DType dtype, long length, long base, long multiplier)
23+
implements ByteArray {
24+
25+
@Override
26+
public byte getByte(long i) {
27+
Objects.checkIndex(i, length);
28+
return (byte) (base + i * multiplier);
29+
}
30+
31+
@Override
32+
public void forEachByte(ByteConsumer c) {
33+
long n = length;
34+
long b = base;
35+
long m = multiplier;
36+
for (long i = 0; i < n; i++) {
37+
c.accept((byte) (b + i * m));
38+
}
39+
}
40+
41+
@Override
42+
public long fold(long identity, LongBinaryOperator op) {
43+
boolean unsigned = dtype instanceof DType.Primitive p && p.ptype() == PType.U8;
44+
long n = length;
45+
long b = base;
46+
long m = multiplier;
47+
long acc = identity;
48+
for (long i = 0; i < n; i++) {
49+
byte raw = (byte) (b + i * m);
50+
acc = op.applyAsLong(acc, unsigned ? Byte.toUnsignedLong(raw) : raw);
51+
}
52+
return acc;
53+
}
54+
55+
/// Zero-copy truncation: the formula is unchanged for the leading rows, so only the
56+
/// row count shrinks.
57+
///
58+
/// @param rows number of leading rows to keep
59+
/// @return a length-`rows` sequence over the same base and multiplier
60+
@Override
61+
public Array limited(long rows) {
62+
return rows >= length ? this : new LazySequenceByteArray(dtype, rows, base, multiplier);
63+
}
64+
}
Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
package io.github.dfa1.vortex.reader.array;
2+
3+
import io.github.dfa1.vortex.core.model.DType;
4+
5+
import java.util.Objects;
6+
import java.util.function.DoubleBinaryOperator;
7+
import java.util.function.DoubleConsumer;
8+
9+
/// Metadata-only [DoubleArray] for `vortex.sequence` columns: `A[i] = base + i * multiplier`.
10+
///
11+
/// The encoding carries no buffers at all — base and multiplier live in proto3 metadata — so
12+
/// every row is computable in O(1) and no allocation is needed regardless of row count.
13+
///
14+
/// The row index is converted to `double` before the multiply, matching the eager decode this
15+
/// replaces — so results stay bit-identical, including the precision loss past 2^53 rows.
16+
///
17+
/// @param dtype logical primitive type (F64)
18+
/// @param length total logical row count
19+
/// @param base value at row 0
20+
/// @param multiplier step added per row
21+
public record LazySequenceDoubleArray(DType dtype, long length, double base, double multiplier)
22+
implements DoubleArray {
23+
24+
@Override
25+
public double getDouble(long i) {
26+
Objects.checkIndex(i, length);
27+
return base + i * multiplier;
28+
}
29+
30+
@Override
31+
public void forEachDouble(DoubleConsumer c) {
32+
long n = length;
33+
double b = base;
34+
double m = multiplier;
35+
for (long i = 0; i < n; i++) {
36+
c.accept(b + i * m);
37+
}
38+
}
39+
40+
@Override
41+
public double fold(double identity, DoubleBinaryOperator op) {
42+
long n = length;
43+
double b = base;
44+
double m = multiplier;
45+
double acc = identity;
46+
for (long i = 0; i < n; i++) {
47+
acc = op.applyAsDouble(acc, b + i * m);
48+
}
49+
return acc;
50+
}
51+
52+
/// Zero-copy truncation: the formula is unchanged for the leading rows, so only the
53+
/// row count shrinks.
54+
///
55+
/// @param rows number of leading rows to keep
56+
/// @return a length-`rows` sequence over the same base and multiplier
57+
@Override
58+
public Array limited(long rows) {
59+
return rows >= length ? this : new LazySequenceDoubleArray(dtype, rows, base, multiplier);
60+
}
61+
}
Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
package io.github.dfa1.vortex.reader.array;
2+
3+
import io.github.dfa1.vortex.core.io.VortexFormat;
4+
import io.github.dfa1.vortex.core.model.DType;
5+
6+
import java.lang.foreign.MemorySegment;
7+
import java.lang.foreign.SegmentAllocator;
8+
import java.util.Objects;
9+
10+
/// Metadata-only [Float16Array] for `vortex.sequence` columns: `A[i] = base + i * multiplier`.
11+
///
12+
/// The encoding carries no buffers at all — base and multiplier live in proto3 metadata — so
13+
/// every row is computable in O(1) and no allocation is needed regardless of row count.
14+
///
15+
/// The step is computed in `float` and round-tripped through half precision on every read, so
16+
/// values match the eager decode this replaces bit for bit: it wrote
17+
/// [Float#floatToFloat16(float)] into the buffer and widened again on access.
18+
///
19+
/// @param dtype logical primitive type (F16)
20+
/// @param length total logical row count
21+
/// @param base value at row 0, already widened from half precision
22+
/// @param multiplier step added per row, already widened from half precision
23+
public record LazySequenceFloat16Array(DType dtype, long length, float base, float multiplier)
24+
implements Float16Array {
25+
26+
@Override
27+
public float getFloat(long i) {
28+
Objects.checkIndex(i, length);
29+
return Float.float16ToFloat(Float.floatToFloat16(base + i * multiplier));
30+
}
31+
32+
/// Zero-copy truncation: the formula is unchanged for the leading rows, so only the
33+
/// row count shrinks.
34+
///
35+
/// @param rows number of leading rows to keep
36+
/// @return a length-`rows` sequence over the same base and multiplier
37+
@Override
38+
public Array limited(long rows) {
39+
return rows >= length ? this : new LazySequenceFloat16Array(dtype, rows, base, multiplier);
40+
}
41+
42+
/// Materializes the sequence into a fresh little-endian half-precision segment.
43+
/// [Float16Array] declares no default, and this is the only allocation the encoding
44+
/// performs — on demand, for consumers that need a contiguous buffer.
45+
///
46+
/// @param arena allocator for the output segment
47+
/// @return a little-endian `f16` segment of `length()` elements
48+
@Override
49+
public MemorySegment materialize(SegmentAllocator arena) {
50+
long n = length;
51+
float b = base;
52+
float m = multiplier;
53+
MemorySegment dst = arena.allocate(n * 2L, 2);
54+
for (long i = 0; i < n; i++) {
55+
dst.setAtIndex(VortexFormat.LE_SHORT, i, Float.floatToFloat16(b + i * m));
56+
}
57+
return dst;
58+
}
59+
}
Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
package io.github.dfa1.vortex.reader.array;
2+
3+
import io.github.dfa1.vortex.core.model.DType;
4+
5+
import java.util.Objects;
6+
import java.util.function.DoubleBinaryOperator;
7+
8+
/// Metadata-only [FloatArray] for `vortex.sequence` columns: `A[i] = base + i * multiplier`.
9+
///
10+
/// The encoding carries no buffers at all — base and multiplier live in proto3 metadata — so
11+
/// every row is computable in O(1) and no allocation is needed regardless of row count.
12+
///
13+
/// The arithmetic stays in `float`, matching the eager decode this replaces — so results stay
14+
/// bit-identical rather than being sharpened by a wider accumulator.
15+
///
16+
/// @param dtype logical primitive type (F32)
17+
/// @param length total logical row count
18+
/// @param base value at row 0
19+
/// @param multiplier step added per row
20+
public record LazySequenceFloatArray(DType dtype, long length, float base, float multiplier)
21+
implements FloatArray {
22+
23+
@Override
24+
public float getFloat(long i) {
25+
Objects.checkIndex(i, length);
26+
return base + i * multiplier;
27+
}
28+
29+
@Override
30+
public double fold(double identity, DoubleBinaryOperator op) {
31+
long n = length;
32+
float b = base;
33+
float m = multiplier;
34+
double acc = identity;
35+
for (long i = 0; i < n; i++) {
36+
acc = op.applyAsDouble(acc, b + i * m);
37+
}
38+
return acc;
39+
}
40+
41+
/// Zero-copy truncation: the formula is unchanged for the leading rows, so only the
42+
/// row count shrinks.
43+
///
44+
/// @param rows number of leading rows to keep
45+
/// @return a length-`rows` sequence over the same base and multiplier
46+
@Override
47+
public Array limited(long rows) {
48+
return rows >= length ? this : new LazySequenceFloatArray(dtype, rows, base, multiplier);
49+
}
50+
}
Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
package io.github.dfa1.vortex.reader.array;
2+
3+
import io.github.dfa1.vortex.core.model.DType;
4+
5+
import java.util.Objects;
6+
import java.util.function.IntBinaryOperator;
7+
import java.util.function.IntConsumer;
8+
9+
/// Metadata-only [IntArray] for `vortex.sequence` columns: `A[i] = base + i * multiplier`.
10+
///
11+
/// The encoding carries no buffers at all — base and multiplier live in proto3 metadata — so
12+
/// every row is computable in O(1) and no allocation is needed regardless of row count.
13+
///
14+
/// The sum is computed in `long` and narrowed, matching the eager decode this replaces.
15+
///
16+
/// @param dtype logical primitive type (I32 / U32)
17+
/// @param length total logical row count
18+
/// @param base value at row 0
19+
/// @param multiplier step added per row
20+
public record LazySequenceIntArray(DType dtype, long length, long base, long multiplier)
21+
implements IntArray {
22+
23+
@Override
24+
public int getInt(long i) {
25+
Objects.checkIndex(i, length);
26+
return (int) (base + i * multiplier);
27+
}
28+
29+
@Override
30+
public void forEachInt(IntConsumer c) {
31+
long n = length;
32+
long b = base;
33+
long m = multiplier;
34+
for (long i = 0; i < n; i++) {
35+
c.accept((int) (b + i * m));
36+
}
37+
}
38+
39+
@Override
40+
public int fold(int identity, IntBinaryOperator op) {
41+
long n = length;
42+
long b = base;
43+
long m = multiplier;
44+
int acc = identity;
45+
for (long i = 0; i < n; i++) {
46+
acc = op.applyAsInt(acc, (int) (b + i * m));
47+
}
48+
return acc;
49+
}
50+
51+
/// Zero-copy truncation: the formula is unchanged for the leading rows, so only the
52+
/// row count shrinks.
53+
///
54+
/// @param rows number of leading rows to keep
55+
/// @return a length-`rows` sequence over the same base and multiplier
56+
@Override
57+
public Array limited(long rows) {
58+
return rows >= length ? this : new LazySequenceIntArray(dtype, rows, base, multiplier);
59+
}
60+
}
Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
package io.github.dfa1.vortex.reader.array;
2+
3+
import io.github.dfa1.vortex.core.model.DType;
4+
5+
import java.util.Objects;
6+
import java.util.function.LongBinaryOperator;
7+
import java.util.function.LongConsumer;
8+
9+
/// Metadata-only [LongArray] for `vortex.sequence` columns: `A[i] = base + i * multiplier`.
10+
///
11+
/// The encoding carries no buffers at all — base and multiplier live in proto3 metadata — so
12+
/// every row is computable in O(1) and no allocation is needed regardless of row count.
13+
///
14+
/// Arithmetic is `long` and wraps on overflow, matching the eager decode this replaces.
15+
///
16+
/// @param dtype logical primitive type (I64 / U64)
17+
/// @param length total logical row count
18+
/// @param base value at row 0
19+
/// @param multiplier step added per row
20+
public record LazySequenceLongArray(DType dtype, long length, long base, long multiplier)
21+
implements LongArray {
22+
23+
@Override
24+
public long getLong(long i) {
25+
Objects.checkIndex(i, length);
26+
return base + i * multiplier;
27+
}
28+
29+
@Override
30+
public void forEachLong(LongConsumer c) {
31+
long n = length;
32+
long b = base;
33+
long m = multiplier;
34+
for (long i = 0; i < n; i++) {
35+
c.accept(b + i * m);
36+
}
37+
}
38+
39+
@Override
40+
public long fold(long identity, LongBinaryOperator op) {
41+
long n = length;
42+
long b = base;
43+
long m = multiplier;
44+
long acc = identity;
45+
for (long i = 0; i < n; i++) {
46+
acc = op.applyAsLong(acc, b + i * m);
47+
}
48+
return acc;
49+
}
50+
51+
/// Zero-copy truncation: the formula is unchanged for the leading rows, so only the
52+
/// row count shrinks.
53+
///
54+
/// @param rows number of leading rows to keep
55+
/// @return a length-`rows` sequence over the same base and multiplier
56+
@Override
57+
public Array limited(long rows) {
58+
return rows >= length ? this : new LazySequenceLongArray(dtype, rows, base, multiplier);
59+
}
60+
}

0 commit comments

Comments
 (0)