Skip to content

Commit 0311416

Browse files
dfa1claude
andcommitted
docs: explain and document the vortex.map layout
- docs/explanation.md: new "Map column layout" subsection alongside the existing layout-tree examples, diagramming vortex.map's array-encoding cascade (map -> bare listview -> struct{key,value} -> per-field encodings, offsets/sizes, optional validity) and the two independent nullability slots (null map row vs. null entry value) that are easy to conflate. - docs/how-to.md: new "Write and read a Map column" recipe with a compiling write example (ListViewData(StructData(...)) is the write-side value shape for a map column) and read example (MapArray -> unwrap MaskedArray if nullable -> ListViewArray -> StructArray fields by name). Cross-linked between the two files and to reference.md#core-types. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
1 parent ab0b5e0 commit 0311416

2 files changed

Lines changed: 132 additions & 0 deletions

File tree

docs/explanation.md

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -292,6 +292,50 @@ Low-cardinality string column with dict layout:
292292
└─ codes: Flat → SegmentSpec → fastlanes.bitpacked (one code per row)
293293
```
294294

295+
### Map column layout
296+
297+
`vortex.map` (`DType.Map`) is a good illustration of how an array encoding's own children
298+
cascade below a single `Flat` leaf — unlike the plain-primitive and dict examples above, its
299+
tree has real depth. Logically, a map column stores a variable-length list of `{key, value}`
300+
entries per row; physically, that's exactly a `ListView<Struct{key, value}>` wearing a
301+
`vortex.map` label:
302+
303+
```
304+
Flat → SegmentSpec → vortex.map (0 buffers, no metadata — 1 child: entries)
305+
└─ entries: vortex.listview (must be a *bare* listview — see below)
306+
├─ elements: vortex.struct (non-nullable {key, value} entry structs)
307+
│ ├─ "key": vortex.varbin | vortex.dict | ... (any encoding accepting keyType)
308+
│ └─ "value": vortex.masked(...) when valueType is nullable, else same as key
309+
├─ offsets: vortex.primitive (i32, one per map row — start index into elements)
310+
├─ sizes: vortex.primitive (i32, one per map row — entry count)
311+
└─ [validity: vortex.bool] (present only when the map itself is nullable)
312+
```
313+
314+
`vortex.map` itself carries zero buffers and no metadata — every bit of information about a
315+
map column lives either in its `DType.Map(keyType, valueType, keysSorted, nullable)` schema
316+
(a schema-only producer assertion for `keysSorted`, never checked against the data) or in the
317+
single `entries` child.
318+
319+
**Two independent nullability slots, easy to conflate:**
320+
321+
- **A null *map row*** (`DType.Map(..., nullable=true)`) has no representation on the
322+
`vortex.map` node at all — it's delegated entirely to the `entries` child's own validity,
323+
carried in `vortex.listview`'s optional fourth child slot (a `vortex.bool` bitmap). This is
324+
why `entries` must be a *bare* `vortex.listview`, never wrapped in a `vortex.masked` — a
325+
masked wrapper would give a nullable map two different on-disk representations for the same
326+
logical value, so both the Rust reference and vortex-java reject it. Every other nullable
327+
`DType.List` column not underneath a map is unaffected by this and still wraps in
328+
`vortex.masked` + `vortex.list` as before.
329+
- **A null *entry value*** (`DType.Map(key, value.asNullable(), ...)`, e.g. `{a: 1, b: null}`
330+
inside an otherwise-present map row) is a completely different bit: it rides the entry
331+
struct's own `value` field validity — ordinary `vortex.masked` wrapping, exactly like a
332+
nullable field anywhere else inside a `vortex.struct`. It has nothing to do with the map row
333+
being present or absent.
334+
335+
See [how-to.md#write-and-read-a-map-column](how-to.md#write-and-read-a-map-column) for the
336+
write/read Java API this shape maps to, and `docs/reference.md#core-types` for `DType.Map`'s
337+
field list.
338+
295339
### Pruning by zone maps
296340

297341
`vortex.stats` is the pruning hook. At scan time, when `ScanOptions` carries a

docs/how-to.md

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -293,6 +293,94 @@ java -jar cli/target/vortex-cli-*-all.jar filter data.vortex "price >= 100" > ou
293293
294294
---
295295
296+
## Write and read a Map column
297+
298+
`DType.Map` has no dedicated write-side value type. Physically, a map column's `vortex.map`
299+
node has exactly one child — `entries`, a `ListView<Struct{key, value}>` — so you hand
300+
`writeChunk` the same shape you'd hand a plain `ListView<Struct>` column: a `ListViewData`
301+
whose `elements` is a `StructData` of a keys array and a values array. See
302+
[explanation.md#map-column-layout](explanation.md#map-column-layout) for why the wire format
303+
looks like this and how the two independent nullability slots (map row vs. entry value) work.
304+
305+
**Write:**
306+
307+
```java
308+
import io.github.dfa1.vortex.core.model.ColumnName;
309+
import io.github.dfa1.vortex.core.model.DType;
310+
import io.github.dfa1.vortex.writer.encode.ListViewData;
311+
import io.github.dfa1.vortex.writer.encode.NullableData;
312+
import io.github.dfa1.vortex.writer.encode.StructData;
313+
314+
// map<utf8, i64?> — non-nullable string keys, nullable long values
315+
DType.Map mapType = new DType.Map(DType.UTF8, DType.I64.asNullable(), false, false);
316+
DType.Struct schema = new DType.Struct(List.of(ColumnName.of("attrs")), List.of(mapType), false);
317+
318+
// 3 rows: {a:1, b:2}, {} (empty map), {c:null}
319+
String[] keys = {"a", "b", "c"};
320+
long[] values = {1L, 2L, 0L}; // placeholder at the null entry
321+
boolean[] valueValidity = {true, true, false}; // per-entry value validity
322+
StructData entryStructs = new StructData(List.of(keys, new NullableData(values, valueValidity)));
323+
324+
int[] offsets = {0, 2, 2}; // row i's entries start at entryStructs[offsets[i]]
325+
int[] sizes = {2, 0, 1}; // row i has sizes[i] entries
326+
ListViewData column = new ListViewData(entryStructs, offsets, sizes, 3);
327+
328+
try (var ch = FileChannel.open(Path.of("attrs.vortex"), StandardOpenOption.CREATE, StandardOpenOption.WRITE);
329+
var writer = VortexWriter.create(ch, schema, WriteOptions.defaults())) {
330+
writer.writeChunk(Map.of(ColumnName.of("attrs"), column));
331+
}
332+
```
333+
334+
A *nullable map row* (as opposed to a nullable value inside a present map) wraps the whole
335+
`ListViewData` in `NullableData` instead — `mapType.asNullable()` in the schema, and
336+
`new NullableData(column, new boolean[]{true, false, true})` in place of `column` above.
337+
338+
**Read:**
339+
340+
```java
341+
import io.github.dfa1.vortex.reader.array.IntArray;
342+
import io.github.dfa1.vortex.reader.array.ListViewArray;
343+
import io.github.dfa1.vortex.reader.array.MapArray;
344+
import io.github.dfa1.vortex.reader.array.MaskedArray;
345+
import io.github.dfa1.vortex.reader.array.StructArray;
346+
import io.github.dfa1.vortex.reader.array.VarBinArray;
347+
348+
try (var reader = VortexReader.open(Path.of("attrs.vortex"));
349+
var iter = reader.scan(ScanOptions.all())) {
350+
while (iter.hasNext()) {
351+
try (var chunk = iter.next()) {
352+
MapArray map = chunk.column("attrs");
353+
354+
// If the map itself is nullable, entries() is a MaskedArray; unwrap it first.
355+
var entries = map.entries() instanceof MaskedArray masked
356+
? (ListViewArray) masked.inner() : (ListViewArray) map.entries();
357+
StructArray entryStructs = (StructArray) entries.elements();
358+
VarBinArray keys = (VarBinArray) entryStructs.field("key");
359+
var values = entryStructs.field("value"); // MaskedArray, since the value type is nullable here
360+
// A file written by vortex-java's own writer always emits I32 offsets/sizes; a file
361+
// from another producer (e.g. the Rust reference) may pick a narrower or wider integer
362+
// width, so switch on the concrete Array subtype there instead of casting to IntArray.
363+
IntArray offsets = (IntArray) entries.offsets();
364+
IntArray sizes = (IntArray) entries.sizes();
365+
366+
for (long row = 0; row < map.length(); row++) {
367+
long start = offsets.getInt(row);
368+
long end = start + sizes.getInt(row);
369+
for (long i = start; i < end; i++) {
370+
// keys.getBytes(i) / values at index i are this row's i-th {key, value} pair
371+
}
372+
}
373+
}
374+
}
375+
}
376+
```
377+
378+
`ScanOptions.all()`/CLI `inspect` show `vortex.map` in a file's layout tree as a `vortex.listview`
379+
child under the `vortex.map` node — see `docs/reference.md#core-types` for `DType.Map`'s full
380+
field list (`keyType`, `valueType`, `keysSorted`, `nullable`) and `entriesDtype()`.
381+
382+
---
383+
296384
## Read files with unknown encodings
297385
298386
By default, a file containing an unrecognized encoding ID throws `VortexException`.

0 commit comments

Comments
 (0)