Skip to content

Commit a326fbb

Browse files
dfa1claude
andcommitted
docs(adr): flesh out Option B C-Data detail in ADR 0016
Add concrete Arrow C-Data Interface requirements: ArrowSchema/ArrowArray struct fields, per-Array-subtype format codes + buffer order, and the three work items (validity bitmap packing, lazy materialise, lifetime/release pin). Record the relationship to a standalone Array.materialize(arena) delegation seam, which would produce the Arrow values buffer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 27446d8 commit a326fbb

1 file changed

Lines changed: 73 additions & 0 deletions

File tree

docs/adr/0016-vortex-arrow-bridge.md

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,79 @@ dependency entirely. The consumer imports via any Arrow C-Data-aware runtime.
6969
structs and lifetime/release-callback management over FFM — significant complexity;
7070
validity/offset buffer layout must match Arrow's spec exactly.
7171

72+
#### Option B — concrete C-Data requirements
73+
74+
The interface is two C structs filled via FFM `MemoryLayout.structLayout(...)`:
75+
76+
- `ArrowSchema``format` (type-code string), `name`, `metadata`, `flags`
77+
(`ARROW_FLAG_NULLABLE = 2`), `n_children`, `children`, `dictionary`, `release`,
78+
`private_data`.
79+
- `ArrowArray``length`, `null_count` (`-1` = unknown), `offset`, `n_buffers`,
80+
`n_children`, `buffers` (array of pointers), `children`, `dictionary`, `release`,
81+
`private_data`.
82+
83+
Buffer count and order per Vortex `Array` subtype (validity is always `buffers[0]`):
84+
85+
| Vortex Array | `format` | n_buffers | buffers (in order) |
86+
|--------------------------|----------------|-----------|------------------------------------|
87+
| `LongArray` (I64/U64) | `l` / `L` | 2 | validity, values |
88+
| `IntArray` | `i` / `I` | 2 | validity, values |
89+
| `ShortArray` | `s` / `S` | 2 | validity, values |
90+
| `ByteArray` | `c` / `C` | 2 | validity, values |
91+
| `DoubleArray` | `g` | 2 | validity, values |
92+
| `FloatArray` | `f` | 2 | validity, values |
93+
| `Float16Array` | `e` | 2 | validity, values |
94+
| `BoolArray` | `b` | 2 | validity, data (packed bitmap) |
95+
| `VarBinArray.OffsetMode` | `u` / `z` | 3 | validity, offsets (i32), data |
96+
| (large offsets) | `U` / `Z` | 3 | validity, offsets (i64), data |
97+
| `VarBinArray.ViewMode` | `vu` / `vz` | ≥3 | validity, views (16B), data buf(s) |
98+
| `DecimalArray` | `d:P,S[,bits]` | 2 | validity, values (fixed width) |
99+
| `StructArray` | `+s` | 1 | validity; children = fields |
100+
| `ListArray` | `+l` / `+L` | 2 | validity, offsets; 1 child |
101+
| `FixedSizeListArray` | `+w:N` | 1 | validity; 1 child |
102+
| `Dict*` | indices format | 2 | validity, codes; `dictionary` → values |
103+
| `NullArray` | `n` | 0 ||
104+
105+
Extension types (date/time/timestamp/uuid) map either to a native Arrow code
106+
(`tdD` date32, `tsu:tz` timestamp) or to the storage type plus an
107+
`ARROW:extension:name` entry in `metadata`.
108+
109+
Three pieces of work beyond just handing over the existing mmap slices:
110+
111+
1. **Validity bitmap.** Arrow wants a packed LSB bitmap (`1 = valid`, byte-aligned,
112+
length rounded up to 8) plus a `null_count`. Our `MaskedArray.validity()` is a
113+
per-element `BoolArray`, so it must be packed; `validity == null` emits a null
114+
buffer pointer with `null_count = 0`.
115+
2. **Lazy materialisation.** Lazy arrays (ZigZag/FoR/ALP/Dict/RLE) store the
116+
*encoded* form, which is not the Arrow values layout, so they must be materialised
117+
into a contiguous LE segment first. This is exactly the producer step that
118+
`ArraySegments.of(...)` (or a future `Array.materialize(arena)` delegation seam,
119+
see below) performs, so the internal materialise path feeds the `values` buffer
120+
directly. Primitive values, VarBin data+offsets, and StringView are already
121+
Arrow-shaped (zero-copy).
122+
3. **Lifetime / release contract.** Buffers are zero-copy slices of the mmap'd file
123+
(lifetime = `VortexReader` arena), but the Arrow consumer outlives the producing
124+
stack frame. `private_data` must hold a strong reference pinning the arena alive;
125+
the `release` callback (an FFM upcall stub) drops that reference, frees the
126+
malloc'd pointer arrays, and sets `release = NULL`. Freeing the arena before the
127+
consumer calls `release` is a use-after-unmap → native segfault, not a Java
128+
exception. This is the highest-risk part.
129+
130+
### Relationship to the internal materialise seam
131+
132+
`ArraySegments.of(Array, SegmentAllocator)` already centralises "turn any array
133+
(lazy or eager) into a contiguous LE primitive segment", and currently re-states each
134+
encoding's decode formula (ZigZag/FoR/ALP) in a large switch separate from the
135+
per-element accessor on the lazy array. A standalone refactor — moving that bulk
136+
materialisation onto the array types as an `Array.materialize(SegmentAllocator)`
137+
delegation (mirroring the existing `Array.limited(...)` pattern, kept on a
138+
package-private seam to avoid widening the public API) — stands on its own as a
139+
locality cleanup. It is **not** an Arrow feature, but it is the natural producer of
140+
the Arrow `values` buffer, so Option B should build on it rather than duplicate it.
141+
The contiguous LE segment it yields already matches Arrow's primitive values-buffer
142+
layout; the gap to a full Arrow array is validity + offsets + children, per the table
143+
above.
144+
72145
### Option C — No bridge; document manual conversion
73146

74147
Ship nothing; point users at the typed views and let them copy into Arrow themselves.

0 commit comments

Comments
 (0)