@@ -69,6 +69,79 @@ dependency entirely. The consumer imports via any Arrow C-Data-aware runtime.
6969 structs and lifetime/release-callback management over FFM — significant complexity;
7070 validity/offset buffer layout must match Arrow's spec exactly.
7171
72+ #### Option B — concrete C-Data requirements
73+
74+ The interface is two C structs filled via FFM ` MemoryLayout.structLayout(...) ` :
75+
76+ - ` ArrowSchema ` — ` format ` (type-code string), ` name ` , ` metadata ` , ` flags `
77+ (` ARROW_FLAG_NULLABLE = 2 ` ), ` n_children ` , ` children ` , ` dictionary ` , ` release ` ,
78+ ` private_data ` .
79+ - ` ArrowArray ` — ` length ` , ` null_count ` (` -1 ` = unknown), ` offset ` , ` n_buffers ` ,
80+ ` n_children ` , ` buffers ` (array of pointers), ` children ` , ` dictionary ` , ` release ` ,
81+ ` private_data ` .
82+
83+ Buffer count and order per Vortex ` Array ` subtype (validity is always ` buffers[0] ` ):
84+
85+ | Vortex Array | ` format ` | n_buffers | buffers (in order) |
86+ | --------------------------| ----------------| -----------| ------------------------------------|
87+ | ` LongArray ` (I64/U64) | ` l ` / ` L ` | 2 | validity, values |
88+ | ` IntArray ` | ` i ` / ` I ` | 2 | validity, values |
89+ | ` ShortArray ` | ` s ` / ` S ` | 2 | validity, values |
90+ | ` ByteArray ` | ` c ` / ` C ` | 2 | validity, values |
91+ | ` DoubleArray ` | ` g ` | 2 | validity, values |
92+ | ` FloatArray ` | ` f ` | 2 | validity, values |
93+ | ` Float16Array ` | ` e ` | 2 | validity, values |
94+ | ` BoolArray ` | ` b ` | 2 | validity, data (packed bitmap) |
95+ | ` VarBinArray.OffsetMode ` | ` u ` / ` z ` | 3 | validity, offsets (i32), data |
96+ | (large offsets) | ` U ` / ` Z ` | 3 | validity, offsets (i64), data |
97+ | ` VarBinArray.ViewMode ` | ` vu ` / ` vz ` | ≥3 | validity, views (16B), data buf(s) |
98+ | ` DecimalArray ` | ` d:P,S[,bits] ` | 2 | validity, values (fixed width) |
99+ | ` StructArray ` | ` +s ` | 1 | validity; children = fields |
100+ | ` ListArray ` | ` +l ` / ` +L ` | 2 | validity, offsets; 1 child |
101+ | ` FixedSizeListArray ` | ` +w:N ` | 1 | validity; 1 child |
102+ | ` Dict* ` | indices format | 2 | validity, codes; ` dictionary ` → values |
103+ | ` NullArray ` | ` n ` | 0 | — |
104+
105+ Extension types (date/time/timestamp/uuid) map either to a native Arrow code
106+ (` tdD ` date32, ` tsu:tz ` timestamp) or to the storage type plus an
107+ ` ARROW:extension:name ` entry in ` metadata ` .
108+
109+ Three pieces of work beyond just handing over the existing mmap slices:
110+
111+ 1 . ** Validity bitmap.** Arrow wants a packed LSB bitmap (` 1 = valid ` , byte-aligned,
112+ length rounded up to 8) plus a ` null_count ` . Our ` MaskedArray.validity() ` is a
113+ per-element ` BoolArray ` , so it must be packed; ` validity == null ` emits a null
114+ buffer pointer with ` null_count = 0 ` .
115+ 2 . ** Lazy materialisation.** Lazy arrays (ZigZag/FoR/ALP/Dict/RLE) store the
116+ * encoded* form, which is not the Arrow values layout, so they must be materialised
117+ into a contiguous LE segment first. This is exactly the producer step that
118+ ` ArraySegments.of(...) ` (or a future ` Array.materialize(arena) ` delegation seam,
119+ see below) performs, so the internal materialise path feeds the ` values ` buffer
120+ directly. Primitive values, VarBin data+offsets, and StringView are already
121+ Arrow-shaped (zero-copy).
122+ 3 . ** Lifetime / release contract.** Buffers are zero-copy slices of the mmap'd file
123+ (lifetime = ` VortexReader ` arena), but the Arrow consumer outlives the producing
124+ stack frame. ` private_data ` must hold a strong reference pinning the arena alive;
125+ the ` release ` callback (an FFM upcall stub) drops that reference, frees the
126+ malloc'd pointer arrays, and sets ` release = NULL ` . Freeing the arena before the
127+ consumer calls ` release ` is a use-after-unmap → native segfault, not a Java
128+ exception. This is the highest-risk part.
129+
130+ ### Relationship to the internal materialise seam
131+
132+ ` ArraySegments.of(Array, SegmentAllocator) ` already centralises "turn any array
133+ (lazy or eager) into a contiguous LE primitive segment", and currently re-states each
134+ encoding's decode formula (ZigZag/FoR/ALP) in a large switch separate from the
135+ per-element accessor on the lazy array. A standalone refactor — moving that bulk
136+ materialisation onto the array types as an ` Array.materialize(SegmentAllocator) `
137+ delegation (mirroring the existing ` Array.limited(...) ` pattern, kept on a
138+ package-private seam to avoid widening the public API) — stands on its own as a
139+ locality cleanup. It is ** not** an Arrow feature, but it is the natural producer of
140+ the Arrow ` values ` buffer, so Option B should build on it rather than duplicate it.
141+ The contiguous LE segment it yields already matches Arrow's primitive values-buffer
142+ layout; the gap to a full Arrow array is validity + offsets + children, per the table
143+ above.
144+
72145### Option C — No bridge; document manual conversion
73146
74147Ship nothing; point users at the typed views and let them copy into Arrow themselves.
0 commit comments