Skip to content

Commit cf73887

Browse files
dfa1claude
andcommitted
docs(compatibility): bump reference to v0.74.0, document Union/onpair/Variant gaps
The Rust reference implementation moved from v0.72.0 to v0.74.0 (two minor releases) while this doc continued to claim v0.72.0 as the tested ceiling. Audit findings against current Java code: - `DType::Union` (`fbs.DType.Type.Union = 12`, added in Rust 0.71.0) is not decoded — `PostscriptParser.convertDType` switch has no Union case and falls through to the default branch, throwing `VortexException("unsupported DType typeType=12")`. The sealed `DType` interface in core has no `Union` variant either, so adding support is a multi-touch change. - `vortex.onpair` (experimental in Rust 0.74.0, encodings/onpair/) is not registered. Files using it require `Registry.allowUnknown()` to open. - `vortex.variant`: decode is complete (incl. shredded child); encode still throws `"encode not yet implemented"`. Rust 0.73.0 (#7945) added their write path + parquet-variant IO tests, widening the asymmetry. Adds a new "Known wire-format gaps" section above the encoding table for visibility; adds a row for `vortex.onpair`; rewords the Variant row to call out the 0.73+ write-side asymmetry; flags the S3 fixture matrix as locked to v0.72.0 and in need of a re-run when the v0.74.0 fixture set publishes. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
1 parent 42177ca commit cf73887

1 file changed

Lines changed: 16 additions & 2 deletions

File tree

docs/compatibility.md

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,17 @@
11
# Compatibility
22

3-
Tested against the [Rust reference implementation](https://github.com/vortex-data/vortex) v0.72.0.
3+
Tested against the [Rust reference implementation](https://github.com/vortex-data/vortex) v0.74.0.
44
For the rest of the API surface (reader, writer, scan, CLI), see [reference.md](reference.md).
55

6+
## Known wire-format gaps
7+
8+
| Item | Introduced | Java status |
9+
|------|------------|-------------|
10+
| `DType::Union` (`fbs.DType.Type.Union = 12`) | Rust 0.71.0 | ❌ Decode throws `VortexException("unsupported DType typeType=12")`. No `DType.Union` variant in Java's sealed type. |
11+
| `vortex.onpair` experimental string encoding | Rust 0.74.0 | ❌ Not registered. Files using it fail to decode unless `Registry.allowUnknown()` is enabled. |
12+
| `vortex.variant` write path | Rust 0.73.0 (`Allow writing Variant to files`, #7945) | ❌ Java decode works; Java encode throws `"encode not yet implemented"`. Java→Rust round-trip not possible for Variant columns. |
13+
| Arrow extension array import affecting Variant shape | Rust 0.74.0 (#8125) | Untested. Re-run integration fixtures against v0.74.0 once published. |
14+
615
## Encodings
716

817
| Encoding ID | Class | Decode | Encode | Notes |
@@ -39,7 +48,8 @@ For the rest of the API surface (reader, writer, scan, CLI), see [reference.md](
3948
| `fastlanes.for` | `FrameOfReferenceEncoding` ||| Integer PTypes |
4049
| `fastlanes.rle` | `RleEncoding` ||| Chunk-based RLE |
4150
| `vortex.patched` | `PatchedEncoding` ||| Primitive PTypes; encode not yet implemented |
42-
| `vortex.variant` | `VariantEncoding` ||| Encode not yet implemented |
51+
| `vortex.variant` | `VariantEncoding` ||| Decode (incl. shredded child); encode not yet implemented (Rust 0.73+) |
52+
| `vortex.onpair` | _none_ ||| Experimental in Rust 0.74.0; not yet ported |
4353

4454
### Unknown encodings
4555

@@ -123,6 +133,10 @@ themselves.
123133

124134
## S3 Fixture Status (v0.72.0)
125135

136+
> **Note:** the fixture matrix below is locked to `v0.72.0/`. The Rust reference is
137+
> now at `v0.74.0`; re-run the integration suite against `v0.74.0/arrays/` once
138+
> upstream publishes the corresponding fixture set, and refresh this section.
139+
126140
Cross-language round-trips tested against Rust-written fixture files hosted at
127141
`s3://vortex-compat-fixtures/v0.72.0/arrays/`.
128142

0 commit comments

Comments
 (0)