Add arrow import/export for vortex-json#8339
Conversation
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
AdamGS
force-pushed
the
adamg/json-arrow-export
branch
from
eac53a7 to
5430d3a
Compare
Merging this PR will degrade performance by 11.93%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | bitwise_not_vortex_buffer_mut[128] |
215.3 ns | 244.4 ns | -11.93% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing adamg/json-arrow-export (a9ded61) with develop (a289c23)
| return Ok(None); | ||
| } | ||
|
|
||
| let mut field = Field::new(name, DataType::Utf8, dtype.is_nullable()); |
There was a problem hiding this comment.
do we always want Utf8? why not stringview?
in my WKB branch I just delegated this to the session
| return Ok(None); | ||
| } | ||
|
|
||
| let storage_dtype = DType::from_arrow(field); |
There was a problem hiding this comment.
we should not use DType::from_arrow anymore. should probably be deprecated actually. I think you can just use Utf8 directly in this case?
| return Ok(ArrowImport::Unsupported(array)); | ||
| } | ||
|
|
||
| let storage = ArrayRef::from_arrow(array.as_ref(), field.is_nullable())?; |
There was a problem hiding this comment.
we should deprecate this as well and use the session
There was a problem hiding this comment.
mind if I do it in a followup? I'll need to wire the session here
3 participants
Summary
Adds import/export from
vortex-jsonto Arrow's JSON canonical extension type.