[Java] VectorSchemaRoot convertion into ArrowArrayStream

[igor-suhorukov]

Describe the enhancement requested

DuckDB support ArrowArrayStream as virtual table. JDBC driver of DuckDB has "register" ArrowStream method on DuckDBConnection.

VectorSchemaRoot convertion into ArrowArrayStream object will allows perform complex SQL transformation with CTE,window function on existing Arrow data from java process without data format transformation and also as workaround to write parquet file from VectorSchemaRoot.

try (DuckDBConnection connection = (DuckDBConnection) DriverManager.getConnection("jdbc:duckdb:")){
    connection.registerArrowStream("arrow_stream", arrowArrayStream);
    try (Statement preparedStatement = connection.createStatement()){
        preparedStatement.executeUpdate("COPY (SELECT * FROM arrow_stream) TO '"+filePath+"' (FORMAT 'PARQUET', CODEC 'ZSTD')");
    }
}

@lidavidm do you have idea how to implement such conversion?

Component(s)

Java

[lidavidm]

I think you'd have to write an ArrowReader implementation that wraps a VectorSchemaRoot, like this: https://github.com/apache/arrow-adbc/blob/main/java/driver/flight-sql/src/main/java/org/apache/arrow/adbc/driver/flightsql/RootArrowReader.java

@davisusanibar @lwhite1 this might be the kind of thing to just add to arrow-vector (C++/Python have similar adapters)

[igor-suhorukov]

Thank you @lidavidm. It works for me with copy of RootArrowReader.java in my classpath.
Good idea add it to arrow-vector module!

[davisusanibar]

I think you'd have to write an ArrowReader implementation that wraps a VectorSchemaRoot, like this: https://github.com/apache/arrow-adbc/blob/main/java/driver/flight-sql/src/main/java/org/apache/arrow/adbc/driver/flightsql/RootArrowReader.java

@davisusanibar @lwhite1 this might be the kind of thing to just add to arrow-vector (C++/Python have similar adapters)

Please @lidavidm could you help me to provide a link for adapters on C++/Python?

[lidavidm]

Table.to_batches (Python)/TableBatchReader (C++)

[ssvendsen]

To help with my understanding: In the referred RootArrowReader, is there a reason why a new VectorLoader is created in loadNextBatch() at line 52?
Would it not work to just call loadRecordBatch to let the VectorLoader of the parent ArrowReader handle the loading?

[lidavidm]

Ah, yes, it's redundant

[laurentperez]

Hi

I believe I have the same use case as @igor-suhorukov , I found this GH issue by googling. I have a VectorSchemaRoot and I'd like to feed it to duckdb to copy a parquet out of it.

The updated Reader link is https://github.com/apache/arrow-adbc/blob/main/java/driver/jdbc/src/main/java/org/apache/arrow/adbc/driver/jdbc/RootArrowReader.java

The parameters for registerArrowStream are : String, Object . per https://javadoc.io/doc/org.duckdb/duckdb_jdbc/latest/org/duckdb/DuckDBConnection.html#registerArrowStream(java.lang.String,java.lang.Object)

I'm not sure how to register the RootArrowReader ; would you share an example ?

ty

[lidavidm]

https://github.com/duckdb/duckdb-java/blob/b0000256d783a770eed29bd69268f469193c3ed4/src/main/java/org/duckdb/DuckDBConnection.java#L356-L373

I suppose they don't want to declare an explicit dependency on Arrow. The object to pass needs to be an ArrowArrayStream which you can get by exporting an ArrowReader.

arrow-java/c/src/main/java/org/apache/arrow/c/Data.java

Lines 222 to 232 in be89bec

	/**
	* Export a reader as an ArrowArrayStream using the C Stream Interface.
	*
	* @param allocator Buffer allocator for allocating C data interface fields
	* @param reader Reader to export
	* @param out C struct to export the stream
	*/
	public static void exportArrayStream(
	BufferAllocator allocator, ArrowReader reader, ArrowArrayStream out) {
	new ArrayStreamExporter(allocator).export(out, reader);
	}

[hugopendlebury]

@laurentperez did you ever get with working ? I've just come across this post and I'm keen to get it working. I only want to write a parquet file and this seems like a lot of work in java.

I have seen @lidavidm post about exportArrayStream, but I don't know what to use as the 3rd argument.

Any idea how to create an ArrowArrayStream ?

If I pass the RootArrowReader duckdb throws since it expects an instance of org.apache.arrow.c.ArrowArrayStream.

It is possible I can just use an older version of duckdb ? @igor-suhorukov do you happen to know which version of duck-db jar you used ?

[hugopendlebury]

@laurentperez did you ever get with working ? I've just come across this post and I'm keen to get it working. I only want to write a parquet file and this seems like a lot of work in java.

I have seen @lidavidm post about exportArrayStream, but I don't know what to use as the 3rd argument.

Any idea how to create an ArrowArrayStream ?

If I pass the RootArrowReader duckdb throws since it expects an instance of org.apache.arrow.c.ArrowArrayStream.

It is possible I can just use an older version of duckdb ? @igor-suhorukov do you happen to know which version of duck-db jar you used ?

Answering my own question.

I used ArrowArrayStream.allocateNew(allocator);