[Feature Request] IOBinding in Java API

[znadrich-qf]

Describe the feature request

The docs recommend using IOBinding but it does not seem available in the Java API. Can it be added?

Describe scenario use case

The docs say that using IOBinding helps optimize GPU performance

[Craigacp]

IOBinding allows you to preallocate inputs and outputs on the target device. So if you want to chain models together then you can leave those outputs on the device to pass into the next model, or if your input data is already present on the device you can wrap it directly there. At some point however they need to leave the device and move back to somewhere the JVM can see them which is typically CPU memory (until the JVM can natively access GPU resources). Is there a source of GPU data that you'd like to feed into ORT from Java, or a chain of models you want to accelerate?

It's not clear to me if IOBinding is the ORT team's preferred long term solution for these issues. The C# API now prefers direct creation of OrtValues on the target devices rather than using an IOBinding to wrap them, and we'd probably prefer this in Java too but I've not had time to work on it.

[znadrich-qf]

@Craigacp yes, it would be a chain of models that I'd like to execute without pulling the resulting tensor from each model to the CPU/JVM. My only other best guess on how to do that would be to package one ONNX model file

[Craigacp]

Yeah, you could merge the models into one larger ONNX file. That's likely more efficient as the optimizer gets more of a view of what you're doing, but may have a bunch of other problems depending on what the overall goal is.

As there's now a OrtValue copy method we could port across this class from C#, then expose the copy method so users could create a tensor and copy it to the device memory, and finally rewrite OnnxTensor so that it knows about Java managed memory, ORT CPU memory and other memory pools exposed by an allocator (as it currently only understands the first two of those types). This would be a fair amount of work which I don't have time to do in the near term.

[znadrich-qf]

Thank you for the quick response

[Craigacp]

One other question, at the moment in Java we don't support output pinning for Sequence or Map types, only for Tensors (I'm not actually sure if ORT itself supports pinning for those things as they are a bit hard to construct from outside). Do you need any Sequence or Map outputs in this pipeline?

[znadrich-qf]

I'm a little new to the library so bear with me. I access a specific key from the OrtSession.Result using .get(key), if I call getType on the value then I get ONNX_TYPE_TENSOR which leads me to believe it's not a Sequence or Map type

[Craigacp]

Yes, that's a tensor.