Not able to run Exclude-Embed models

[balachandarsv]

Describe the bug
I have taken a model which is supported by this library, and have removed the embeddings from the model using exclude-embeds option in builder.py. Now when i try to run the model, it is still asking for input_ids instead of input_embeds.

To Reproduce
Steps to reproduce the behavior:
Qwen/Qwen2.5-0.5 model converted by giving exclude-embeds option and run inference in java.

Desktop (please complete the following information):

• OS: Mac M3

Can someone share idea on how to make this work?

[kunal-vaishnavi]

This is a known issue inside ONNX Runtime GenAI. The exclude_embeds flag has currently been used internally to generate the text component of the Phi-3 vision and Phi-3.5 vision ONNX models. Since those models still accept input_ids as an input to the embedding component and produce inputs_embeds as an output, the inputs_embeds input to the text component has been an ORT GenAI managed input and not a user managed input.

onnxruntime-genai/src/models/multi_modal_vision_model.cpp

Lines 114 to 123 in 41c2543

	DecoderState::DecoderState(const MultiModalVisionModel& model, DeviceSpan<int32_t> sequence_lengths, const GeneratorParams& params, const CapturedGraphInfo* captured_graph_info)
	: State{params, model},
	model_{model},
	captured_graph_info_{captured_graph_info},
	position_inputs_{model, *this, sequence_lengths} {
	inputs_embeds_.Add();
	position_inputs_.Add();
	logits_.Add();
	kv_cache_.Add();
	}

The class that manages these embeddings can be found here. However, user access to the class has not been enabled currently.

onnxruntime-genai/src/models/embeddings.cpp

Lines 19 to 22 in 41c2543

	// Embeddings are only transient inputs and outputs.
	// They are never the user provided/requested model inputs/outputs
	// So only create the transient input and reuse that ortvalue for previous
	// steps in the pipeline.

Similar to how the generator object contains an AppendTokens method for adding input_ids, a new method such as AppendEmbeds will need to be added to add inputs_embeds. Then, that method can be called by other language bindings (e.g. Java, Python, etc) and the requirement that input_ids must exist as a model input can be relaxed.

[balachandarsv]

Thanks for the response. Any possibility of getting this implemented anytime soon?

[kunal-vaishnavi]

We are currently preparing for a stable release so this will have to wait until after the release is completed. If you need this feature sooner or you would like to help add it, please feel free to open a PR to contribute as well.