Including tokenizer to onnx model / basic usage of the onnxruntime-extensions

[MLRadfys]

Hi and thanks for this great library!

Iam very new to onnx and Iam trying to include the Roberta tokenizer into a Roberta onnx model.
As far as I have understood, one can get the onnx graph for the tokenizer using:

import onnxruntime as _ort
from transformers import RobertaTokenizer
from onnxruntime_extensions import OrtPyFunction, gen_processing_models

# Roberta tokenizer
tokenizer = AutoTokenizer.from_pretrained("roberta-base", model_max_length=512)
tokenizer_onnx = OrtPyFunction(gen_processing_models(spm_hf_tokenizer, pre_kwargs={})[0])

Now Iam wondering what the next step is? How can I combine the onnx tokenizer (or graph) with a model?

Thanks in advance for any help,

cheers,

M

[wenbingl]

gen_processing_model returns two ONNX models, one for pre-processing, and other is for post-processing if post_kwargs presents in the kwargs.

If you want to combine any processing model into the ONNX model, please use this function https://onnx.ai/onnx/api/compose.html#onnx.compose.merge_models

[MLRadfys]

Thank you so much for the quick reply!

I will give it a try! Are there any alternatives?
I just looked at the different test functions and saw that one can create a model using

node = [helper.make_node( 'RobertaTokenizer', ['string_input'], ['input_ids'], vocab=_get_file_content(vocab_file), merges=_get_file_content(merges_file), name='bpetok', padding_length=max_length, domain='ai.onnx.contrib')]

graph = helper.make_graph(node, 'test0', [input1], [output1]) tokenizer_model = make_onnx_model(graph)

Would it then be possible to create a pipeline using:

full_model = pnp.SequentialProcessingModule(tokenizer_model, Roberta_model)

Or in other words, which one is the easiest and most straight-forward method? :-)

Really appreciate your help!

Cheers,

M

[wenbingl]

Thank you so much for the quick reply!

I will give it a try! Are there any alternatives? I just looked at the different test functions and saw that one can create a model using

node = [helper.make_node( 'RobertaTokenizer', ['string_input'], ['input_ids'], vocab=_get_file_content(vocab_file), merges=_get_file_content(merges_file), name='bpetok', padding_length=max_length, domain='ai.onnx.contrib')]

graph = helper.make_graph(node, 'test0', [input1], [output1]) tokenizer_model = make_onnx_model(graph)

Would it then be possible to create a pipeline using:

full_model = pnp.SequentialProcessingModule(tokenizer_model, Roberta_model)

Or in other words, which one is the easiest and most straight-forward method? :-)

Really appreciate your help!

Cheers,

M

this approach works in a lower level which requires onnx and tokenization data knowledge and is prone to errors. So, it is recommended to only use gen_processing_model API and users can get support if there is any problem.

[MLRadfys]

Alright, I think I solved it using the gen_processing() and merge functions :-)

I attach my solution as a reference for others who encounter a similar problem:

import torch
from onnxruntime_extensions import gen_processing_models
from onnxruntime_extensions import get_library_path
import onnx
import onnxruntime as ort
import numpy as np
from transformers import RobertaForSequenceClassification, RobertaTokenizer

# Step 1: Load the Huggingface Roberta tokenizer and model
input_text = "A test text!"
model_type = "roberta-base"
model = RobertaForSequenceClassification.from_pretrained(model_type)
tokenizer =RobertaTokenizer.from_pretrained(model_type)

# Step 2: Export the tokenizer to ONNX using gen_processing_models
onnx_tokenizer_path = "tokenizer.onnx"

# Generate the tokenizer ONNX model
tokenizer_onnx_model = gen_processing_models(tokenizer, pre_kwargs={})[0]

# Save the tokenizer ONNX model
with open(onnx_tokenizer_path, "wb") as f:
    f.write(tokenizer_onnx_model.SerializeToString())

# Step 3: Export the Huggingface Roberta model to ONNX
onnx_model_path = "model.onnx"
dummy_input = tokenizer("This is a dummy input", return_tensors="pt")


# 5. Export the model to ONNX
torch.onnx.export(
    model,                                                              # model to be exported
    (dummy_input['input_ids'],dummy_input["attention_mask"]),           # model input (dummy input)
    onnx_model_path,                                                    # where to save the ONNX model
    input_names=["input_ids", "attention_mask_input"],                  # input tensor name
    output_names=["logits"],                                            # output tensor names
    dynamic_axes={"input_ids": {0: "batch_size", 1: "sequence_length"}, # dynamic axes
    "logits": {0: "batch_size"}
    }
)

# Step 4: Merge the tokenizer and model ONNX files into one
onnx_combined_model_path = "combined_model_tokenizer.onnx"

# Load the tokenizer and model ONNX files
tokenizer_onnx_model = onnx.load(onnx_tokenizer_path)
model_onnx_model = onnx.load(onnx_model_path)

# Inspect the ONNX models to find the correct input/output names
print("Tokenizer Model Inputs:", [node.name for node in tokenizer_onnx_model.graph.input])
print("Tokenizer Model Outputs:", [node.name for node in tokenizer_onnx_model.graph.output])
print("Model Inputs:", [node.name for node in model_onnx_model.graph.input])
print("Model Outputs:", [node.name for node in model_onnx_model.graph.output])

# Merge the tokenizer and model ONNX files
combined_model = onnx.compose.merge_models(
    tokenizer_onnx_model,
    model_onnx_model,
    io_map=[('input_ids', 'input_ids'), ('attention_mask', 'attention_mask_input')]
)

# Save the combined model
onnx.save(combined_model, onnx_combined_model_path)

# Step 5: Test the combined ONNX model using an Inference session with ONNX Runtime Extensions
# Initialize ONNX Runtime SessionOptions and load custom ops library
sess_options = ort.SessionOptions()
sess_options.register_custom_ops_library(get_library_path())

# Initialize ONNX Runtime Inference session with Extensions
session = ort.InferenceSession(onnx_combined_model_path, sess_options=sess_options, providers=['CPUExecutionProvider'])

# Prepare dummy input text
input_feed = {"input_text": np.asarray([input_text])}  # Assuming "input_text" is the input expected by the tokenizer

# Run the model
outputs = session.run(None, input_feed)

# Print the outputs
print("logits:", outputs[1][0])

Thanks for the help!

Cheers,

M

[r4ghu]

Hi @MLRadfys ,
Thanks for this example. Its very useful to understand how to convert a HuggingFace Tokenizer to OnnxRuntime. I am able to successfully encode text to inputIds.

I am currently stuck in the process of decoding Ids back to text. Is there some example which I can use for reference.

[wenbingl]

Hi @MLRadfys , Thanks for this example. Its very useful to understand how to convert a HuggingFace Tokenizer to OnnxRuntime. I am able to successfully encode text to inputIds.

I am currently stuck in the process of decoding Ids back to text. Is there some example which I can use for reference.

There is an example here:

onnxruntime-extensions/test/test_autotokenizer.py

Line 51 in bc965e9

m_tok, m_detok = gen_processing_models(

. The second return value of
gen_processing_models is the ONNX model which can decode the ids into text.

[VietAnhNguyen20]

Hi @MLRadfys and @wenbingl ,
Thank you for providing such an excellent and helpful example! I truly appreciate your insights.
I have a quick question regarding my current work. I am converting an XLMRobertaTokenizerFast tokenizer into its ONNX version. However, I noticed that the ONNX tokenizer outputs the following:

• tokens (same as input_ids)
• instance_indices
• token_indices

I did not see attention_mask included in the output. I understand that attention_mask is typically used for filtering padding tokens. To address this, I could add an additional step to generate an array of all ones with the same length as input_ids (since my goal is to generate embeddings for each sentence, and I don't need to do the padding step).

However, I would like to confirm if this approach is correct or if there might be an issue in my conversion process ?
Below is the code I am currently using:

from transformers import XLMRobertaTokenizerFast, XLMRobertaModel, AutoTokenizer, AutoModel
import torch
import onnxruntime as _ort
from onnxruntime_extensions import gen_processing_models
from onnxruntime_extensions import get_library_path as _lib_path
import onnx
import numpy as np

tokenizer =  XLMRobertaTokenizerFast.from_pretrained("sentence-transformers/paraphrase-multilingual-mpnet-base-v2")
model = XLMRobertaModel.from_pretrained("sentence-transformers/paraphrase-multilingual-mpnet-base-v2")


onnx_tokenizer_path = "tokenizer.onnx"

tokenizer_onnx_model = gen_processing_models(tokenizer, pre_kwargs={"WITH_DEFAULT_INPUTS": True}, post_kwargs={"WITH_DEFAULT_INPUTS": True})[0]

with open(onnx_tokenizer_path, "wb") as f:
    f.write(tokenizer_onnx_model.SerializeToString())


onnx_model_path = "model.onnx"
dummy_input = tokenizer("This is a dummy input", return_tensors="pt")

torch.onnx.export(
    model, 
    tuple(dummy_input.values()),
    onnx_model_path,  
    input_names=['input_ids', 'attention_mask'], 
    output_names=['last_hidden_state', 'pooler_output'], 
    dynamic_axes={'input_ids': {0: 'batch_size', 1: 'sequence'}, 
                  'attention_mask': {0: 'batch_size', 1: 'sequence'}, 
                  'last_hidden_state': {0: 'batch_size', 1: 'sequence'},
                  'pooler_output': {0: 'batch_size', 1: 'sequence'}
                  }, 
    do_constant_folding=True, 
    opset_version=18, 
)
tokenizer_onnx_model = onnx.load(onnx_tokenizer_path)
model_onnx_model = onnx.load(onnx_model_path)

print("Tokenizer Model Inputs:", [node.name for node in tokenizer_onnx_model.graph.input])
print("Tokenizer Model Outputs:", [node.name for node in tokenizer_onnx_model.graph.output])
print("Principal Model Inputs:", [node.name for node in model_onnx_model.graph.input])
print("Principal Model Outputs:", [node.name for node in model_onnx_model.graph.output])

The results are:

Tokenizer Model Inputs: ['inputs']
Tokenizer Model Outputs: ['tokens', 'instance_indices', 'token_indices']
Principal Model Inputs: ['input_ids', 'attention_mask']
Principal Model Outputs: ['last_hidden_state', 'pooler_output']

[wenbingl]

Yes, you are right. The attention_mask is a binary mask which is 1 on non-padding id in tokenization scenarios.

[github-actions]

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.

[github-actions]

This issue has been automatically closed due to inactivity. Please reactivate if further support is needed.

[Vladyslav44]

Alright, I think I solved it using the gen_processing() and merge functions :-)

I attach my solution as a reference for others who encounter a similar problem:

import torch
from onnxruntime_extensions import gen_processing_models
from onnxruntime_extensions import get_library_path
import onnx
import onnxruntime as ort
import numpy as np
from transformers import RobertaForSequenceClassification, RobertaTokenizer

Step 1: Load the Huggingface Roberta tokenizer and model

input_text = "A test text!"
model_type = "roberta-base"
model = RobertaForSequenceClassification.from_pretrained(model_type)
tokenizer =RobertaTokenizer.from_pretrained(model_type)

Step 2: Export the tokenizer to ONNX using gen_processing_models

onnx_tokenizer_path = "tokenizer.onnx"

Generate the tokenizer ONNX model

tokenizer_onnx_model = gen_processing_models(tokenizer, pre_kwargs={})[0]

Save the tokenizer ONNX model

with open(onnx_tokenizer_path, "wb") as f:
f.write(tokenizer_onnx_model.SerializeToString())

Step 3: Export the Huggingface Roberta model to ONNX

onnx_model_path = "model.onnx"
dummy_input = tokenizer("This is a dummy input", return_tensors="pt")

5. Export the model to ONNX

torch.onnx.export(
model, # model to be exported
(dummy_input['input_ids'],dummy_input["attention_mask"]), # model input (dummy input)
onnx_model_path, # where to save the ONNX model
input_names=["input_ids", "attention_mask_input"], # input tensor name
output_names=["logits"], # output tensor names
dynamic_axes={"input_ids": {0: "batch_size", 1: "sequence_length"}, # dynamic axes
"logits": {0: "batch_size"}
}
)

Step 4: Merge the tokenizer and model ONNX files into one

onnx_combined_model_path = "combined_model_tokenizer.onnx"

Load the tokenizer and model ONNX files

tokenizer_onnx_model = onnx.load(onnx_tokenizer_path)
model_onnx_model = onnx.load(onnx_model_path)

Inspect the ONNX models to find the correct input/output names

print("Tokenizer Model Inputs:", [node.name for node in tokenizer_onnx_model.graph.input])
print("Tokenizer Model Outputs:", [node.name for node in tokenizer_onnx_model.graph.output])
print("Model Inputs:", [node.name for node in model_onnx_model.graph.input])
print("Model Outputs:", [node.name for node in model_onnx_model.graph.output])

Merge the tokenizer and model ONNX files

combined_model = onnx.compose.merge_models(
tokenizer_onnx_model,
model_onnx_model,
io_map=[('input_ids', 'input_ids'), ('attention_mask', 'attention_mask_input')]
)

Save the combined model

onnx.save(combined_model, onnx_combined_model_path)

Step 5: Test the combined ONNX model using an Inference session with ONNX Runtime Extensions

Initialize ONNX Runtime SessionOptions and load custom ops library

sess_options = ort.SessionOptions()
sess_options.register_custom_ops_library(get_library_path())

Initialize ONNX Runtime Inference session with Extensions

session = ort.InferenceSession(onnx_combined_model_path, sess_options=sess_options, providers=['CPUExecutionProvider'])

Prepare dummy input text

input_feed = {"input_text": np.asarray([input_text])} # Assuming "input_text" is the input expected by the tokenizer

Run the model

outputs = session.run(None, input_feed)

Print the outputs

print("logits:", outputs[1][0])
Thanks for the help!

Cheers,

M

Hi, can you help me, please? #967

[Vladyslav44]

Alright, I think I solved it using the gen_processing() and merge functions :-)

I attach my solution as a reference for others who encounter a similar problem:

import torch
from onnxruntime_extensions import gen_processing_models
from onnxruntime_extensions import get_library_path
import onnx
import onnxruntime as ort
import numpy as np
from transformers import RobertaForSequenceClassification, RobertaTokenizer

Step 1: Load the Huggingface Roberta tokenizer and model

input_text = "A test text!"
model_type = "roberta-base"
model = RobertaForSequenceClassification.from_pretrained(model_type)
tokenizer =RobertaTokenizer.from_pretrained(model_type)

Step 2: Export the tokenizer to ONNX using gen_processing_models

onnx_tokenizer_path = "tokenizer.onnx"

Generate the tokenizer ONNX model

tokenizer_onnx_model = gen_processing_models(tokenizer, pre_kwargs={})[0]

Save the tokenizer ONNX model

with open(onnx_tokenizer_path, "wb") as f:
f.write(tokenizer_onnx_model.SerializeToString())

Step 3: Export the Huggingface Roberta model to ONNX

onnx_model_path = "model.onnx"
dummy_input = tokenizer("This is a dummy input", return_tensors="pt")

5. Export the model to ONNX

torch.onnx.export(
model, # model to be exported
(dummy_input['input_ids'],dummy_input["attention_mask"]), # model input (dummy input)
onnx_model_path, # where to save the ONNX model
input_names=["input_ids", "attention_mask_input"], # input tensor name
output_names=["logits"], # output tensor names
dynamic_axes={"input_ids": {0: "batch_size", 1: "sequence_length"}, # dynamic axes
"logits": {0: "batch_size"}
}
)

Step 4: Merge the tokenizer and model ONNX files into one

onnx_combined_model_path = "combined_model_tokenizer.onnx"

Load the tokenizer and model ONNX files

tokenizer_onnx_model = onnx.load(onnx_tokenizer_path)
model_onnx_model = onnx.load(onnx_model_path)

Inspect the ONNX models to find the correct input/output names

print("Tokenizer Model Inputs:", [node.name for node in tokenizer_onnx_model.graph.input])
print("Tokenizer Model Outputs:", [node.name for node in tokenizer_onnx_model.graph.output])
print("Model Inputs:", [node.name for node in model_onnx_model.graph.input])
print("Model Outputs:", [node.name for node in model_onnx_model.graph.output])

Merge the tokenizer and model ONNX files

combined_model = onnx.compose.merge_models(
tokenizer_onnx_model,
model_onnx_model,
io_map=[('input_ids', 'input_ids'), ('attention_mask', 'attention_mask_input')]
)

Save the combined model

onnx.save(combined_model, onnx_combined_model_path)

Step 5: Test the combined ONNX model using an Inference session with ONNX Runtime Extensions

Initialize ONNX Runtime SessionOptions and load custom ops library

sess_options = ort.SessionOptions()
sess_options.register_custom_ops_library(get_library_path())

Initialize ONNX Runtime Inference session with Extensions

session = ort.InferenceSession(onnx_combined_model_path, sess_options=sess_options, providers=['CPUExecutionProvider'])

Prepare dummy input text

input_feed = {"input_text": np.asarray([input_text])} # Assuming "input_text" is the input expected by the tokenizer

Run the model

outputs = session.run(None, input_feed)

Print the outputs

print("logits:", outputs[1][0])
Thanks for the help!

Cheers,

M

Hi, can you help me, please? #967