[Feature]: Profiling the inference time and memory footprint

[sethzhao506]

What feature would you like to see in FastForward?

Hi Qualcomm Team,

Just wondering if fastforward will support profiling for inference time and memory footprint with full-precision and quantized model say in TensorRT platform (export the model to .onnx and then profile it on TensorRT) to examine the real-world effect.

Why is this feature important to you?

This will test the real-life inference speed improvement and memory reduction.

How would you prioritize this feature?

Must Have - Now

List related issue numbers (if any)

No response

[jpeters-qc]

Hi sethzhao506, thanks for reaching out.

Our long-term goal with FastForward is to support exports to multiple targets and runtimes, which would enable exactly this kind of profiling. At the moment, we're focusing on building out the export path for Qualcomm devices first. While we recognize the importance of supporting platforms like TensorRT, we’re not able to share a timeline for non-Qualcomm targets just yet.