ONNX for TensorRT — Day 3: Turn dynamic shapes into TensorRT profiles

[rui-ren]

Goal

Read tensor dtypes and shapes from the Day 1 MLP, then define the TensorRT profile required by its dynamic batch dimension.

No ONNX Runtime. No large-model source exploration. No essay.

Model

Use:

dailys/day01/onnx/tiny_mlp_ir_v2.onnx

Its expected tensor flow is:

x      [batch, 4]  float32
weight [4, 3]      float32
Wx     [batch, 3]  float32
bias   [3]         float32
y      [batch, 3]  float32

Exercise

1. Create one shape-inspection script

Create dailys/day03/inspect_shapes.py. It must:

1. Load the model with onnx.load().
2. Run onnx.checker.check_model().
3. Run onnx.shape_inference.infer_shapes().
4. Print the name, element type, and shape of:
   • graph inputs
   • initializers
   • intermediate values
   • graph outputs

Use ONNX metadata rather than hard-coding the expected results.

A dimension with dim_value is fixed. A dimension with dim_param, such as batch, is symbolic and dynamic at engine-build time.

2. Read the tensor flow

From the script output, confirm only these facts:

• x is float32 with shape [batch, 4].
• weight is a fixed initializer with shape [4, 3].
• Wx and y have shape [batch, 3].
• Only the batch axis needs a TensorRT optimization-profile range.

No separate notes document is required.

3. Define one TensorRT profile

Use this profile for input x:

min: [1, 4]
opt: [4, 4]
max: [8, 4]

Interpretation:

• minimum supported batch: 1
• preferred/tuned batch: 4
• maximum supported batch: 8
• feature width: always 4

If trtexec is available in the target TensorRT environment, the corresponding gate is:

trtexec \
  --onnx=dailys/day01/onnx/tiny_mlp_ir_v2.onnx \
  --minShapes=x:1x4 \
  --optShapes=x:4x4 \
  --maxShapes=x:8x4 \
  --skipInference

If trtexec is unavailable, record the TensorRT step as blocked. Do not install TensorRT just for this lesson and do not substitute ONNX Runtime.

Done when

• dailys/day03/inspect_shapes.py runs successfully
• ONNX checker and shape inference succeed
• The script prints x, weight, bias, Wx, pre_activation, and y
• Each printed value includes its dtype and shape
• batch is identified as dynamic; 4 and 3 are identified as fixed
• The min/opt/max profile is understood

One takeaway

TensorRT requires a bounded min/opt/max profile for each dynamic graph-input dimension. Fixed dimensions and initializer shapes do not need profile ranges.