ONNX for TensorRT — Day 4: Separate runtime inputs from external weights

[rui-ren]

Goal

Use the Day 1 MLP to see the difference between runtime inputs, initializers, and external weight data.

No ONNX Runtime. No Mobius internals. No essay.

Keep only this model

dailys/day01/onnx/tiny_mlp_ir_v2.onnx

In this model:

• x is supplied on every inference call.
• weight and bias are fixed initializers.
• External data changes where initializer bytes are stored, not what the graph computes.

Exercise

Create dailys/day04/external_weights.py.

1. Identify runtime inputs and initializers

Load the model with onnx.load() and print:

• graph input names
• initializer names

Expected classification:

runtime input: x
initializers: weight, bias

When classifying runtime inputs generally, exclude any graph input whose name also appears in graph.initializer.

2. Save all weights as external data

Create dailys/day04/onnx/, then save a copy using:

onnx.save_model(
    model,
    output_path,
    save_as_external_data=True,
    all_tensors_to_one_file=True,
    location="tiny_mlp_external.data",
    size_threshold=0,
)

Use size_threshold=0 because this educational model is tiny. It forces weight and bias into the sidecar file.

Make the script repeatable by deleting old output and sidecar files before saving.

Expected files:

dailys/day04/onnx/tiny_mlp_external.onnx
dailys/day04/onnx/tiny_mlp_external.data

3. Inspect without loading weight bytes

Load the saved model with:

metadata_only = onnx.load(output_path, load_external_data=False)

For each initializer, print:

• name
• shape
• data_location
• entries in external_data

Both initializers should report external storage and reference tiny_mlp_external.data.

4. Verify the complete artifact

Load normally so ONNX resolves the sidecar file, then check it:

loaded = onnx.load(output_path)
onnx.checker.check_model(loaded)

Also print both file sizes. The sidecar may be very small here; real model weights are much larger.

5. TensorRT gate

If trtexec exists in the target TensorRT environment, keep the ONNX file and sidecar together and run:

trtexec \
  --onnx=dailys/day04/onnx/tiny_mlp_external.onnx \
  --minShapes=x:1x4 \
  --optShapes=x:4x4 \
  --maxShapes=x:8x4 \
  --skipInference

If trtexec is unavailable, record this step as blocked. Do not install TensorRT solely for this lesson and do not substitute ONNX Runtime.

Done when

• dailys/day04/external_weights.py runs successfully
• x is identified as the runtime input
• weight and bias are identified as initializers
• The ONNX file and .data sidecar are created
• Both initializers reference the sidecar when loaded with load_external_data=False
• Normal loading and onnx.checker.check_model() succeed

One takeaway

The application supplies runtime inputs. Initializers are fixed model parameters. External data moves initializer bytes out of the ONNX protobuf, so the ONNX file and sidecar must stay together when TensorRT parses the model.