fix(perf): support composite (dual-encoder) models in winml perf#866
Conversation
`winml perf` assumed every model exposes a single `io_config`/`_session`, so composite models (CLIP/SigLIP zero-shot-image-classification) crashed with `AttributeError: ... has no attribute io_config` during input generation. Make `PerfBenchmark` composite-aware: - `_aggregate_io_config()` unions the sub-models inputs (their union is exactly the composite forward() kwargs) for input generation/display. - Time the full `forward()` pass via an external PerfStats; single-session models keep recording pure-ORT time inside session.perf(). The monitored loop is refactored to take a run-iteration callable so both paths share it. - Device/EP/task are resolved from a representative sub-model. - `_probe_composite_outputs()` runs one forward() and introspects the result so reported outputs are the composite task-level tensors (e.g. logits_per_image) rather than a deduped union of sub-model ONNX outputs. Add tests/unit/commands/test_perf_composite.py covering aggregation, output describing/probing, input generation, device/EP/task resolution, and the full-forward timing path.
DingmaomaoBJTU
left a comment
There was a problem hiding this comment.
Overall this PR is well-structured and the approach of duck-typing on sub_models (to avoid torch import at load time) is a reasonable trade-off with a clear explanation. A few issues worth addressing before merge.
- Tighten _is_composite duck-type check with isinstance(dict) guard - Promote _sub_models to a property for accessor consistency - Isolate per-sub-model failures: log failing sub-model name before re-raise - Warn when --op-tracing is passed for composite models (silently skipped) - Reference WinMLCompositeModel via a bare local annotation to satisfy CodeQL - Add empty sub_models edge-case tests (run + report)
Replace the duck-typed sub_models check with a concrete isinstance(..., WinMLCompositeModel) check, imported function-locally so composite_model (and torch) is only imported when a model is benchmarked, never at module load — keeping the winml perf --help import budget intact. Test fake now subclasses WinMLCompositeModel so the isinstance check matches.
The _run helper in TestPerfModuleQuantCompileToggles still mocked the old 3-tuple return of resolve_loader_config; it now returns 4 values (loader_config, hf_config, resolved_class, resolution) to match the current signature, fixing the unpack error in the perf --module path.
DingmaomaoBJTU
left a comment
There was a problem hiding this comment.
Follow-up review: 4 of 5 previous findings are fully resolved. One remaining concern on error isolation (see inline comments), and one new minor issue introduced via the merge.
DingmaomaoBJTU
left a comment
There was a problem hiding this comment.
New info-level issue (not inlineable — modified line from merge context): perf.py:719 — _resolution variable is named with an underscore prefix, signalling ''intentionally unused'', yet occupies a named binding. Use plain _ as the fourth discard: parent_loader_cfg, _, _, _ = resolve_loader_config(...). If the TaskResolution value is needed later, rename it at that point without the leading underscore.
Re-raise as RuntimeError("Sub-model '<name>' failed: <exc>") from exc so
the CLI error itself names the failing sub-model, instead of a bare re-raise
that only surfaced the name in logs. Add a test asserting the RuntimeError
names the component and chains the original exception via __cause__.
DingmaomaoBJTU
left a comment
There was a problem hiding this comment.
4/5 previous findings are fully resolved. The two remaining items are non-blocking:
- perf.py:374 (warning, partial): \logger.error\ now names the failing sub-model, which is good. The bare
aise\ still surfaces the original exception without sub-model context at the CLI, but this is a UX improvement rather than a correctness issue — the log output is sufficient for diagnosis. - perf.py:719 (info): _resolution\ could be plain _\ since it's unused, but this is cosmetic.
Core logic is correct: sub-models are benchmarked independently, results are keyed per component, the duck-type check is properly tightened, the --op-tracing\ gap is closed, and the test coverage is solid. Approving.
Problem
winml perfcrashed on composite (dual-encoder) models such as SigLIP/CLIP:PerfBenchmarkassumed every model exposes a singleio_config/_session. Compositemodels have neither — they orchestrate multiple sub-models (e.g. an image encoder and a
text encoder), each with its own ONNX session. The failure is device-independent: the
(model_type, task)registry routes SigLIP to the composite class regardless of--device.Fix
Benchmark each sub-model individually — the same way
--modulebenchmarks eachsubmodule — instead of timing the aggregate
forward()pass:run()returns a singleBenchmarkResultfor single-session models, or a{sub_model_name: BenchmarkResult}mapping for composites._run_sub_models()iteratessub_modelsand benchmarks each through the standardsingle-session pipeline by spawning a child
PerfBenchmarkwith the already-loadedsub-model. Every sub-model reuses the existing compile / input-generation / simple +
monitored loops / result-collection code, so per-sub-model measurement semantics match
single-model runs exactly (pure-ORT time inside
session.perf()).report_composite_results()renders a per-sub-model summary table and writes onecombined JSON nesting each sub-model's full
BenchmarkResult.to_dict()undercomponents(model_id,task,component_count,components).This removes the previous aggregate machinery (
_aggregate_io_config,_describe_outputs,_probe_composite_outputs,_composite_run_iteration) and thecomposite branches inside the benchmark loops.
_is_compositestays duck-typed onsub_models(rather thanisinstance(..., WinMLCompositeModel)) on purpose: anisinstancecheck needs a runtimeimport of
composite_model, which imports torch and would blow thewinml perf --helpimport budget (
tests/cli/test_import_time.py). KeepingWinMLCompositeModelaTYPE_CHECKING-only import also lets the unit tests use lightweight duck-typed fakes.
Result
Tests
tests/unit/commands/test_perf_composite.pyis rewritten for the per-sub-model behavior:each sub-model produces its own
BenchmarkResultwith its own I/O / task / device / EP,every sub-session is compiled and run
warmup + iterationstimes, andreport_composite_resultsemits the combined per-component JSON (and JSON-mode stdout).Existing
test_perf_cli.pyandtest_perf_optracing_cli.pystill pass — no regression.🤖 Generated with Claude Code