feat(loader): support published ONNX SAM composites

[ssss141414]

Summary

Adds generalized support for published ONNX-only SAM composites, exercised on Xenova/slimsam-77-uniform for CPU mask-generation in fp32 and fp16. The L2 effort ships reusable two-graph discovery/build/eval support plus four component recipes; Goal L3 passed on both precision tuples. Current head 487ff8a7ed976e28b9b2625acdd69a8346cb6c0e also makes graph-pair selection fail closed when multiple equally preferred pairs are valid, adds genuine ambiguity/family-preference tests, and restores whole-package mypy.

Model metadata

What the model does

SlimSAM is a promptable image-segmentation model: a user supplies a 1024-pixel-normalized RGB image plus spatial prompts, and the composite image encoder and prompt/mask decoder return three candidate masks with predicted IoU scores. Confidence: verified from the pinned model card and config.

Primary user stories

• Interactive subject extraction from foreground/background point prompts — verified from the pinned model card.
• Compute image embeddings once and reuse them with new point prompts — mapped from the published graph split and SAM source structure.

Supported tasks

• mask-generation: checkpoint, Transformers, and WinML — verified.
• feature-extraction: Optimum ONNX and WinML — verified.

Model architecture

SamModel (promptable segmentation; config-instantiated parameters: 9,723,528)
├── Image preprocessing (RGB; longest edge 1024; normalize; pad 1024x1024)
├── Vision encoder → vision_encoder.onnx
│   ├── 16x16 patch embedding (3 → 168; 64x64 grid)
│   ├── Vision transformer blocks x12 (12 heads; MLP 168 → 696 → 168)
│   │   ├── Window attention size 14 x8 blocks
│   │   └── Global attention at blocks 2, 5, 8, 11
│   └── Neck projection → image + positional embeddings [B,256,64,64]
├── Prompt encoder → prompt_encoder_mask_decoder.onnx
│   └── Point coordinates/labels → sparse prompt embeddings
├── Mask decoder → prompt_encoder_mask_decoder.onnx
│   ├── Two-way attention blocks x2 (width 256; 8 heads; MLP 2048)
│   ├── Mask hypernetworks → 3 low-resolution candidate masks
│   └── IoU prediction head → 3 candidate scores
└── Runtime post-processing (resize/crop to original image; choose candidate)

• Source/confidence: pinned checkpoint config and Transformers SAM source (verified/mapped); the repository publishes the encoder and decoder as two cooperating ONNX graphs at revision 5850ab45f587c112167512ffef949107115e26a0.

Validation and support evidence

1. Baseline

Pinned baseline: winml-cli e7509b1e908c74beff0a5655b8f8d7de69c5afae, WinML CLI 0.2.0.

• Exact checkpoint revision: 5850ab45f587c112167512ffef949107115e26a0. It is ONNX-only and has no PyTorch weights.
• Published fp32 source hashes: image encoder 9f8433273a6750b587779baa0cf5508111001bf7e7acfcf585d370139fd366d0; prompt decoder f4514391764fbd56e08e119060d874ecd7d52994bfb1968af159e12d4943b5bb.
• Optimum probe: vendor tasks = feature-extraction; after WinML registration = feature-extraction, mask-generation; classification = VENDOR+OVERRIDE.
• winml config --task mask-generation exited 0, and inspect resolved mask-generation / SAMMaskGeneration / SamMaskGenerationIOConfig.
• Recipe-free baseline build exited 2 after 17.489 s, emitted no ONNX artifact, and reported Model class 'SAMMaskGeneration' not found for task 'feature-extraction'. Build had fallen back to feature-extraction despite inspect/config resolving mask-generation.
• Baseline perf was unavailable because build emitted no artifact. Supplementary published-fp32 CPU perf passed: encoder mean/p50 1063.618/1055.979 ms (0.94 samples/s); one-point decoder mean/p50 26.607/26.708 ms (37.58 samples/s).
• Baseline artifact eval was unavailable. Supplementary fp32 composite eval loaded both sessions but the pre-change evaluator required absent input_boxes; published fp16 graphs failed ORT initialization with a float/float16 Cast mismatch.

2. Goal

Axis	Commitment
Effort	L2 — generalized class-of-models support for ONNX-only SAM composites
Goal ceiling	L3 — task evaluation on every frozen CPU/cpu precision tuple
Outcome	L2 — reusable task-family/code support plus recipes
Success definition	fp32 and fp16 encoder+decoder builds; composite perf; numeric comparison against the exact published fp32 pair; pinned-dataset mask-generation eval

No charter downgrade was issued.

3. Outcome

• Current PR head: 487ff8a7.
• Highest Goal verdict: L3 PASS; full frozen coverage for CPU/cpu fp32 and fp16 composite tuples; no deferred tuples.
• Reviewer blockers 1–3 are addressed at current head: optional model-ID narrowing is explicit; multiple equally preferred valid graph pairs are rejected deterministically; tests cover genuine ambiguity, unquantized-family preference, fp32 fallback, native-PyTorch fallback, and malformed published ONNX.
• Producer validation: ambiguity/fallback focus 5 passed; whole-package mypy Success: no issues found in 403 source files; broad relevant suite 1577 passed; four recipe builds 4/4 PASS with Build complete, model.onnx, and winml_build_config.json. All current GitHub checks pass, including lint, CodeQL, and all CI test partitions.
• Tester recheck c836baf13860a5ac02124284a6a3c6a7e2f9f574229219ac5aa7bd0312ac35b6 at current head: terminal PASS; refreshed four builds plus bounded fp32 L1–L3 spot checks preserved artifact selection and execution contracts. Rebuilt bytes differ only by an unused com.microsoft.dml opset import; graph I/O, node contracts/attributes, initializer values, IR, and used domains are execution-equivalent.
• L0: 4/4 component builds passed. L1: composite CPU benchmark passed for both precisions. L2: exact-published-fp32-reference comparison passed; exact-revision PyTorch baseline is BLOCKED_BY_CHECKPOINT_CONTENT. L3: five-sample point-prompt evaluation passed for both precisions.
• Durable Lane A provenance: PR 159, foundational commit 7da173de, and clarification commit fc7c1165. These record sam-001…sam-006, _meta-082, direct-ONNX evidence, and durable feature-gap FILE: references.

Methodology friction observed: _meta-082 added.

Direct-ONNX HTP evidence

All four resolved build configs retain export=null. Direct published-ONNX ingestion bypasses HTPExporter, so each artifact records htp_metadata=NOT_PRODUCED_DIRECT_ONNX and metadata_exists=false; HTP parameter/module/trace counts are not fabricated. Substitute evidence is the pinned configuration/source architecture, exact graph I/O, topology, and node-scope component mapping.

Precision	Artifact	HTP status	Topology / component evidence
fp32	image encoder	NOT_PRODUCED_DIRECT_ONNX	1782 nodes; vision_encoder; pixel_values → image/positional embeddings
fp32	prompt decoder	NOT_PRODUCED_DIRECT_ONNX	810 nodes; prompt_encoder, mask_decoder; points/labels + embeddings → IoU/masks
fp16	image encoder	NOT_PRODUCED_DIRECT_ONNX	1785 nodes; same graph boundary after WinML fp16 conversion
fp16	prompt decoder	NOT_PRODUCED_DIRECT_ONNX	815 nodes; same graph boundary after WinML fp16 conversion

Durable feature-gap references:

• FILE: copilot-skills/dev_skill/adding-model-support/agents/learner.md — direct-ONNX HTP evidence exception.
• FILE: copilot-skills/dev_skill/adding-model-support/agents/reviewer.md — direct-ONNX HTP review branch.
• FILE: src/winml/modelkit/commands/perf.py — expose graph-optimization/session options for converted SAM fp16 graphs requiring ORT_DISABLE_ALL.
• FILE: copilot-skills/dev_skill/adding-model-support/agents/tester.md — preserve the pinned published fp32 pair as the L2 reference when the exact revision has no PyTorch weights; collect capable-host runtime evidence before promoting static EP rows.

4. Per-EP/device/precision results — including perf and eval data

Goal ladder

Tier	CPU/cpu fp32	CPU/cpu fp16	Evidence
L0	PASS	PASS	Encoder + decoder fresh builds; 4/4 total
L1	PASS	PASS_WITH_RUNTIME_CAVEAT	Composite benchmark, ORT_DISABLE_ALL
L2	PASS	PASS	Exact published fp32 ONNX pair as reference
L3	PASS	PASS	Pinned five-row point-prompt subset

Composite perf (CPU, 3 warmups, 20 measured iterations)

Precision	Mean	p50	p90	Throughput	Encoder mean	Decoder mean
fp32	1219.4138750011916 ms	1213.7237499991897 ms	1243.2789499987848 ms	0.8200661157796181 samples/s	1187.7435800022795 ms	31.67029499891214 ms
fp16	1584.7703800012823 ms	1572.3618500051089 ms	1608.407509996323 ms	0.6310062407900322 samples/s	1545.1620049978374 ms	39.60837500344496 ms

L2 numeric comparison against the exact published fp32 pair

Precision	Mask IoU	Mask Dice	Embeddings cosine	Predicted-masks cosine
fp32	1.0	1.0	0.9999999999999487	0.9999999995887173
fp16	0.9930252396005821	0.9965004154183137	0.9999998499461553	0.9998631504023249

L3 task evaluation

Dataset: mattmdjaga/human_parsing_dataset, revision db120bb5c18c146a8fbd2160f7575a288269fe7d, split train, first five deterministic valid rows sample_0000..sample_0004, no shuffle, no streaming, seed 42, prompt_mode=point.

Precision	mIoU	Dice	Samples / skipped	Delta vs fp32
fp32	0.40013556173273895	0.5235086894209545	5 / 0	—
fp16	0.39918580013126925	0.5229150662134483	5 / 0	mIoU -0.0009497616014697; Dice -0.0005936232075062

fp16 ORT caveat: default ORT session creation fails for both components; ORT_DISABLE_ALL passes for both. Standard winml perf exits 2 without JSON. Exact error: Attempting to get index by a name which does not exist: InsertedPrecisionFreeCast_/vision_encoder/layers.3/layer_norm1/Constant_output_0 for node /vision_encoder/layers.0/layer_norm1/Mul/SimplifiedLayerNormFusion/. Under the passing policy, fp16 is slower than fp32 on this CPU.

5. Delta

Four new CPU/cpu component recipes:

• fp32 image encoder and prompt decoder: direct published fp32 ONNX ingestion; no export, quantization, optimization, or compile.
• fp16 image encoder and prompt decoder: the same component roles plus quant.mode=fp16 and fp16_keep_io_types=true; WinML converts the exact fp32 sources because the published fp16 variants are not runnable.

Generalized source delta:

• Discover and validate runnable published encoder/decoder graphs from I/O contracts; distinguish full-precision and quantized families; preserve the uniquely valid full-precision pair; reject malformed graphs and multiple equally preferred valid pairs.
• Generate portable component-role configs, resolve roles at build time, and route both graphs through direct ONNX ingestion while preserving native-PyTorch SAM fallback when no published pair exists.
• Resolve composite mask-generation model IDs for evaluation and infer point-only decoder capability without inventing unsupported box tensors.
• Reject pattern matches with incomplete required-input metadata instead of aborting analysis in dtype-dependent validation.
• Narrow the optional model ID with a runtime invariant before source resolution; no cast, ignore, or suppression.

Current scope: 21 files, 899 insertions, 55 deletions — 4 recipes, 11 production Python files, and 6 unit-test files. The production recipe README is untouched.

6. Analyze summary — component level and op level

ANALYZE-PARTIAL-SUCCESS: all four processes exited 1 because unavailable host EP plugins could not register, but each emitted valid fail-closed JSON for all 11 requested EP/device targets. This is static rules analysis, not measured runtime support; every static row records runtime_support=false, and only CPU composite execution was measured.

Component-level summary

Artifact	Architecture coverage	Mapping	Actionable static findings
fp32 image encoder	patch embed; vision blocks x12; neck	1775 mapped / 7 unmapped	QNN partial: Mod, Reshape, Transpose; OpenVINO partial/unsupported arithmetic/casts
fp32 prompt decoder	prompt encoder; two-way decoder; mask/IoU heads	809 mapped / 1 unmapped	OpenVINO unsupported: Mul, Div
fp16 image encoder	patch embed; vision blocks x12; neck	1775 mapped / 10 unmapped	same grouped rule outcomes as fp32 encoder
fp16 prompt decoder	prompt encoder; two-way decoder; mask/IoU heads	809 mapped / 6 unmapped	same grouped rule outcomes as fp32 decoder

Op-level summary

Artifact	Graph	Dominant ops	EP roll-up
fp32 image encoder	1782 ops / 26 types	Unsqueeze 245; Add 201; Mul 166	TensorRT 12 supported + unknown remainder; QNN partial; OpenVINO unsupported set; five rule-less targets all unknown
fp32 prompt decoder	810 ops / 32 types	Gather 137; Unsqueeze 124; Add 93	TensorRT 8 supported + unknown remainder; QNN known set has no partial/unsupported; OpenVINO Mul/Div unsupported; five rule-less targets all unknown
fp16 image encoder	1785 ops / 26 types	Unsqueeze 245; Add 201; Mul 166	same grouped classifications as fp32 image encoder
fp16 prompt decoder	815 ops / 32 types	Gather 137; Unsqueeze 124; Add 93	same grouped classifications as fp32 prompt decoder

Rules: v0.2.0, archive SHA-256 6dcabbe7f5493fbc2bd23de195ab6e35b3578d510f18c2e220bc5538a1f232cc, 1746 parquet files.

7. Reproduce commands

$MODEL='Xenova/slimsam-77-uniform'
$MODEL_REVISION='5850ab45f587c112167512ffef949107115e26a0'
$DATASET_REVISION='db120bb5c18c146a8fbd2160f7575a288269fe7d'
$OUT='slimsam-test-output'

winml config -m facebook/sam-vit-base --task mask-generation --ep cpu --device cpu --precision fp32 --no-compile -o "$OUT/facebook-sam.json" --overwrite --no-color
winml config -m $MODEL --task mask-generation --ep cpu --device cpu --precision fp32 --no-compile -o "$OUT/slimsam.json" --overwrite --no-color

foreach ($p in @('fp32','fp16')) {
  foreach ($role in @('image-encoder','prompt-decoder')) {
    winml build -c "examples/recipes/Xenova_slimsam-77-uniform/cpu/cpu/mask-generation_${p}_config_${role}.json" -m $MODEL -o "$OUT/build/${p}_${role}" --precision $p --no-compile --no-optimize --no-analyze --rebuild --no-color
  }
  winml eval -m "image-encoder=$OUT/build/${p}_image-encoder/model.onnx" -m "prompt-decoder=$OUT/build/${p}_prompt-decoder/model.onnx" --model-id $MODEL --task mask-generation --ep cpu --device cpu --precision $p --dataset mattmdjaga/human_parsing_dataset --dataset-revision $DATASET_REVISION --split train --samples 5 --no-shuffle --no-streaming --column prompt_mode=point --output "$OUT/eval_${p}.json" --overwrite --no-color
}

$RULES_ZIP="$OUT/rules-v0.2.0.zip"
Invoke-WebRequest 'https://github.com/microsoft/winml-cli/releases/download/v0.2.0/rules-v0.2.0.zip' -OutFile $RULES_ZIP
if ((Get-FileHash $RULES_ZIP -Algorithm SHA256).Hash.ToLowerInvariant() -ne '6dcabbe7f5493fbc2bd23de195ab6e35b3578d510f18c2e220bc5538a1f232cc') { throw 'rules hash mismatch' }
Expand-Archive $RULES_ZIP "$OUT/rules-v0.2.0" -Force
$env:WINMLCLI_RULES_DIR="$OUT/rules-v0.2.0"
foreach ($p in @('fp32','fp16')) {
  foreach ($role in @('image-encoder','prompt-decoder')) {
    winml analyze --model "$OUT/build/${p}_${role}/model.onnx" --ep all --device all --output "$OUT/analyze_${p}_${role}.json" --overwrite --format json --no-color
  }
}
Remove-Item Env:WINMLCLI_RULES_DIR

[ssss141414]

Explainer re-entry complete for model 5 only at current head 487ff8a7.

Reviewer blockers are reflected in the durable PR body:

• producer: fail-closed ambiguity selection, genuine ambiguity/family-preference regression coverage, and whole-package mypy resolution;
• tester: terminal recheck recorded at handoff SHA-256 c836baf13860a5ac02124284a6a3c6a7e2f9f574229219ac5aa7bd0312ac35b6;
• learner/Lane A: durable commits 7da173de + fc7c1165, structured NOT_PRODUCED_DIRECT_ONNX evidence for all four artifacts, and feature-gap FILE: references;
• explainer: exact declaration Methodology friction observed: _meta-082 added.

The PR remains draft with model-scale-by-skill; all current GitHub checks pass. No model 6–8 work was included.

[ssss141414]

APPROVE — independent re-review (model 5 only)

Reviewed exact PR head 487ff8a7ed976e28b9b2625acdd69a8346cb6c0e against current origin/main e7509b1e908c74beff0a5655b8f8d7de69c5afae. No model 6–8 work was inspected, added, or used.

State, conversation, and checks

• PR state: OPEN / DRAFT; model-scale-by-skill present. This approval does not promote the PR.
• Scope: 21 files, exactly the four SlimSAM recipes plus generalized source/tests; production recipe README unchanged; git diff --check passed; branch is 0 commits behind current main.
• Conversation enumeration: 0 inline comments, 1 conversation comment, 0 submitted reviews, 0 review threads, 0 open threads.
• CI: 9/9 checks completed successfully — lint, CodeQL/Analyze, all five test partitions, CodeQL policy check, and CLA.

Seven prior blockers

1. Mypy — RESOLVED. Canonical python -m mypy -p winml.modelkit reports Success: no issues found in 403 source files; the previous nullable model call is now guarded before resolve_composite_onnx_sources(). CI lint is green at this head.
2. Ambiguity fail-closed — RESOLVED. Equally preferred graph pairs now raise a deterministic ValueError; unquantized and precision source families are ranked explicitly instead of selecting a lexical minimum.
3. Genuine tests — RESOLVED. Tests construct two independently valid pairs and assert rejection, and separately prove unquantized-family preference. Independent focused run: 69 passed; focused Ruff passed.
4. Lane A provenance — RESOLVED. gim-home/ModelKitArtifacts#159 is open at fc7c1165; both foundational 7da173de and clarification fc7c1165 are present, and _meta-082 cites durable winml-cli head 487ff8a7.
5. Structured direct-ONNX HTP evidence — RESOLVED. Four durable artifact rows record resolved_export=null, htp_metadata=NOT_PRODUCED_DIRECT_ONNX, and metadata_exists=false, with pinned revision, exact graph I/O, node counts/scopes, topology, and component mapping rather than fabricated HTP counts.
6. Durable feature gaps — RESOLVED. Five FILE: references identify the exact skill/runtime surfaces for the direct-ONNX exception, fp16 session option, L2 reference, and capable-host follow-up.
7. Methodology declaration — RESOLVED. The durable PR body contains exactly once: Methodology friction observed: _meta-082 added.

Prior full evidence revalidated

The prior independent evidence remains applicable because the four recipes and runtime implementation are unchanged by the remediation commit. It includes fresh 4/4 builds with Build complete, structural ONNX checks, genuine fp16 conversion, CPU composite L1 for fp32/fp16, exact-published-pair L2, pinned five-sample L3 with no skipped rows, native PyTorch SAM fallback, component/op analysis, and the documented fp16 ORT_DISABLE_ALL caveat. Frozen coverage is full for CPU/cpu fp32 and fp16; there are no deferred tuples.

Engineering review of the remediation found no new concern: graph-family matching remains contract-driven and model-ID-independent, ambiguity now fails closed, malformed graph sets still fail, and native-PyTorch fallback remains limited to the no-published-ONNX case.

Verdict: APPROVE. Preserve Draft.