recipe(segformer): add nvidia b4 ADE recipe coverage

[kujin66]

Summary

Adds verified CPU/CPU recipe coverage for nvidia/segformer-b4-finetuned-ade-512-512 image segmentation. This is a Lane B L0 recipe-only contribution: two checked-in recipes (fp32 and fp16) under the exact-evidence layout, no modelkit source changes, no catalog/index changes, and no examples/recipes/README.md change.

Highest verified Goal on the reachable host is L3 PASS for both shipped CPU tuples: build, CPU perf, numeric compare vs PyTorch, and task-metric eval all completed. The w8a16 candidate was removed because its artifact validation showed fp32 semantics, so it is not shipped or claimed here.

Model metadata

What the model does

nvidia/segformer-b4-finetuned-ade-512-512 is a SegFormer semantic image segmentation checkpoint fine-tuned for ADE-style scene parsing. A user supplies an RGB image and the model produces per-pixel logits for 150 semantic labels.

Evidence/confidence: pinned Hugging Face checkpoint config (architectures=['SegformerForSemanticSegmentation'], model_type='segformer', num_labels=150) and winml inspect task resolution (verified).

Primary user stories

• A user supplies a scene image to obtain an ADE20K-compatible semantic mask for scene understanding (verified from checkpoint task/config and recipe eval dataset).
• A user supplies validation images and annotations to measure segmentation quality with mean IoU / overall accuracy (verified from winml eval on danjacobellis/scene_parse_150).
• A user supplies the same checkpoint to compare fp32 and fp16 CPU footprint/quality behavior through checked-in recipes (verified from built artifacts and perf/eval results).

Supported tasks

Task	Support surface	Evidence	Confidence
image-segmentation	checkpoint, Transformers, WinML	AutoModelForSemanticSegmentation; WinML loader WinMLModelForImageSegmentation; exporter SegformerIOConfig	verified
semantic-segmentation	WinML/Optimum registry alias	Optimum probe after WinML registration includes semantic-segmentation	mapped

Model architecture

SegFormer B4 uses hierarchical overlap patch embeddings and a multi-stage transformer encoder, followed by an all-MLP segmentation decode head.

SegformerForSemanticSegmentation
|-- SegformerModel encoder
|   |-- Overlap patch embeddings x4 stages (hidden sizes 64, 128, 320, 512)
|   |-- Transformer encoder blocks by stage: 3, 8, 27, 3
|   |   |-- Efficient self-attention (heads 1, 2, 5, 8)
|   |   |-- Mix-FFN / GELU feed-forward blocks
|   |   `-- Residual + LayerNorm
|   `-- Multi-scale hidden states
`-- SegformerDecodeHead
    |-- Per-stage projection to decoder hidden size 768
    |-- Multi-scale resize and fusion
    `-- Classifier logits [1, 150, 128, 128] for 512x512 input

Evidence/confidence: config values from pinned checkpoint; HTP export metadata reports SegformerForSemanticSegmentation, 64,108,374 parameters, 937 modules, 339 traced modules, input pixel_values [1,3,512,512]; ONNX output logits [1,150,128,128] (verified/mapped).

Validation and support evidence

1. Baseline

• Current base after rebase: origin/main=5deebd422e95f28fe8fd912ee50ce874710187b3.
• Current PR head prepared for push: 71fa1c3e.
• WinML version: 0.2.0; runtime providers observed: ['DmlExecutionProvider', 'CPUExecutionProvider'].
• winml inspect -m nvidia/segformer-b4-finetuned-ade-512-512 --format json resolves model_type=segformer, architecture SegformerForSemanticSegmentation, task image-segmentation, loader AutoModelForSemanticSegmentation, exporter SegformerIOConfig, WinML class WinMLModelForImageSegmentation.
• Baseline winml config -m nvidia/segformer-b4-finetuned-ade-512-512 -t image-segmentation succeeds and emits a 512x512 pixel_values input and logits output config. The shipped recipes refine this into explicit CPU/CPU checked-in coverage for fp32 and fp16.
• Optimum probe for segformer: vendor tasks []; after WinML registration ['feature-extraction', 'image-classification', 'image-segmentation', 'semantic-segmentation']; added by WinML same list. Effort remains recipe-only because WinML already registers the exporter/loader.

2. Goal

• Effort: L0 recipe-only.
• Goal ceiling: L3, because CPU build, perf, numeric compare, and dataset eval are reachable on this host.
• Outcome: L0; checked-in recipe files only.
• Success definition: each shipped (EP, device, precision) tuple must have exact L0 build/structure evidence, L1 perf data, L2 numeric compare data, and L3 task metric data.

3. Outcome

• Shipped recipes:
  • examples/recipes/nvidia_segformer-b4-finetuned-ade-512-512/cpu/cpu/image-segmentation_fp32_config.json
  • examples/recipes/nvidia_segformer-b4-finetuned-ade-512-512/cpu/cpu/image-segmentation_fp16_config.json
• Highest reached Goal verdict: L3 PASS for CPU/CPU fp32 and CPU/CPU fp16.
• Coverage: full for the target set of this PR (cpu/cpu/fp32, cpu/cpu/fp16). No accelerator EP/device recipe is shipped.
• Methodology friction observed: exact-evidence precision validation removed the prior w8a16 candidate because winml_build_config.json had quant: null, the final artifact size matched fp32, and initializer data types were fp32 rather than quantized.

4. Per-EP/device/precision results

Tier	EP / device	Precision	Verdict	Mean	p50	Throughput	RAM delta	Task metric
L0	CPUExecutionProvider / cpu	fp32	PASS	-	-	-	-	artifact model.onnx + external data; precision fp32; inputs pixel_values [1,3,512,512]; outputs logits [1,150,128,128]
L1	CPUExecutionProvider / cpu	fp32	PASS	1230.051 ms	1238.8 ms	0.81 samples/s	RSS +1203.29 MB	-
L2	CPUExecutionProvider / cpu	fp32	PASS	-	-	-	-	compare mode, 3 samples: cosine 0.9999999999999606; max abs diff mean 2.7338663736979168e-05
L3	CPUExecutionProvider / cpu	fp32	PASS	-	-	-	-	danjacobellis/scene_parse_150, validation, 1 sample: mean_iou 0.43409010767936707; overall_accuracy 0.7907461524009705
L0	CPUExecutionProvider / cpu	fp16	PASS	-	-	-	-	artifact model.onnx + external data; precision fp16; FLOAT16 initializers; fp32 I/O preserved
L1	CPUExecutionProvider / cpu	fp16	PASS	1517.104 ms	1617.28 ms	0.66 samples/s	RSS +1155.58 MB	-
L2	CPUExecutionProvider / cpu	fp16	PASS	-	-	-	-	compare mode, 3 samples: cosine 0.999999974186835; max abs diff mean 0.026076634724934895
L3	CPUExecutionProvider / cpu	fp16	PASS	-	-	-	-	danjacobellis/scene_parse_150, validation, 1 sample: mean_iou 0.43414419889450073; overall_accuracy 0.7908576726913452

Artifact footprint:

Precision	model.onnx	model.onnx.data	Precision evidence
fp32	801,886 bytes	256,367,616 bytes	initializer types FLOAT + INT64; perf model_info precision fp32
fp16	963,537 bytes	127,970,176 bytes	initializer types FLOAT16 + INT64; perf model_info precision fp16

5. Delta

• Recipe diff relative to auto-config is intentionally per-checkpoint/verified-coverage, not a class-wide code fix:
  • export.input_tensors[0].shape: explicit [1,3,512,512] matching this ADE 512 checkpoint.
  • loader.task: image-segmentation.
  • loader.model_class: AutoModelForSemanticSegmentation.
  • fp16 recipe adds quant.mode='fp16', fp16_keep_io_types=true, and requires the build command to pass --precision fp16.
  • both recipes declare eval dataset danjacobellis/scene_parse_150, split validation, annotation column mapping, and ADE20K label mapping file.
• Reducibility: these are checkpoint/task/eval declarations and exact tuple evidence, so no modelkit source change is required.
• Removed from the previous branch shape: README/catalog/reference edits and the invalid w8a16 recipe claim.
• Production recipe README remains untouched.
• No src/winml/modelkit/ files were changed.

6. Analyze summary - component level and op level

Static rule analysis was run against the fp32 artifact with WINMLCLI_RULES_DIR pointing at the populated ModelKitArtifacts runtime rules. The command exited nonzero after producing useful JSON/console classifications, so this is recorded as ANALYZE-PARTIAL-SUCCESS; it is static compatibility analysis, not runtime execution.

Component-level summary:

Artifact	Architecture coverage	Mapping	Actionable EP findings
fp32	SegformerModel encoder plus SegformerDecodeHead; HTP metadata: 937 modules, 339 traced modules, 64,108,374 params; ONNX graph output logits for 150 classes	mapped from hierarchy tags / HTP metadata / ONNX graph; 1846 ONNX ops in final model	QNN partial support around GELU-style ops: Add, Div, Erf, Mul; rule-less CPU/DML runtime rows are not actionable rule evidence

Op-level summary:

Artifact	Graph	Dominant ops	EP roll-up
fp32	1846 operators / 13 unique op types	Add 373; Reshape 338; Transpose 338; MatMul 332; LayerNormalization 128; Conv 85; Div 82; Mul 82; Softmax 41; Erf 41	NvTensorRTRTX and OpenVINO show full support in static rules; QNN shows partial Add/Div/Erf/Mul; CUDA, MIGraphX, DML, CPU are rule-less/all-unknown in this rule set

7. Reproduce commands

$OUT='temp/segformer-b4-repro'
$env:WINMLCLI_RULES_DIR='<path-to-populated-runtime-rules>'

winml --version
python -c "import onnxruntime as ort; print(ort.get_available_providers())"
winml inspect -m nvidia/segformer-b4-finetuned-ade-512-512 --format json
winml config -m nvidia/segformer-b4-finetuned-ade-512-512 -t image-segmentation -o $OUT\baseline_config

winml build -c examples/recipes/nvidia_segformer-b4-finetuned-ade-512-512/cpu/cpu/image-segmentation_fp32_config.json -m nvidia/segformer-b4-finetuned-ade-512-512 -o $OUT\fp32 --ep cpu --device cpu --no-analyze --no-optimize --no-compile --rebuild
winml perf -m $OUT\fp32\model.onnx --device cpu --ep cpu --iterations 10 --warmup 2 --format json -o $OUT\fp32\perf_cpu.json --overwrite
winml eval -m $OUT\fp32\model.onnx --model-id nvidia/segformer-b4-finetuned-ade-512-512 --task image-segmentation --device cpu --ep cpu --mode compare --samples 3 --format json -o $OUT\fp32\compare_cpu.json --overwrite
winml eval -m $OUT\fp32\model.onnx --model-id nvidia/segformer-b4-finetuned-ade-512-512 --task image-segmentation --dataset danjacobellis/scene_parse_150 --split validation --samples 1 --column annotation_column=annotation --label-mapping scripts\e2e_eval\datasets\ade20k_gt_to_model_label.json --device cpu --ep cpu --format json -o $OUT\fp32\eval_cpu_1.json --overwrite
winml analyze --model $OUT\fp32\model.onnx --ep all --output $OUT\fp32\analyze_all.json --format json

winml build -c examples/recipes/nvidia_segformer-b4-finetuned-ade-512-512/cpu/cpu/image-segmentation_fp16_config.json -m nvidia/segformer-b4-finetuned-ade-512-512 -o $OUT\fp16 --ep cpu --device cpu --precision fp16 --no-analyze --no-optimize --no-compile --rebuild
winml perf -m $OUT\fp16\model.onnx --device cpu --ep cpu --iterations 10 --warmup 2 --format json -o $OUT\fp16\perf_cpu.json --overwrite
winml eval -m $OUT\fp16\model.onnx --model-id nvidia/segformer-b4-finetuned-ade-512-512 --task image-segmentation --device cpu --ep cpu --mode compare --samples 3 --format json -o $OUT\fp16\compare_cpu.json --overwrite
winml eval -m $OUT\fp16\model.onnx --model-id nvidia/segformer-b4-finetuned-ade-512-512 --task image-segmentation --dataset danjacobellis/scene_parse_150 --split validation --samples 1 --column annotation_column=annotation --label-mapping scripts\e2e_eval\datasets\ade20k_gt_to_model_label.json --device cpu --ep cpu --format json -o $OUT\fp16\eval_cpu_1.json --overwrite

uv run ruff check src/ tests/
uv run mypy -p winml.modelkit

Quality gates on current local PR head 71fa1c3e:

Gate	Result
uv run ruff check src/ tests/	PASS: All checks passed!
uv run mypy -p winml.modelkit	PASS: Success: no issues found in 409 source files
Affected non-hardware pytest partition	no Python/source/test files changed; recipe-only diff has no direct unit partition beyond final GitHub workflow checks

[kujin66]

Producer-side reviewer sanity check for the SegFormer B4 recipe PR.

Verdict: APPROVE-PENDING-CI (comment-only; GitHub does not allow approving my own PR).

Evidence checked:

• Draft PR exists and body contains all 9 hand-off items: recipe(segformer): add nvidia b4 ADE recipe coverage #1123
• Diff scope from origin/main...HEAD is limited to recipe/catalog/doc/eval-index files for nvidia/segformer-b4.
• Recipe JSON and touched JSON indexes parse with python -m json.tool.
• Baseline auto-config build on current origin/main 904c201 passed: Build complete in 80.5s.
• Recipe builds passed: fp16/unquantized Build complete in 97.7s; w8a16 Build complete in 218.8s.
• Structural validation passed for temp/segformer-b4-w8a16/model.onnx: input pixel_values [1,3,512,512], output logits [1,150,128,128], 926 uint8 initializers.
• L1 CPU perf passed: mean 1456.313 ms, throughput 0.69 samples/s.

Open items before ready-for-merge:

• Wait for pending CI/checks to finish.
• PR is intentionally draft until maintainer/reviewer approval.

[kujin66]

Follow-up evidence added after baseline-delta review:

• Baseline auto-config CPU perf: temp/baseline_segformer_b4/perf_cpu.json -> mean 707.060 ms, throughput 1.41 samples/s, RSS total delta 1202.64 MB, artifact footprint 245.3 MB including external data.
• W8A16 recipe CPU perf: temp/segformer-b4-w8a16/perf_cpu.json -> mean 1456.313 ms, throughput 0.69 samples/s, RSS total delta 1247.51 MB, artifact footprint 64.7 MB.
• Delta is therefore footprint-focused, not CPU-latency-positive: 245.3 MB -> 64.7 MB (-73.6%); CPU latency is 2.06x slower on this CPU host.

Lane A methodology follow-up was also completed locally (kept out of this Lane B model PR):

• _meta-055: current perf JSON flag is --format json, not --json; agents/tester.md updated.
• _meta-056: when runtime rule parquet files are absent, cite exact winml analyze failure and rules-dir listing instead of requiring fabricated analyze_result.json; agents/learner.md and agents/reviewer.md updated.
• model_knowledge/segformer.json updated with the baseline-vs-w8a16 footprint/latency tradeoff.

The local skill directory is not itself a git repo, so I could not open a separate Lane A skill PR from this machine in this session.

[kujin66]

@kujin66 please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

• (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

• (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

[kujin66]

@microsoft-github-policy-service agree [company="{your company}"]

[kujin66]

@microsoft-github-policy-service agree

[kujin66]

Perf evidence has been reorganized into a compact table in the PR body.

Artifact	Precision	Size	Mean Latency	Throughput	RSS Delta	Interpretation
Baseline auto-config	fp32	245.3 MB	707.060 ms	1.41 samples/s	1202.64 MB	baseline
Recipe variant	w8a16	64.7 MB	1456.313 ms	0.69 samples/s	1247.51 MB	-73.6% size, 2.06x slower CPU latency

So the recipe delta is now stated as footprint/catalog coverage, not a CPU speedup claim.

[kujin66]

@microsoft-github-policy-service agree company="Microsoft"

[kujin66]

Current head 71fa1c3e supersedes the older producer-side comments above.

The PR is now recipe-only and ships only exact-evidence CPU/CPU fp32 and fp16 recipes. The previous w8a16 candidate and README/catalog/reference changes were removed after artifact validation showed the w8a16 build had fp32 semantics, so the current PR body is the source of truth for validation evidence.

[kujin66]

Reviewer verdict: APPROVE (comment-only; GitHub does not allow approving my own PR identity).

Reviewed SHA: 71fa1c3ee3fdd310f04730d4306188001d7e0805.

Evidence:

• Diff scope is recipe-only: examples/recipes/nvidia_segformer-b4-finetuned-ade-512-512/cpu/cpu/image-segmentation_fp32_config.json and image-segmentation_fp16_config.json only.
• PR remains Draft and has the model-scale-by-skill label metadata.
• PR body follows the updated contract: Summary, Model metadata, Validation and support evidence with Baseline, Goal, Outcome, per-tuple results, Delta, Analyze summary, and Reproduce commands.
• Current body supersedes older stale comments: no README/catalog/source changes ship, and the invalid w8a16 candidate is removed.
• Local static gates passed on current PR head: uv run ruff check src/ tests/; uv run mypy -p winml.modelkit.
• GitHub checks on the reviewed SHA are all green: CodeQL/Analyze, lint, test (analyze), test (models), test (optim), test (commands), test (remaining), license/cla.
• Review thread enumeration returned zero review threads.

Coverage annotation: full for the target tuples shipped by this PR (cpu/cpu/fp32, cpu/cpu/fp16). No accelerator tuple is claimed or shipped.