Skip to content

Add LayoutLM v1 document-question-answering support (impira/layoutlm-invoices)#1096

Draft
ssss141414 wants to merge 1 commit into
mainfrom
shzhen/add-layoutlm-invoices
Draft

Add LayoutLM v1 document-question-answering support (impira/layoutlm-invoices)#1096
ssss141414 wants to merge 1 commit into
mainfrom
shzhen/add-layoutlm-invoices

Conversation

@ssss141414

@ssss141414 ssss141414 commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Add verified CPU recipes for impira/layoutlm-invoices using the LayoutLM question-answering support that is now present on current main.

This PR is recipe-only. The earlier duplicate layoutlm.py, registration import, and export tests were removed after rebasing onto current main (5deebd422e95f28fe8fd912ee50ce874710187b3).

Model metadata

  • What the model does: LayoutLM v1 extracts answer spans from OCR tokens and their document bounding boxes.
  • Primary user story: supply tokenized invoice text plus bounding boxes and obtain start/end token logits for document question answering.
  • Supported task: question-answering with LayoutLMForQuestionAnswering.
  • Architecture: LayoutLM text/spatial embeddings → 12 encoder layers → extractive QA span head (start_logits, end_logits).

Validation and support evidence

Baseline

Current origin/main at 5deebd422e95f28fe8fd912ee50ce874710187b3 already contains src/winml/modelkit/models/hf/layoutlm.py from #1093.

Recipe-free current-main probe remains misrouted by checkpoint metadata:

winml build -m impira/layoutlm-invoices -o <baseline> --ep cpu --device cpu --no-analyze --no-optimize --no-quant --no-compile --rebuild
Error: Unrecognized configuration class LayoutLMConfig for AutoModelForNextSentencePrediction.

Using current main's existing LayoutLM support with an explicit model-specific recipe succeeds; no additional source code is required.

Goal

Verify CPU fp32 and fp16 recipe coverage, runtime performance, and numeric parity with PyTorch.

Outcome

Recipe-only CPU support. No source or test files are added by this PR.

Per-EP/device/precision results

EP Device Precision Build/structure Perf Numeric parity Eval
CPUExecutionProvider CPU fp32 PASS — 97.8 s; inputs input_ids, bbox, attention_mask, token_type_ids; outputs start_logits, end_logits; external data 508,803,072 bytes PASS — mean 227.18 ms, P50 228.85 ms, 4.40 samples/s PASS — start max-abs 2.24e-5, cosine 1.0; end max-abs 2.10e-5, cosine 1.0 CLI/data blocked — no authoritative invoice DocQA dataset/revision and exact label mapping is configured for this checkpoint
CPUExecutionProvider CPU fp16 PASS — 107.7 s; FLOAT16 artifact external data 254,401,536 bytes; FP32 I/O retained PASS — mean 283.16 ms, P50 280.27 ms, 3.53 samples/s PASS — start max-abs 0.01734, cosine 0.99999996; end max-abs 0.02539, cosine 0.99999998 CLI/data blocked — same dataset-provenance blocker

Host runtime provider: CPUExecutionProvider.

Delta

Compared with current main, this PR adds only two model-specific recipes. They select the existing main implementation through:

  • loader.task = question-answering
  • loader.model_class = LayoutLMForQuestionAnswering
  • safe 512-token inputs, zero-only token_type_ids, and bounded bbox values
  • explicit fp32/fp16 CPU coverage

Analyze summary

Build hierarchy tagging mapped 638/638 ONNX nodes (100%). The artifact has four named LayoutLM inputs and the expected two extractive-QA outputs. Runtime rule data was unavailable on this host, so no unsupported-op count is claimed.

Quality gates

  • ruff check src/ tests/ — PASS
  • mypy -p winml.modelkit — PASS, 409 source files
  • JSON parsing and git diff --check — PASS
  • Review threads enumerated: 0 total / 0 open

Reproduce commands

$MODEL = 'impira/layoutlm-invoices'
$OUT = 'temp/impira-layoutlm-invoices'
winml build -m $MODEL -c examples/recipes/impira_layoutlm-invoices/cpu/cpu/question-answering_fp32_config.json -o "$OUT/fp32" --ep cpu --device cpu --rebuild
winml build -m $MODEL -c examples/recipes/impira_layoutlm-invoices/cpu/cpu/question-answering_fp16_config.json -o "$OUT/fp16" --ep cpu --device cpu --precision fp16 --rebuild

@ssss141414
ssss141414 force-pushed the shzhen/add-layoutlm-invoices branch from 37278b0 to 175b8e7 Compare July 15, 2026 04:13
@ssss141414 ssss141414 added the model-scale-by-skill Model support PR created or maintained by the adding-model-support skill label Jul 16, 2026
@ssss141414
ssss141414 force-pushed the shzhen/add-layoutlm-invoices branch from c3bb569 to 4c0ec9a Compare July 22, 2026 15:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

model-scale-by-skill Model support PR created or maintained by the adding-model-support skill

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant