Use Azure AI Content Understanding for Enhanced document extraction

[paullizer]

Summary

Enhanced extraction for PDFs and images currently means Azure Document Intelligence prebuilt-layout. It preserves tables and selection marks, but it does not produce any image or figure analysis — which is the structure that matters most for PDFs and Office documents.

Azure AI Content Understanding (prebuilt-documentSearch) returns layout-aware markdown plus AI-generated descriptions of figures, charts, and diagrams, along with structured Chart.js/Mermaid output. Enhanced extraction should use it instead.

User Value

Workspace and chat documents that rely on charts, diagrams, screenshots, and figures become searchable and citable by what the figure actually shows, not just by whatever OCR text happens to sit inside it. Documents with tables and checkbox states keep the quality they have today.

Proposed Behavior

• Standard extraction stays Document Intelligence prebuilt-read and remains required for workspaces and chat file uploads.
• Enhanced extraction uses Content Understanding, gated behind a new Enable Enhanced extraction admin toggle that reveals the Content Understanding connection settings.
• The existing Standard / Enhanced / Auto selector keeps working. Enabling Enhanced defaults the selector to Auto.
• Auto keeps sampling the first pages with Document Intelligence Layout as the cheaper detector, and additionally upgrades to Enhanced when the sample contains figures or images, not just tables and selection marks.

Graceful degradation is required

Content Understanding is only available in Azure commercial regions — it is not offered in Azure Government. Enhanced must therefore never become a hard dependency:

• In Azure Government and custom clouds, Enhanced automatically uses Document Intelligence Layout, and the admin UI should say so plainly rather than presenting configuration that cannot work.
• If Content Understanding is enabled but unconfigured or unreachable, the document falls back to Document Intelligence Layout and the reason is recorded on the document.

Images inside Office files

Content Understanding accepts DOCX/DOC/PPTX as input, but its figure analysis only applies to PDFs and images. Embedded images should be pulled out of the OOXML package and analyzed separately with whichever engine backs the selected extraction mode, then indexed as their own citable chunks. Per-document cost needs bounding via a minimum image size, duplicate detection, and a cap.

Acceptance Criteria

• Admin settings expose an Enable Enhanced extraction toggle plus Content Understanding endpoint, authentication type (key and managed identity), API version, and analyzer IDs.
• A Test Connection button validates the endpoint, credentials, and analyzer before saving, and explains the common setup failures rather than surfacing raw HTTP errors.
• An in-app guide covers Foundry resource creation, supported regions, required model deployment defaults, and the Cognitive Services User role.
• The admin UI is environment-aware: Azure Public shows the Content Understanding configuration; Azure Government and custom clouds show that Enhanced uses Document Intelligence Layout and needs no configuration.
• Enhanced falls back to Document Intelligence Layout on unsupported cloud, missing configuration, or request failure, with the reason recorded on the document and shown in workspace tooltips.
• Auto mode upgrades to Enhanced when sampled pages contain tables, selection marks, or figures/images.
• Images embedded in DOCX and PPTX are analyzed by the active engine and indexed as their own citable chunks, with size filtering, duplicate detection, and a per-document cap.
• Existing deployments already set to Enhanced or Auto keep that behavior across the upgrade rather than silently downgrading to Standard.
• Uploaded Office files are treated as untrusted input: no path traversal from archive entry names, bounded decompression, and hardened XML parsing.
• Extraction badges and tooltips name the engine that actually ran, and the Change Extraction action covers images as well as PDFs.
• Functional tests cover engine resolution and fallback, page reconstruction, figure handling, the upgrade migration, embedded image extraction, and the untrusted-archive defenses.

Notes and Constraints

• Cost: prebuilt-documentSearch invokes a language model per document, so Enhanced is meaningfully more expensive than Document Intelligence Layout. Auto is the best default.
• Setup: Content Understanding requires default completion and embedding model deployments on the Foundry resource; requests fail until those are set.
• Compatibility: existing setting and per-document field names should be preserved so no Cosmos or search index migration is needed.

Related

• Related to [P1] Add multimodal augmentation for complex document page extraction #1137, which adds a parallel multimodal augmentation pass over flagged complex pages. That work is complementary: this issue changes which engine backs Enhanced extraction, while [P1] Add multimodal augmentation for complex document page extraction #1137 layers additional structured analysis on top of whatever extraction ran.

[paullizer]

Implemented and merged into Development via #1278 (merge commit d5390c1f), shipping in v0.250.221.

Leaving this open to match the repo convention — Fixes #1277 only auto-closes on merge to the default branch (main), and issues here stay open until Development reaches it.

What shipped

Enhanced extraction now uses Azure AI Content Understanding (prebuilt-documentSearch), which returns layout-aware markdown plus AI-generated descriptions of figures, charts, and diagrams. Standard extraction is unchanged and still uses Document Intelligence prebuilt-read.

All acceptance criteria are met, including graceful degradation: Enhanced automatically falls back to Document Intelligence Layout in Azure Government and custom clouds, when Content Understanding is unconfigured, or when a request fails, with the reason recorded on the document and surfaced in workspace tooltips.

Validation

• 32 tests across two new functional test files, including real DOCX/PPTX zip round-trips and behavioral tests of page reconstruction against the documented response shape.
• The full 857-file functional suite was run against a baseline at the pre-change commit, diffed per test so the repo's pre-existing failures were excluded. One regression surfaced (test_legacy_doc_ole_extraction.py, an importlib stub set that did not know about the new module) and was fixed in the same PR.
• Two review rounds found six issues — two High, four Medium — all fixed before merge. Every security fix has a regression test confirmed to fail against the pre-fix code rather than assumed correct.

Follow-ups worth tracking separately

1. Cost communication. The upgrade migration switches enable_enhanced_extraction on for deployments already set to Enhanced or Auto, preserving their intent. Because prebuilt-documentSearch invokes a language model per document, those tenants will see higher per-document extraction cost than Document Intelligence Layout. Worth calling out in release communications so it does not arrive as a surprise on a bill.
2. APIM passthrough. Every other Azure service in the app supports an APIM passthrough option; Content Understanding does not yet. Deliberately left out to keep this change's surface smaller, and straightforward to add later following the existing pattern.
3. Chunk sizing. Content Understanding markdown is denser than Document Intelligence Read output because it includes HTML tables and figure descriptions, so chunks are larger. Worth watching against the embedding context cap on figure-heavy corpora.

[paullizer]

Follow-up: EMF/WMF diagrams in Office files are now analyzed.

A real TIC 3.0 architecture document uploaded with Enhanced extraction on produced no image analysis and no log output, despite containing four diagrams. The images were EMF metafiles, and the embedded-image whitelist accepted only raster formats, so extraction matched nothing — and returned before emitting any status message, which is why the workspace log stayed silent.

Why it needed more than a whitelist entry

Neither extraction engine accepts a metafile, so EMF has to be rasterized first, and the usual routes were all closed:

• Pillow's metafile handler is gated on hasattr(Image.core, "drawwmf") and commented "windows only" because it is GDI-backed. Relying on it would have worked in local development and silently failed in the container.
• The runtime is azurelinux/distroless/python, with no shell and no package manager, so LibreOffice or Inkscape would mean copying a very large dependency tree into a distroless image.
• There was no shortcut: the diagrams contain roughly 2,000 genuine vector drawing records each, with no embedded bitmap and no EMF+ wrapper.

functions_emf_render.py renders EMF and WMF in-process using only Pillow, io, and struct — no system packages, no external binaries, no platform branches, and no container change.

Result

All four diagrams rasterize into recognizable Azure architecture renderings, and their text labels are recovered — Event Hub Namespace, Log Analytics Workspace, Azure Firewall Table, SysLog Table, CLAW Registered App — so labels stay searchable even when the vision engine returns no description.

Embedded image processing is also no longer silent. Extraction reports candidates found, images analyzed, and per-reason skip counts, with progress per image, and persists those counts on the document. Previously "no images in this file" and "images found but all skipped" were indistinguishable.

Tracked in #1279, all checks green. Version 0.250.223.

Note for reviewers

Rasterization is a description aid for search and citation rather than a pixel-accurate GDI reimplementation. Gradients, complex clipping, and bitmap blits inside a metafile are not reproduced, and text is drawn with a default font. The separately extracted metafile text keeps labels accurate regardless.

[paullizer]

Follow-up on coverage questions: equations, legacy binary Office formats, and PPTX.

What was already working

PPTX was already covered. Verified end-to-end by placing a real EMF diagram on slide 2 of a test deck: extracted, rasterized, slide attribution correct, 27 labels recovered.

What was not, and is now

Legacy .doc and .ppt were excluded outright. They are OLE compound documents rather than zip packages, so there is no media part to enumerate and the extractor found nothing. Their pictures and embedded equation previews are now carved out of the container by metafile signature, using the length recorded in the metafile's own header, then rasterized and analyzed like any other embedded image. Validation checks record type, signature position, and that the declared length fits the remaining bytes, so a coincidental byte sequence is not mistaken for an image.

Equations were never extracted at all — no formula feature was requested anywhere in the app, so they arrived as whatever OCR guessed. Document Intelligence exposes a FORMULAS add-on, already present in the pinned SDK, which returns equations as LaTeX. It is billed per page, so it is behind a new admin toggle that defaults to off, is guarded by the layout-mode check because the add-on applies to that model only, and the admin UI states plainly that it is a billed add-on.

Boundary worth recording

Equations authored in modern Word are stored as OMML markup rather than images, so they are outside embedded image analysis entirely and the formula add-on does not reach them either. Legacy Equation Editor and MathType objects are stored with a metafile preview, and those are covered by the carving change. Both facts are now written into the feature documentation so the boundary is not misremembered later.

Tracked in #1280, all checks green. Version 0.250.224.

[paullizer]

Shipped in v0.250.228 via #1278 and #1279 and #1280 and #1290.

Release note entries:

• v0.250.228: Existing documents keep their current chunks until they are extracted again.
• v0.250.224: Optional Mathematical Formula Extraction
• v0.250.224: Images in Legacy .doc and .ppt Files Are Now Analyzed
• v0.250.223: Diagrams in Word and PowerPoint Files Are Now Analyzed
• v0.250.223: Embedded Image Processing Is Now Visible in the Workspace Log
• v0.250.221: Enhanced Extraction Now Uses Azure AI Content Understanding
• v0.250.221: Images Inside Word and PowerPoint Files Are Now Analyzed
• v0.250.221: Existing Enhanced and Auto Deployments Keep Their Setting
• v0.250.221: Extraction Badges Name the Engine That Actually Ran
• v0.250.221: Auto Mode Also Upgrades for Figures

Full notes: https://microsoft.github.io/simplechat/explanation/release_notes/

Closing as completed. This issue stayed open after the fix merged because pull requests target Development while the repository default branch is main, so GitHub never applied the Fixes # keyword automatically.