Skip to content

Give exported diagrams their labels back, and say when an export is working - #1390

Open
Paul Lizer (paullizer) wants to merge 1 commit into
paullizer-react-v2-uifrom
paullizer-mermaid-export-fixes
Open

Give exported diagrams their labels back, and say when an export is working#1390
Paul Lizer (paullizer) wants to merge 1 commit into
paullizer-react-v2-uifrom
paullizer-mermaid-export-fixes

Conversation

@paullizer

Copy link
Copy Markdown
Contributor

Exporting a message containing a Mermaid diagram produced the right shapes, arrows and layout with no text in any of the boxes, while the same diagram read correctly on screen. The export was also slow enough to look like it had hung.

Root cause

Two problems that presented as one.

1. The container has no scalable font. The Dockerfile installed only xorg-x11-fonts-Type1 and xorg-x11-fonts-misc — legacy X11 Type1 and bitmap packages, no scalable TrueType Latin face — behind a || true that hid the case where they did not resolve at all. Azure Linux ships no fonts by default. The server renderer then asked Mermaid for Arial, Helvetica, sans-serif, none of which exist there, and the generic sans-serif fallback resolved to nothing. Chromium measured every label as zero-width, Mermaid fell back to its minimum node size, and no glyphs were painted.

The giveaway in the reported files is that every box came out the same width, while on screen the same diagram had boxes of visibly different widths. That is zero-width text measurement, not a colour, theme or <foreignObject> problem.

2. V2 never sent the diagram it had already drawn. downloadMessageExport() posted only message_id and conversation_id. The classic chat sends visual_assets; V2 did not, so every V2 export took the server path and hit the font problem — even though a correct picture already existed in the page.

Investigated and rejected

I chased htmlLabels/<foreignObject> first and was wrong. The rendered SVG carries 11 <text> elements and zero <foreignObject> elements. I also checked whether the browser's serialize-and-repaint rasterizer loses the styles Mermaid adds via insertRule: a diagram using classDef and style produces identical colour histograms through the canvas path and a live screenshot. The browser rasterizers were correct and are unchanged rather than "fixed" speculatively.

Changes

Area Change
functions_mermaid_server_render.py Embed DejaVu Sans as an @font-face from matplotlib (already a TeX-export dependency); capture each diagram with a Playwright element screenshot instead of SVG → <img> → canvas; block only real network schemes so the font's data: URI resolves; report embedded_font_available
Dockerfile Install DejaVu, Liberation and Noto Sans, and rebuild the font cache, so everything rendering in the image has glyphs
v2_ui/src/lib/exportVisuals.ts (new) Registry of on-screen diagrams and the rasterizer that turns them into visual_assets
MermaidDiagram.tsx, endpoints.ts Register each rendered diagram; carry visual_assets on the export request
toastStore.ts, Toaster.tsx, MessageActions.tsx pending tone that does not auto-dismiss and settles in place; spinner and disabled menu entry while an export runs
toast.js, chat-toast.js, chat-message-export.js, chat-messages.js Same behaviour in the classic UI: { autohide: false } plus a { dismiss } handle, a progress toast always cleared in finally, and the data-pending-label its inline buttons already had

Why an element screenshot

The renderer was serializing its SVG, reloading it through an <img> and painting it onto a canvas — a reconstruction of a browser render, performed inside a real browser that was already open. That isolated context drops <foreignObject> content and will not load a font that is not already present. Screenshotting the mounted element renders exactly what a browser would show, and is less code.

Why this shipped broken

test_server_renders_diagrams_with_visible_labels claimed to verify that "a rendered diagram keeps its label text" but asserted only painted_ratio > 0.01. Boxes and arrows clear that easily, so a diagram with no text at all passed.

It now renders the same graph with and without label text and requires the labelled one to have materially more dark pixels and to be wider. On the fixture: 12,295 dark pixels vs 0, and 945px wide vs 372px.

Validation

  • test_export_mermaid_server_render.py — 13/13
  • test_message_export_progress_feedback.py — 8/8, including a parity check that executes the browser's own normalizeVisualSource in Node against Python's normalize_visual_source. A mismatch there does not fail loudly: every client asset is silently discarded and every diagram quietly re-rendered server-side.
  • test_export_mermaid_browser_rasterizer.py — 3/3
  • test_conversation_export_mermaid_tex_images.py — 18/18
  • test_deep_research_chromium_build_opt_out.py — 3/3
  • test_docs_app_surface_coverage.py — 7/7, test_docs_site_quality.py — 6/6
  • route_tests/ — 3/3
  • npm run build in application/v2_ui — clean

End to end: a real .docx generated through the actual export route embeds a 945×549 diagram with full label text and correctly varying box widths.

Before After
Server-rendered diagram Uniform empty boxes Labels drawn in the embedded font
V2 export with a diagram Always launched headless Chromium Sends the on-screen picture; no browser launch
Exported colours Server defaults The colours the reader chose
During an export Nothing on screen Pending toast plus a disabled, spinning menu entry

Notes for review

  • Pre-existing red test repaired. test_deep_research_chromium_build_opt_out.py asserted an exact deployer version of 1.0.4 and had been failing since the deployer moved past it (now 1.0.26). It covers the Dockerfile changed here, so a red test would have masked real regressions; it now uses the repo's own assert_version_at_least helper, which the versioning instructions require.
  • No deployers/ changes, so deployers/version.txt is not bumped. The Dockerfile lives under application/ and introduces no new build argument or parameter.
  • On the slowness: gunicorn.conf.py already allows 900s per request, so nothing was cutting exports off server-side. The practical ceiling is the Azure App Service front-end idle timeout of roughly 230s, which cannot be configured from this repo. Skipping the browser launch and reporting progress were the available mitigations; PowerPoint remains inherently slow because a model plans the slides.
  • Server-rendered diagrams now always use the embedded DejaVu Sans, so their lettering can differ slightly from a browser-rendered one that uses the reader's own font stack. That is the trade for rendering identically on every deployment.

Version bumped to 0.261.034. Fix doc at docs/explanation/fixes/MERMAID_EXPORT_LABEL_TEXT_FIX.md; feature doc and release notes updated.

…orking

An exported Mermaid diagram came out with every shape and arrow in place and no
text in any of the boxes, while the same diagram read correctly on screen.

The image ships no scalable Latin typeface: the Dockerfile installed only the
legacy X11 Type1 and bitmap packages, behind a `|| true` that hid the case where
they did not resolve at all. The server renderer then asked for `Arial,
Helvetica, sans-serif`, which resolves to nothing there, so Chromium measured
every label as zero-width, Mermaid fell back to its minimum node size, and no
glyphs were painted. The giveaway was every box coming out the same width when
on screen they varied with their labels.

The renderer now embeds DejaVu Sans from matplotlib, already required for TeX
export, so it no longer depends on what the host happens to carry. It also
captures each diagram with an element screenshot rather than serializing the SVG
and repainting it through an `<img>` onto a canvas: it already has a real
browser page open, and the isolated image context silently drops anything it
cannot reproduce. Scalable fonts are added to the image as well, for everything
else that renders in it.

V2 sent only the message and conversation id, so every one of its exports took
the server path. It now sends the diagram already on screen, which skips the
browser launch entirely and keeps the colours the reader chose.

Neither interface said anything while an export ran, and a PowerPoint waits on a
model planning its slides, so it read as hung. Both now raise a notice on click
that is replaced by the result, and disable the menu entry while it runs.

The existing label test asserted only `painted_ratio > 0.01`, which boxes and
arrows clear on their own; it now compares a labelled graph against the same
graph with empty labels on dark-pixel count and width.

Investigated and rejected: htmlLabels and `<foreignObject>` were never involved.
The rendered SVG carries 11 `<text>` elements and no `<foreignObject>`, and a
diagram using classDef produces identical colour histograms through the canvas
path and a live screenshot, so the browser rasterizers were left alone.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant