Draw: Action Pen — handwriting recognition (Ink to Text)

Excel's Draw tab (Microsoft 365, Windows) has an Action Pen in the Drawing Tools group: writing in cells with it is converted automatically to typed Excel data. GridCraft's Draw tab has pen, highlighter, eraser, lasso and Ink to Shape, but every stroke stays ink — there is no handwriting recognition.

Update (2026-10-17): the recognizer must be cross-platform

Handwriting recognition must be cross-platform. An implementation that depends on a Windows-only service — Windows Ink / Windows.UI.Input.Inking, or any other Windows-specific handwriting feature — is not acceptable. The recognizer must be one implementation that builds and runs on Windows, macOS, Linux and BSD; wasm may stay disabled with a clear message, but must not break the build. Recognition must stay offline: no Office component, no Microsoft cloud service and no network round-trip.

This supersedes the original Windows-only requirement below. PR #1778 (Windows Ink) has been closed because it is not cross-platform; a new implementation is required.

What that implies

There is no OS-independent handwriting service: Windows.UI.Input.Inking has no macOS/Linux equivalent, macOS Vision is OCR and platform-specific, and Linux has no standard handwriting service. The only realistic cross-platform, offline design is a recognition model executed in-process:

1. Prefer a pretrained model with clear, redistributable licence terms. Creating or training a model is possible but is significant work, and needs its own plan (training data and its licence, conversion, accuracy validation). The options below explain why PdfCraft's pipeline was not adopted.
2. Run it with a cross-platform inference engine. RTen (.rten/ONNX) is the leading pure-Rust candidate and fits GridCraft; the Nemotron proposal would instead require a GGUF/GGML llama.cpp runtime, which does not exist yet (see below).
3. Rasterize the Action Pen strokes (or extract stroke features) into the model's input, recognize, then enter the text through the normal cell.set path.
4. Treat the model as an optional, hash-pinned download (as PdfCraft does for its OCR models): absent or corrupt model → the tool is disabled with a clear message, never a crash. Do not commit model binaries.

Considered: PdfCraft's model pipeline (not adopted)

PdfCraft (storytold/pdfcraft) already runs OCR and historical handwriting recognition offline, cross-platform, in pure Rust. Its pipeline was evaluated as the reference design:

• crates/ocr uses ocrs + rten (MIT/Apache-2.0); inference is Rust with no native sidecar, and the crate's wasm compile is checked.
• Model files are not committed. ATTRIBUTION.toml lists them as [[fetched]] kind = "model" with a URL and SHA-256; cargo xtask models downloads and verifies them into assets/models/ and writes the licence text.
• Models::find() searches $PDFCRAFT_MODELS, then beside the executable (models/, macOS ../Resources/models, Linux/BSD ../share/<app>/models), then the source tree. Release packages copy the models in; a source checkout fetches them.
• Its handwriting add-on (PR rows/cols parameters: accept a single row or column #1059) is a ~13 MB Apache-2.0 Kraken PP-OCRv6 model on Zenodo, converted to ONNX without changing the weights. It renders a line to a 96-pixel-high RGB image, runs it through RTen and CTC-decodes a 1,623-class dictionary. It is an explicit optional install; without it, recognition reports a clear error and leaves the document unchanged.

Why it was not adopted:

1. No suitable model in PdfCraft. Its only handwriting weights are historical (Fraktur/Kurrent and other historical scripts). They are not validated for modern English/Spanish handwriting, and their dictionary carries historical characters. PdfCraft's modern model is printed-text OCR. Adopting the code without a modern model would add the crate, the RTen dependency, line rasterization, add-on download/packaging and attribution plumbing while leaving Action Pen unable to recognize text.
2. It is line-image HTR, not online-stroke recognition. PdfCraft recognizes rendered page lines after segmentation. Action Pen captures raw stroke points in cells, so the pipeline would need a new rasterization step; its single-column, upright-line segmenter is built for page scans and must be adapted (its short-line crop shortcut helps, but multi-stroke cell words still need batching).
3. The handwriting path is unmerged and contributor-hosted. The recognizer is open PR rows/cols parameters: accept a single row or column #1059, and its model/dictionary artifacts live in a contributor's ocr-historical-v1 release (lukashold/pdfcraft). The licence is Apache-2.0, but provenance and maintenance are not settled in the project; adopting it now would make GridCraft depend on an unmerged path and a private release.
4. Its model, not its code, was the blocker. Even with the infrastructure in place, the issue needed a modern multilingual handwriting model. The proposal below uses Nemotron OCR v2 instead of adopting PdfCraft's historical add-on.

Decision: do not adopt PdfCraft's historical implementation. Reuse its optional-add-on mechanics for the proposed model below; the rest of its pipeline (RTen inference, model discovery, hash-pinned download, graceful degradation) remains the reference.

Proposed model: NVIDIA Nemotron OCR v2 (proposal only; no cross-platform runtime yet)

Decision (2026-10-17): propose nvidia/nemotron-ocr-v2 as the multilingual model. PaddleOCR-VL was considered and rejected for Action Pen because its paper's handwriting coverage is limited to Chinese and English; PP-OCRv6's handwriting benchmarks are likewise HW-CN/HW-EN only.

• Languages: 11 — English, Chinese (Simplified/Traditional), Japanese, Korean, Russian, Spanish, French, German, Italian, Dutch, Portuguese.
• Architecture: RegNetX-8GF text detector + pre-norm Transformer recognizer + a relational layout model. v2_english is 53.8M parameters; v2_multilingual is 83.9M parameters (6 layers, d_model 512, 14,244-character set).
• Artifacts: PyTorch checkpoints only (.pth): v2_english ~216 MB, v2_multilingual ~336 MB. No ONNX, GGUF or Rust artifacts are published.
• Runtime today: Python 3.12 + PyTorch + a compiled C++ CUDA extension; Linux amd64 + NVIDIA GPU + CUDA only. There is no Rust runtime and no CPU/Windows/macOS path.
• Proposed runtime path: convert to GGUF/GGML and run it through a llama.cpp variant. This is a proposal, not an existing capability: llama.cpp's current Nemotron support is for Nemotron VL/LLM (tools/mtmd/models/nemotron-v2-vl.cpp), not OCR v2, and the three-module OCR graph (RegNetX detector, Transformer recognizer, relational model) would need new conversion and graph support. MLX and CoreML ports exist, but they are Apple-only.
• Licence: NVIDIA Open Model License (not OSI/Apache-2.0). Commercial use is permitted under its terms; review redistribution before shipping.

Caveats:

• Handwriting is unvalidated. The model card describes document/scene OCR; it makes no handwriting claim. Action Pen is handwriting-only, so this is the central risk. However the blog article about it mentions handwriting https://huggingface.co/blog/nvidia/nemotron-ocr-v2 with synthetic data. testing is necessary to see if it generalizes to human-made handwritten text.
• 11 languages, not 100+; coverage is narrower than PP-OCRv6's 50-language unified model or PaddleOCR-VL's 109.
• The runtime work is a project of its own; until a cross-platform runtime exists, this remains a proposal, not an implementation. If a Rust-native path is required, microsoft/trocr-base-handwritten (MIT, RTen-native) remains the English fallback.

Library / model research (2026-10-17)

No ready-made cross-platform drop-in was found:

• Rust crates: no handwriting-recognition crate. ink-stroke-modeler-rs only smooths strokes; libochw and hanzi_lookup are Chinese character lookup; cink renders signature images; ocrs, rusto-rs and tesseract-rs are printed-text OCR engines, not online-ink recognizers.

• Zinnia (BSD, portable C++ SVM online recognizer): unmaintained (last push 2022), character-level only (word segmentation is left to the caller), no Rust binding, CJK-oriented models, and a C++ toolchain dependency would not fit the pure-Rust/wasm layering.

• Multilingual OCR/HTR candidates: nvidia/nemotron-ocr-v2 (11 languages, .pth only, Linux/NVIDIA/CUDA, NVIDIA Open Model License, no handwriting claim) is the proposed model above. PaddlePaddle/PP-OCRv6_rec (50 languages in one model, Apache-2.0, ONNX 4.46/21.16/76.55 MB, CTC, handwriting benchmarks only HW-CN/HW-EN) is the strongest Rust/RTen-compatible alternative. PaddleOCR-VL-1.6 (109 languages, handwritten support in its card, but the paper's handwriting coverage is Chinese/English) and microsoft/trocr-base-handwritten (MIT, RTen-native, English-only) are fallbacks. Ravi752/trocr-iam-handwriting, espnet/iam_handwriting_ocr and PellelNitram/OnlineHTR are not ready artifacts.

• Sibling repos: WordCraft has no handwriting-to-text implementation to reuse. Issue Keyboard: Shift+Space and Ctrl+Space select without starting an edit (#338) #395 shipped Lasso Select, Add Pen, Ink to Shape and Ink Replay and explicitly left Ink to Math out; crates/engine/src/cmd/ink_shape.rs is a geometric shape recognizer that leaves letters as ink; the open Conditional formatting: home.conditionalFormat without a rule opens nothing (New Rule, Alt, H, L, N) #1165 recognizes only a handful of geometric math symbols. None of it is an HTR model.

So this issue now records the cross-platform requirement, the rejected PdfCraft historical add-on, and the Nemotron OCR v2 proposal. Implementation is blocked on a cross-platform runtime; no Rust/CPU/Windows/macOS path exists today. TrOCR remains the Rust-native English fallback.

Excel behaviour (public documentation)

• Draw ▸ Drawing Tools ▸ Action Pen; start writing in the cells and the handwriting is converted into text.
• Gestures: cross out / scribble on a cell deletes its content; a diagonal line or scribble across a region deletes it; circling cells selects them; writing over an existing value replaces it.
• Highlighter Snap to Cells: with a highlighter and Snap to Cells on, drawing across cells sets their fill to the highlighter colour (drawing again removes it).
• Recognition uses the Office authoring language. Excel prefers a connected service; its offline fallback is the Windows handwriting experience. GridCraft's cross-platform implementation uses an offline model instead (see above).

Reference: https://support.microsoft.com/en-us/excel/handwrite-data-into-excel

What GridCraft needs

1. A new Draw tool draw.actionPen (toggle, same shape as draw.pen), plus a recognizer service the tool calls when a stroke ends.
2. Recognition through one cross-platform, offline, model-based recognizer. Proposed model: Nemotron OCR v2 via a GGUF/GGML llama.cpp variant; no such runtime exists yet, so the proposal is blocked on it. Fallbacks: microsoft/trocr-base-handwritten (RTen-native, English) or PP-OCRv6_rec (ONNX, 50 languages). No Windows-specific API and no per-platform recognizer code.
3. The recognized text enters the cell under the stroke through the normal cell-input path, so numbers and dates become typed values exactly as if they had been keyed.
4. Graceful platform behaviour on every platform and build without a loaded model (including wasm): the tool is absent/disabled with a clear message; never a crash.

Scope of the first PR

Core recognition: pen toggle, stroke capture, cross-platform recognition, typed value into the target cell, one undo step per stroke, wasm build unchanged. The proposal includes the GGUF/GGML conversion/runtime work, stroke rasterization, optional model download/verification, attribution and packaging; the model is never committed. No implementation starts until a cross-platform runtime is proven. The Action Pen gestures (delete by scribble, circle to select, overwrite to replace, highlighter Snap to Cells) are filed separately and build on the same recognizer.

Dedupe

No other issue covers handwriting recognition; repository searches for handwriting, hand writing, ink to text, ink recognition, action pen, recogni return nothing else. PR #1187 (open) is typed ink/InkML/trackpad drawing — it does not recognize handwriting. PR #712 (merged) made ink strokes survive XLSX saves; #551 (closed) fixed ink/icons reopening as rectangles.