Welcome to Math2LaTeX, an open-source implementation of a photomath-style scan-and-render LaTeX OCR model! We use the power of PyTorch to scan your handwritten math, whether on tablet or paper and convert it to LaTeX code. No more typing out complex equations - write it down, scan it, and let LaTeX-OCR do the rest.
From top to bottom: Ground truth, predicted LaTeX.
On test samples from the CRHOME 2013 handwritten digit competition:
From a tablet handwritten validation image:
- Handwriting Recognition: Uses a deep learning model trained on a large dataset of handwritten mathematical symbols and equations.
- LaTeX Conversion: Converts recognized handwriting into LaTeX code, ready to be used in your documents.
- Open-Source: All the details of our implementation can be found in this repository.
To get started with Math2LaTeX, you'll need to have Python 3.6+ and PyTorch 1.0+ installed. To finetune your own model, follow the instructions below to set up the dataset.
- Go to Kaggle's Handwritten Mathematical Expressions and download the dataset. Move
archive.zipinto theMath2LaTeXdirectory. - Run the following:
conda create -n latexocr python==3.11
conda activate latexocr
pip install -r requirements.txt
bash ./setup.shYou should see all checks passed after running setup.sh.
4. Images can be found in img_data, and image name / label pairs are in img_data/labels.csv.
- BLIP baseline.
- TrOCR experiments.
- Handwritten text data for evaluation.
- Pretrain on additional rendered latex data found at https://zenodo.org/api/records/56198/files-archive.
- RCNN + TrOCR segmentation-OCR pipeline.
- Model distillation and quantization.
- Rearrange code structure to a python package.
To begin, run train_TrOCR.ipynb in scripts. Scroll down to the "Validation on REAL Handwritten Digits" header to run the model on your own validation images.
Call python scripts/train_TrOCR.py with the --gpu flag to indicate which GPU to use. Default is 0.
Contributions to LaTeX-OCR are welcome! Whether it's bug reports, feature requests, or new code, we appreciate all help.
LaTeX-OCR is licensed under the MIT License. See LICENSE for more details.
This project was originally developed as a course project at UCLA in collaboration with Leon Lenk, Artin Kim, and Maxine Wu. The original repository can be found at LaTeX-OCR. Special thanks to Leon for the initial collaboration that laid the foundation for this expanded version.
We thank Professor Bolei Zhou as well as all the members of ACM AI.