A unified multimodal framework for cycle-aware keyframe grounding and structured echocardiographic video report generation.
[Paper] · [Installation] · [Data] · [Training] · [Evaluation] · [Citation]
- Clinically aligned tasks: jointly supports cycle- and pathology-conditioned keyframe grounding and video report generation.
- EchoMM-120k: a large-scale instruction dataset designed for temporal localization and professional reporting.
- Cycle-aware reinforcement learning: rewards logical consistency across valid cardiac cycles instead of rigid frame-index matching.
- End-to-end recipe: provides data construction, multi-stage training, inference, and evaluation pipelines.
- 2026.07 — Training and evaluation code released. 🎉
- 2026.05 — EchoMLLM accepted to Findings of ACL 2026.
Echocardiographic diagnosis requires two complementary capabilities: rigorous quantitative keyframe localization for evidence verification and comprehensive qualitative synthesis for diagnostic reporting. Existing multimodal large language models (MLLMs) struggle with these requirements because of workflow misalignment, limited echo-video instruction data, and cyclic temporal ambiguity—semantically identical cardiac phases recur throughout a video, making single-frame supervision inherently ambiguous.
Core question: when several frames across cardiac cycles represent the same phase, which prediction should receive credit?
EchoMLLM addresses this challenge with two fine-grained tasks and a multi-stage training paradigm. Its cycle-aware objective rewards logical consistency across any valid cycle, encouraging invariant reasoning rather than memorization of a single frame index.
Abstract
Echocardiography analysis demands a dual capability: rigorous quantitative keyframe localization for evidence verification and comprehensive qualitative synthesis for diagnostic reporting. However, current Multi-Modal Large Language Models (MLLMs) struggle to meet these clinical requirements due to a misalignment with diagnostic workflows, a scarcity of video instruction data, and the critical challenge of cyclic temporal ambiguity—where the repetitive nature of cardiac cycles renders standard single-frame supervision ill-posed. To bridge this gap, we introduce EchoMLLM, a unified framework designed for real-world echocardiography video understanding. First, we align model capabilities with clinical needs by defining two fine-grained tasks: cycle- and pathology-conditioned keyframe grounding and video report generation. To facilitate this, we curate EchoMM-120k, a large-scale instruction dataset specifically constructed to support temporal localization and professional reporting. Furthermore, to resolve the cyclic ambiguity, we propose a multi-stage training paradigm incorporating a novel cycle-aware Reinforcement Learning (RL) strategy. By prioritizing logical consistency over rigid index matching, our approach moves beyond rote memorization to elicit invariant reasoning. Extensive experiments demonstrate that EchoMLLM achieves state-of-the-art performance against both generalist and medical baselines.
EchoMLLM/
├── data/ # Schemas, samples, and data-mixing utility
├── data_construction/ # Keyframe, report-matching, and CoT pipelines
├── training/
│ ├── stage1_sft/ # LLaMA-Factory SFT and CoT cold-start
│ └── stage2_rl/ # veRL cycle-aware GRPO
├── evaluation/ # Report inference and evaluation utilities
├── assets/ # Paper and figures
└── requirements.txt # Shared runtime dependencies
Each major directory includes a dedicated README with task-specific commands and schemas.
- Linux, NVIDIA GPU(s), and a CUDA toolchain compatible with the installed PyTorch build
- Conda (or another Python environment manager) and Git
- Access to the required source datasets and base-model checkpoints
Both training frameworks are vendored in this repository: LLaMA-Factory v0.9.2 and veRL v0.4.0. Each stage documents its changes from upstream in a PATCHES.md file.
# 1. Clone the repository
git clone https://github.com/Caffeiner07/EchoMLLM.git
cd EchoMLLM
# 2. Create the environment
conda create -n echomllm python=3.10 -y
conda activate echomllm
# 3. Install shared and stage-specific dependencies
pip install -r requirements.txt
pip install -e "training/stage1_sft/LLaMA-Factory[torch,metrics]"
pip install -e training/stage2_rl/verlVerify the environment:
python -c "import torch, transformers; print('torch', torch.__version__, 'transformers', transformers.__version__, 'cuda', torch.cuda.is_available())"We build EchoMM-120k from public sources. Obtain each dataset under its own license, generate the instruction JSON files, update the video paths, and register them in training/stage1_sft/LLaMA-Factory/data/dataset_info.json.
| Task | Source datasets | Documentation |
|---|---|---|
| Keyframe Grounding | EchoNet-LVH, EchoNet-Dynamic, EchoCP, HMC-QU | Data guide |
| Report Generation | MIMIC-IV-ECHO ↔ MIMIC-IV-Note | Data guide |
Tiny schema-faithful samples are included in data/; full patient data and source videos are not redistributed. See the data construction guide to regenerate the corpus and the data guide for schemas and expected filenames.
| Role | Model |
|---|---|
| Base model | Qwen/Qwen2.5-VL-3B-Instruct |
| CoT teacher | Qwen/Qwen2.5-VL-72B-Instruct |
EchoMLLM follows a two-stage recipe, with each stage initialized from the preceding checkpoint:
# Stage 1a: synergistic multi-task SFT
pushd training/stage1_sft/LLaMA-Factory
bash examples/echomllm/run_multitask_sft.sh
# Stage 1b: CoT cold-start SFT (initializes Stage 2)
bash examples/echomllm/run_cot_sft.sh
popd
# Stage 2: cycle-aware GRPO
pushd training/stage2_rl/verl
MODEL=/path/to/stage1_cot_ckpt \
bash examples/grpo_trainer/run_qwen25_vl-video-3b.sh
popdSee the training guide for details and per-framework patches.
Merge the fine-tuned model and run report inference:
# Merge the FSDP checkpoint
pushd training/stage2_rl/verl
python scripts/model_merger.py merge --backend fsdp \
--local_dir /path/to/global_step_XXXX/actor/ \
--target_dir /path/to/echomllm_huggingface/
popd
# Run report generation
python evaluation/infer_us_report.py \
--model_path /path/to/echomllm_huggingface \
--input_json /path/to/test.json \
--output_json /path/to/predictions.json \
--num_gpus 8- Keyframe grounding: cycle-aware minimum-distance metric
- Report generation: LLM-as-a-judge evaluation
Prediction records contain video, prompt, ground_truth, and prediction. See the evaluation guide for supported inference scripts, arguments, and scoring details.
- Release the EchoMM-120k dataset
- Release pretrained model checkpoints
If you find this work useful, please consider starring the repository and citing our paper:
@inproceedings{huang-etal-2026-echomllm,
title = {{E}cho{MLLM}: Incentivizing Echocardiographic Video Understanding with Keyframe Grounding and Report Generation},
author = {Huang, Heyu and Sun, Wanran and Chen, Chi and Chen, Bo and Guo, Zonghao and Li, Yuhua and Li, Ruixuan and He, Kunlun and Sun, Maosong},
booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
year = {2026},
pages = {20053--20071},
publisher = {Association for Computational Linguistics},
url = {https://aclanthology.org/2026.findings-acl.1001/}
}This repository is built upon LLaMA-Factory and veRL, and the model is based on Qwen2.5-VL. Data derives from EchoNet-LVH, EchoNet-Dynamic, EchoCP, HMC-QU, and the MIMIC-IV suite. We thank the authors and maintainers of these valuable resources.
Code and configurations are released under the Apache-2.0 License. Third-party datasets and base/teacher models retain their respective licenses. EchoMLLM is a research prototype and assistive tool—not a diagnostic device—and must not be used for clinical decisions without rigorous validation and human oversight.