TumorChain: Interleaved Multimodal Chain-of-Thought Reasoning for Traceable Clinical Tumor Analysis

Sijing Li^1,2*, Zhongwei Qiu^2,3,1*, Jiang Liu¹, Wenqiao Zhang^1†, Tianwei Lin^1,2,
Yihan Xie¹, Jianxiang An¹, Boxiang Yun², Chenglin Yang¹, Jun Xiao¹, Guangyu Guo^2,1,
Jiawen Yao², Wei Liu², Yuan Gao², Ke Yan², Weiwei Cao², Zhilin Zheng²
Tony C. W. MOK², Kai Cao⁴, Yu Shi⁵, Jiuyu Zhang⁵, Jian Zhou⁶
Beng Chin Ooi¹, Yingda Xia†², Ling Zhang²

¹Zhejiang University ²DAMO Academy, Alibaba Group ³Hupan Lab ⁴Shanghai Institution of Pancreatic Disease ⁵Shengjing Hospital of China Medical University ⁶Sun Yat-sen University Cancer Center

🌟 Overview

Welcome to TumorChain!

Our goal is to advance clinical tumor analysis through reliable multimodal reasoning at scale. This project presents a cohesive three-part framework—Dataset, Benchmark, and Model—to enable safe, explainable, and reproducible tumor assessment in high-stakes settings.

👏 Core Vision:

• Establish a closed-loop multimodal reasoning pipeline that standardizes the path from findings to impressions to pathology.
• Create high-quality benchmarks and reproducible evaluation protocols to enable cross-institution comparison and robust generalization.
• Deliver an interpretable, calibrated, and traceable multimodal framework that reduces hallucinations and supports real-world clinical decision-making.

📫 Data collection and statistics

We introduce TumorCoT-1.5M — a large-scale dataset comprising 1.5 million Chain-of-Thought (CoT) labeled VQA prompts, paired with 3D CT scans, featuring stepwise reasoning and cross-modal alignments along the findings–impression–pathology trajectory.

🎡 Model Architecture

TumorChain is a multi-modal, iterative interleaved reasoning framework for 3D CT tumor analysis that fuses a 3D vision encoder, organ segmentation model, auxiliary classification model, an MLP projector, and a large language model (LLM) to perform stepwise, evidence-grounded reasoning from findings to impressions to pathology, with traceable evidence and calibrated uncertainty.

🛠️ Getting Started

😊 We will release our task definitions, benchmarks, and evaluation protocols in the near future to advance safe, explainable, and reproducible multimodal reasoning for high-stakes tumor analysis. 🚀