Patho-R1: A Multimodal Reinforcement Learning-Based Pathology Expert Reasoner
Paper • 2505.11404 • Published • 2
Patho-R1: A Multimodal Reinforcement Learning-Based Pathology Expert Reasoner
[Arxiv] | [Github Repo] | [Cite]
Introduction📝
While vision-language models have shown impressive progress in general medical domains, pathology remains a challenging subfield due to its high-resolution image requirements and complex diagnostic reasoning.
To address this gap, we introduce Patho-R1-3B, a multimodal pathology reasoner designed to enhance diagnostic understanding through structured reasoning. Patho-R1-3B is trained using a three-stage pipeline:
- Continued pretraining on 3.5M pathology figure-caption pairs for domain knowledge acquisition
- Supervised fine-tuning on 500k expert-annotated Chain-of-Thought samples to encourage reasoning
- Reinforcement learning with Decoupled CLIP and Dynamic sAmpling Policy Optimization to refine response quality
Experimental results show that Patho-R1-3B achieves strong performance across key pathology tasks, including multiple choice questions and visual question answering, highlighting its potential for real-world pathology AI applications.
Quickstart🏃
Here we show a code snippet to show you how to use the chat model with transformers and qwen_vl_utils:
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
"WenchuanZhang/Patho-R1-3B",
torch_dtype="auto", device_map="auto"
)
processor = AutoProcessor.from_pretrained("WenchuanZhang/Patho-R1-3B")
# example question from Pathmmu-test-dataset
# ground truth: D
# Reasoning style options (choose one):
# - Chain-of-Draft, a concise reasoning prompting strategy (COD):
# You are a pathology expert, your task is to think step by step, but only keep a minimum draft for each thinking step, with 5 words at most. Return the answer at the end of the response after a separator. Use the following format:<think> Your step-by-step reasoning </think><answer> Your final answer </answer>
# - Chain-of-Thought (COT):
messages = [
{ "role": "system",
"content": "You are a pathology expert, your task is to answer question step by step. Use the following format:<think> Your step-by-step reasoning </think><answer> Your final answer </answer>"},
{
"role": "user",
"content": [
{
"type": "image",
"image": "./images/example.jpg",
},
{"type": "text", "text": "What feature in the provided micrograph is indicative of chronic inflammation? /n A. Granuloma formation /n B. Multinucleated giant cells /n C. Neutrophilic infiltration /n D. Plasma cells with eccentrically placed nuclei"},
],
}
]
# Preparation for inference
text = processor.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
text=[text],
images=image_inputs,
videos=video_inputs,
padding=True,
return_tensors="pt",
)
inputs = inputs.to(model.device)
# Inference: Generation of the output
generated_ids = model.generate(**inputs, max_new_tokens=2048)
generated_ids_trimmed = [
out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)
Acknowledgements🎖
We gratefully acknowledge the contributions of the open-source community, particularly the following projects which laid the foundation for various components of this work:
- Qwen for providing powerful vision language models that significantly advanced our multimodal understanding and generation capabilities.
- DocLayout-YOLO for document layout detection.
- PaddleOCR for comprehensive optical character recognition.
- ModelScope Swift for efficient model serving and deployment tools.
- LLaMA-Factory for robust LLM training and fine-tuning pipelines.
- VERL for valuable visual-language pretraining resources.
- DeepSeek for high-quality models and infrastructure supporting text understanding.
We thank the authors and contributors of these repositories for their dedication and impactful work, which made our development of Patho-R1-3B possible.
Citation❤️
If you find our work helpful, a citation would be greatly appreciated:
@article{zhang2025patho,
title={Patho-R1: A Multimodal Reinforcement Learning-Based Pathology Expert Reasoner},
author={Zhang, Wenchuan and Zhang, Penghao and Guo, Jingru and Cheng, Tao and Chen, Jie and Zhang, Shuwan and Zhang, Zhang and Yi, Yuhao and Bu, Hong},
journal={arXiv preprint arXiv:2505.11404},
year={2025}
}
- Downloads last month
- 91