Earticle

초록

영어

Biomedical image captioning has become a rapidly advancing research area aimed at supporting clinical workflows by automatically generating descriptive medical reports. However, existing models often suffer from hallucinations, where clinically incorrect findings are described, and semantic misalignment, where captions fail to reflect key visual cues. These issues largely arise from architectures trained on general-domain data, relying on a single encoder, or models lacking robust visual–textual grounding. To overcome these challenges, MedSwinGPT, a reward-guided dual-encoder prefix-fusion model is proposed. It integrates MedCLIP (medical domain encoder) and Swin Transformer (general visual encoder) through a single linear projection to capture complementary global and local visual information. The fused representation conditions BioGPT via prefix tokens, enabling domain-aware and semantically coherent caption generation. To strengthen visual–textual alignment, we jointly optimize Cross- Entropy (CE) and Contrastive Learning (CL) objectives, followed by Self-Critical Sequence Training (SCST) fine-tuning with a multiobjective reward combining BERTScore and contrastive similarity. Evaluated on the ROCO radiology dataset, our reward-guided MedSwinGPT surpasses existing baselines across standard metrics. Qualitative results further demonstrate improved clinical accuracy, semantic grounding, and reduced hallucinations, underscoring its potential for reliable biomedical caption generation.

목차

Abstract
I. INTRODUCTION
II. RELATED WORK
A. Vision–Language Models for Image Captioning
B. Medical Image Captioning
C. Optimization with Reinforcement Learning
D. Motivations for this Work
III. METHODOLOGY
A. MedSwinGPT Model Architecture
B. Self-Critical Sequence Training (SCST)
IV. EXPERIMENTS
A. Dataset
B. Training Details
C. Evaluation Setup
D. Evaluation Metrics
V. RESULTS AND DISCUSSION
A. Quantitative Analysis
B. Qualitative Analysis
VI. CONCLUSION
ACNKOWLEDEMENT
REFERENCES

저자

• Tepy Sokun Chriv [ Department of Electrical Engineering and Computer Science GIST Gwangju, South Korea ]
• Fatima Unse [ Department of Electrical Engineering and Computer Science GIST Gwangju, South Korea ]
• Zubia Naz [ Department of Electrical Engineering and Computer Science GIST Gwangju, South Korea ]
• Linh Van Ma [ Department of Electrical Engineering and Computer Science GIST Gwangju, South Korea ]
• Wookjin Choi [ Department of Radiation Oncology, Thomas Jefferson University Philadelphia, USA ]
• Moongu Jeon [ Department of Electrical Engineering and Computer Science GIST Gwangju, South Korea/School of Electrical Engineering and Technology University of Washington Tacoma, Washington, USA ] Corresponding Author

참고문헌