Integration of Multimodal Large Language Models in Medical Imaging and Omics Data: A Comprehensive Review

Abstract

Purpose: This review focuses on how Multimodal Large Language Models (MLLMs) and multimodal AI models are advancing healthcare by integrating medical imaging and omics data. By integrating imaging techniques such as MRI, CT, and PET with genomics, transcriptomics, and proteomics, these models offer a comprehensive understanding of diseases, particularly in areas like cancer diagnosis and treatment. The study also highlights the challenges of managing complex datasets and ensuring effective feature selection.

Materials and Methods: Analysed studies leveraging advanced AI models, such as Convolutional Neural Networks (CNNs) and Multimodal Neural Networks (MM-Nets), to integrate diverse data sources. These models enhance medical imaging with omics data to improve disease prediction and management. Applications reviewed include cancer subtype classification, survival outcome prediction, and precision medicine, with a particular focus on non-invasive diagnostic tools.

Results: The findings underscore the transformative potential of multimodal healthcare. They significantly improve the identification of biomarkers and enable personalized treatment approaches. For instance, models like VGG19-CNN and PAGE-Net demonstrated higher accuracy in predicting cancer-specific outcomes and integrating genomic and imaging data. Moreover, the applications to single-cell analysis and radiomics showcased their ability to uncover molecular-level insights, advancing precision medicine.

Conclusion: represents a breakthrough in healthcare, combining diverse data types to deliver actionable insights for disease management. While challenges such as handling complex datasets and ensuring model transparency remain, ongoing advancements in AI technologies are paving the way for their wider adoption. These models hold immense promise for improving diagnostics, guiding treatment strategies, and enhancing patient outcomes, marking a significant step toward the era of personalized medicine.