A multimodal generative model for structured and unstructured electronic health records.

Electronic health records (EHRs) are rich clinical data sources but complex repositories of patient data, spanning structured elements (demographics, vitals, lab results, codes), unstructured clinical notes and other modalities of data. Harnessing this heterogeneity for AI-driven clinical insight remains challenging: most current approaches either serialize numeric EHR data into text, risking loss of temporal and quantitative detail, or learn patient embeddings from structured data alone without generative capability. We present Generative Deep Patient (GDP), a multimodal generative model trained on Medical Information Mart for Intensive Care (MIMIC)-IV that jointly models structured EHR time-series and unstructured clinical texts. GDP encodes structured EHR events via a Convolutional Neural Network (CNN)-Transformer encoder and fuses them with clinical text representations through cross-modal attention into a Large Language Model Meta AI (LLaMA)-based generative decoder. GDP is trained using a combination of generative pretraining and auxiliary temporal objectives, followed by multi-task fine-tuning for clinical prediction and narrative generation. Evaluated on the MIMIC-IV dataset, GDP achieved strong predictive performance for heart failure (Area Under the Receiver Operating Characteristic [AUROC] = 0.923), type 2 diabetes (AUROC = 0.817), and 30-day readmission (AUROC = 0.627). In narrative generation, GDP produced clinically coherent discharge summaries with Recall-Oriented Understudy for Gisting Evaluation (ROUGE)-L = 0.135 and Bidirectional Encoder Representations from Transformers Score (BERTScore)-F1 = 0.545. Human evaluation demonstrated high faithfulness, fluency, and clinical utility. These findings demonstrate that unified multimodal generative modeling of structured EHR and clinical text is feasible and yields competitive performance on multiple downstream tasks in MIMIC-IV, informing future EHR-scale multimodal model development.