Explainable AI for Electronic Health Records and Clinical Time-Series Prediction: A Systematic Review

Abstract

Machine learning models trained on electronic health records (EHRs) and clinical time-series data now predict sepsis onset, in-hospital mortality, and readmission risk with substantial accuracy, but their high dimensionality and temporal complexity make them especially opaque to the clinicians who must act on their output. We conducted a systematic review, following PRISMA guidelines, of explainable AI (XAI) methods applied specifically to EHR and clinical time-series prediction. From 312 records identified across five databases, 30 studies met inclusion criteria and were organized into three categories: post-hoc feature-attribution methods (chiefly SHAP and LIME) applied to gradient-boosted or deep models; ante-hoc, intrinsically interpretable architectures, including attention-based recurrent networks and additive models; and studies evaluating trust, fairness, or clinical validity of these explanations. Post-hoc attribution dominates the literature and reliably identifies established clinical risk factors, but several studies show that attention weights and feature-importance scores can track spurious correlates of clinical workflow, such as medication-administration timing, rather than the underlying physiology. Ante-hoc and constraint-based models offer a promising, less-explored alternative, and fairness-auditing studies demonstrate that interpretability tools can expose demographic disparities in model behavior. Across categories, however, evidence for a genuine causal link between explanation and improved clinical decision-making remains sparse. We argue that EHR-focused XAI research should shift emphasis from producing more post-hoc explanations toward validating their clinical utility, expanding ante-hoc model adoption, and standardizing evaluation of explanation quality and fairness.