Understanding how AI systems make decisions is essential for anyone working with AI in healthcare — whether as a researcher, clinician, regulator, or student.
As artificial intelligence becomes increasingly integrated into medical imaging workflows, two terms appear constantly in the conversation: explainability and interpretability. While they are often used interchangeably, they describe fundamentally different approaches to understanding how AI systems make decisions.
Interpretability means you can directly understand how a model works internally. The model's structure is simple enough that a human can trace the path from input to output without any extra tools.
Each feature has a visible weight; the output is a weighted sum
A series of if/then branching rules you can follow
Explicit point systems like CHA₂DS₂-VASc stroke risk score
Trade-off: Interpretable models are transparent, but they often cannot capture the complex patterns in medical images that deep learning can. They work well for structured clinical data but struggle with raw image analysis.
Explainability means you can provide a human-understandable justification for why a model made a specific prediction — even when the model itself is too complex to understand directly. These techniques are applied after the model has already made its prediction (called "post-hoc" methods).
"Black Box"
Post-hoc Explanation
This is essential for deep learning models like convolutional neural networks (CNNs) and transformers, which are widely used in medical imaging. These models can process thousands of image features simultaneously and often outperform simpler models — but their internal decision-making is opaque.
| Aspect | Interpretability | Explainability |
|---|---|---|
| What is it? | The model itself is transparent — you can understand its internal mechanics directly | Post-hoc tools that justify a complex model's output in human terms |
| When? | Built into the model from the start | Applied after the model makes a prediction |
| Which models? | Simple models: linear regression, decision trees, rule-based systems | Complex models: CNNs, deep neural networks, transformers |
| What it shows | How the decision was made — the exact path from input to output | Why this decision was made — which features or regions influenced it |
| Trade-off | High transparency, but often lower accuracy on complex tasks | Enables use of powerful models, but explanations are approximations |
| Clinical analogy | "Your risk score is 7 because of factors A, B, and C" | "The AI focused on this region of the scan when making its diagnosis" |
Key Point: Interpretability and explainability are complementary, not competing. For medical imaging, deep learning models deliver the best diagnostic performance but operate as black boxes. Explainability methods bridge the gap between that performance and the clinical trust required to use them safely.
The methods below are all explainability techniques applied to complex models after a prediction has been made. None of them make the model itself interpretable — they provide external explanations of the model's behaviour.
Clinicians are trained to understand the reasoning behind a diagnosis. Research shows physicians are more likely to adopt AI tools when they can evaluate the reasoning behind the output.
During COVID-19, XAI analysis revealed some models were basing predictions on text annotations or scanner markers rather than actual lung pathology — preventing flawed models from reaching patients.
Explainability methods can reveal when a model's decisions are influenced by demographic features or scanner type rather than actual pathology, allowing correction before deployment.
The EU AI Act classifies medical AI as 'high-risk' requiring transparency. The FDA has authorized over 950 AI/ML devices and is developing frameworks for ongoing monitoring.
XAI methods can reveal patterns AI finds in medical images, potentially identifying new imaging biomarkers and generating research hypotheses. AI becomes a scientific collaborator.
Patients have a right to understand how decisions about their care are made. Explainability ensures healthcare providers can communicate the AI's role in meaningful terms.
CNN classifiers trained on the BRATS dataset achieve over 97% accuracy for tumour detection. Grad-CAM heatmaps confirm the model focuses on actual tumour locations. SHAP validates reliance on tissue intensity and lesion boundaries rather than artifacts.
XAI analysis revealed that some highly-accurate COVID-19 classifiers were relying on text labels and hospital-specific characteristics rather than lung pathology. This became one of the most cited examples of why XAI is essential.
LRP applied to 3D CNNs processing DaTscan SPECT images shows the model focusing on striatal uptake patterns — the primary biomarker for Parkinson's disease — demonstrating clinically meaningful feature learning.
Research examined the impact of sex bias on AI models for knee ultrasound interpretation. Explainability methods identified when demographic factors inappropriately influenced predictions.