Explainable AI
Research into AI systems whose outputs can be understood, audited and justified by the people who use or are affected by them.
Overview
Explainability is not a single property but a family of related properties. The field has generated both technical methods for explaining black-box models and a more fundamental challenge: whether those explanations are the right response to opacity in high-stakes AI decisions.
Explainable AI (XAI) is the research programme concerned with AI systems whose outputs — predictions, classifications, recommendations — can be understood, inspected, audited and challenged by humans. Zachary Lipton's 2018 paper "The Mythos of Model Interpretability" provided the clearest taxonomy of what interpretability actually encompasses: simulatability (a human can step through the model's reasoning, as with a decision tree), decomposability (each component has an interpretable meaning), and algorithmic transparency (the training and inference process is open to inspection). These three properties are largely independent, and conflating them has produced significant confusion about what any given explanation actually provides.
DARPA's XAI program (2016–2021) provided substantial formal impetus: a funded research initiative explicitly aimed at AI systems that can explain their decisions, characterise their strengths and weaknesses, and support user understanding and calibration. The program generated a wave of post-hoc explanation methods — techniques for explaining individual predictions from models that are not intrinsically interpretable. Marco Tulio Ribeiro, Sameer Singh and Carlos Guestrin's LIME (2016) provides model-agnostic local explanations: for any prediction, LIME generates a locally interpretable approximation that explains that specific decision, regardless of the underlying model's structure. Scott Lundberg and Su-In Lee's SHAP (2017) provides a theoretically grounded alternative using Shapley values from cooperative game theory: each feature's contribution to a prediction is computed in a way that is consistent across all possible feature orderings and satisfies several desirable axiomatic properties.
Cynthia Rudin's influential 2019 paper in Nature Machine Intelligence challenges the entire post-hoc explanation enterprise for high-stakes settings: inherently interpretable models — decision trees, sparse linear models, rule lists — should be used instead of black-box models with explanations applied afterward, because the accuracy-interpretability tradeoff is largely illusory on structured tabular data, and because post-hoc explanations of black-box models are approximations that may not accurately represent the model's actual reasoning. This is a direct challenge to the assumption that XAI methods are a sufficient response to opacity in consequential AI systems.
The regulatory dimension provides a floor: EU GDPR Recital 71 establishes that individuals subjected to automated decisions with significant effects have a right to "meaningful information about the logic involved." This creates a legal minimum for explainability in regulated contexts that must be met regardless of methodological preferences — and that applies to any AI system making inferences that affect employment, education, credit, or other legally protected interests.
Key Texts
Foundational works in this research tradition.
The definitional paper: simulatability, decomposability, and algorithmic transparency as distinct properties. What interpretability actually means, what practitioners conflate, and what any given explanation method actually provides — and what it does not. The essential starting point before engaging with specific methods.
LIME: model-agnostic local explanations. For any individual prediction, generate a locally interpretable approximation that explains that decision, regardless of model structure. Widely deployed; the canonical example of post-hoc explanation for black-box models.
SHAP: Shapley-value-based feature attribution. Each feature's contribution to a prediction is computed consistently across all feature orderings, satisfying desirable axiomatic properties. Provides both local (per-prediction) and global (model-level) interpretations. Now the most widely used feature attribution method.
The radical challenge: the accuracy-interpretability tradeoff is largely illusory on structured data. Post-hoc explanations are approximations that may not represent what the model actually does. In high-stakes settings, use inherently interpretable models — decision trees, rule lists, scoring systems — rather than black boxes with explanations applied afterward.
The legal floor for explainability: individuals have a right to "meaningful information about the logic involved" in automated decisions with legal or similarly significant effects. Creates a minimum explainability requirement for any AI system in regulated contexts affecting EU data subjects.
Related Research
Connected areas of inquiry.