What is Explainable AI?

Why it is often mandatory

Explainability is frequently a requirement rather than a nicety. Regulations in credit, insurance, and employment can oblige you to give an affected person a reason for an adverse decision. Clinicians will not act on a recommendation they cannot interrogate. Debugging is impossible if you cannot see why a model produced an output. And bias auditing requires knowing which features drove decisions.

There are two broad routes. Use a model that is interpretable by construction — linear and logistic regression with sensible features, shallow decision trees, generalised additive models, rule lists. Or use a complex model and generate explanations after the fact.

The main post-hoc methods

Feature importance ranks which features the model relies on overall. Useful for sanity-checking and for spotting leakage — if an ID field is your top feature, something is wrong. It says nothing about individual decisions.

SHAP assigns each feature a contribution to a specific prediction, with a solid theoretical grounding in cooperative game theory and the useful property that contributions sum to the prediction. It is the practical default for per-decision explanation. Be aware that its correctness depends on assumptions about feature independence that correlated features violate, and that it can be expensive to compute exactly.

LIME fits a simple local model around one prediction. Intuitive and fast, and less stable — small changes in sampling can produce noticeably different explanations, which undermines trust.

Counterfactuals answer 'what would have to change for a different outcome'. For an applicant told they were declined, 'income £4,000 higher, or two fewer recent applications' is far more actionable than a list of weighted features. For customer-facing explanation this is usually the right format.

Attention and saliency maps highlight influential regions of an image or text. Visually compelling, and weaker evidence than they appear — attention weights are not a faithful account of causal influence.

The honest limitations

An explanation is a model of a model. It is an approximation, and it can be confidently wrong. Two methods can produce different explanations for the same prediction, and there is often no principled way to adjudicate.

More seriously, explanations describe correlation within the model, not causation in the world. SHAP telling you postcode drove a decision does not mean postcode causes the outcome; it may proxy for something else entirely — which is precisely why explanations are useful for fairness auditing and cannot be read as causal claims.

There is also a documented risk that explanations create unwarranted confidence. Stakeholders shown a plausible explanation tend to trust the underlying model more, whether or not it deserves it. And explanations can be gamed: it is possible to construct models that behave one way and explain themselves another.

The practical position: prefer an inherently interpretable model when the accuracy cost is small, and it often is on tabular data. Where a complex model is genuinely needed, use post-hoc explanations for debugging, bias investigation, and regulatory response — but treat them as evidence to investigate rather than as ground truth, and validate them against domain expertise. See confusion matrix for the complementary question of which errors the model makes.