What is Confusion Matrix?

The plain definition

For a binary classifier there are four possible outcomes per prediction. True positive — predicted yes, actually yes. False positive — predicted yes, actually no. True negative — predicted no, actually no. False negative — predicted no, actually yes.

Arranged in a two-by-two table with actuals on one axis and predictions on the other, that is the confusion matrix. The reason it matters is that a single accuracy figure collapses all four counts into one number and destroys the distinction between error types — which is usually the distinction the business cares most about.

Why accuracy misleads

Suppose one transaction in a thousand is fraudulent. A model that predicts 'not fraud' for every input achieves 99.9% accuracy and catches nothing. Accuracy is dominated by the majority class, so on any imbalanced problem it is close to meaningless.

The metrics that survive imbalance are derived from the matrix. Precision is true positives over all predicted positives — of the cases we flagged, how many were real. Recall (sensitivity) is true positives over all actual positives — of the real cases, how many did we catch. F1 is their harmonic mean, useful as a single summary and worth treating with suspicion, because it implicitly asserts that precision and recall matter equally, which is rarely true.

Specificity is recall for the negative class. In screening contexts it is the number that determines how many healthy people get an alarming letter.

The trade-off is a business decision

Precision and recall move against each other, and the dial that controls them is the decision threshold. Most classifiers output a probability; the threshold converts it into a decision. Lower the threshold and you catch more real cases while raising false alarms. Raise it and your flags become more reliable while more real cases slip through. The default of 0.5 carries no special authority and is very often wrong.

Choosing it correctly requires the cost of each error type. If a missed fraudulent transaction costs £400 and reviewing a false alarm costs £3, then roughly one hundred and thirty false alarms are worth preventing one miss, and the threshold should be set accordingly — low. If a false positive means wrongly accusing a customer, or an unnecessary invasive medical procedure, the calculus inverts.

This is why the confusion matrix belongs in conversations with stakeholders, not just in model reports. Precision and recall are the technical names for 'how often we cry wolf' and 'how often we miss', and domain experts have well-formed views about which is worse once the question is put to them in those terms.

Threshold-free curves, and multi-class

Because the matrix depends on a chosen threshold, two curves summarise performance across all thresholds. ROC-AUC plots true positive rate against false positive rate; it is threshold-independent and, on heavily imbalanced data, misleadingly flattering, because the false positive rate has a very large denominator. PR-AUC plots precision against recall and is the more informative choice when positives are rare.

For more than two classes the matrix becomes N-by-N, with actuals down one axis and predictions across the other. The off-diagonal cells are the useful part: they show precisely which classes get mistaken for which. That is diagnostic information a single accuracy number cannot give you — if two categories account for most of the confusion, the fix may be merging them, adding features that distinguish them, or improving the labelling guideline that annotators found ambiguous.