Why AI projects fail — and the four questions that predict it

The failure is almost never the model

There is a persistent assumption that AI projects fail for technical reasons — the wrong architecture, insufficient compute, a model that was not clever enough. In our experience that is rare. The modelling stage is usually the part that goes to plan.

Failures cluster at the two ends. At the front, in problem framing: the thing being predicted is not the thing the business cares about, or the prediction arrives too late to change a decision, or a feature that made the model accurate in testing will not exist at prediction time. At the back, in deployment: the model works but nothing consumes it, or it degrades quietly and nobody notices.

What follows are the four questions we now ask during scoping. They are unglamorous, and each one has killed or reshaped projects before budget was committed — which is the point.

1. What decision changes because of this prediction?

This question sounds trivial and it eliminates a surprising number of proposals. If nobody can name the decision, the specific person or system making it, and what they would do differently, the project has no route to value regardless of how accurate the model is.

Two follow-ups matter as much. When is the decision made? A churn prediction delivered after the customer has cancelled is an interesting statistic. A demand forecast that arrives after purchase orders are placed changes nothing. The prediction must land inside the window where action is still possible, and that window is often narrower than people assume.

And how accurate does it need to be to change behaviour? If a purchasing manager will only override their own judgement when a forecast is right 90% of the time, and the achievable accuracy is 75%, the model will be built and ignored. That is worth discovering in a scoping conversation rather than in month five.

2. Will this feature exist at prediction time?

This is target leakage, and it is the single most common way a project produces a model that scores brilliantly in a notebook and is worthless in production.

The canonical example: a churn model that includes cancellation_reason. It will score in the mid-nineties, because that field is essentially the answer. In production it is empty until after the customer has already left. The same pattern appears as final_invoice_amount in a cost prediction, resolution_code in a ticket-routing model, or a status field that only gets populated once the outcome is known.

Leakage also arrives through time. If you split a time-ordered process randomly, the model trains on the future and validates on the past, and the reported accuracy is fiction. Anything that must run forward in time needs a strict temporal split. It arrives through grouping too: if the same customer, patient, or device appears in both training and validation, you are measuring memorisation of that entity rather than generalisation to new ones.

The discipline that catches all three is to write down, for every feature, the exact moment its value becomes available — and compare that against the moment the prediction is needed. It is tedious and it is cheaper than the alternative. See supervised learning and training data for more.

3. What does the baseline actually achieve?

Before any machine learning, we build something deliberately unambitious: a simple rule, a moving average, or gradient-boosted trees on the obvious features. This is not a formality or a warm-up.

It establishes the reference point that makes every later result interpretable. A model at 82% accuracy sounds respectable until you learn that predicting the majority class gets 79%. Conversely, a modest-looking improvement over a strong baseline can be worth a great deal. Without the baseline, nobody in the room can tell which situation they are in — including the people building it.

It also frequently ends the project early in the best possible way. A meaningful share of problems are adequately solved by the simple version, and a deployed, explainable, fast model that changes a decision is worth far more than a sophisticated one that is still in development. On tabular business data specifically, well-tuned gradient-boosted trees are hard to beat — see XGBoost vs neural networks.

4. Who owns this in twelve months?

Machine learning systems are unusual in that they degrade without anyone touching them. Ordinary software breaks when someone changes code. A model breaks when the world changes — customer behaviour shifts, a supplier changes packaging, fraud tactics adapt. Nothing throws an error. The service returns HTTP 200 for every request while its predictions quietly get worse. See model drift.

So ownership has to be real, and it needs three things attached: monitoring that would actually detect degradation, an agreed retraining cadence derived from measured decay rather than habit, and a rollback path that someone has tested. If the answer to who owns it is a consultancy that will have moved on, or a data scientist who is leaving, the project has a defined shelf life whether or not anyone has said so.

This is also why we treat the handover documentation as a deliverable rather than a courtesy — specifically the honest part. What the model is bad at, which inputs put it outside its training distribution, what error rate to expect on which segments, and what should trigger a retrain.

The uncomfortable one

There is a fifth question that belongs here, though it rarely gets asked by the party being paid to build the system: does this need machine learning at all?

A well-written rule engine, a better search index, a redesigned form, or fixing the data-entry process upstream resolves a meaningful share of problems that arrive described as AI problems. Those solutions are faster to build, cheaper to run, and have behaviour you can reason about and explain to a regulator.

The projects that succeed tend to be the ones where someone was willing to ask that question early and accept the answer. The ones that fail expensively are frequently the ones where the technology was chosen before the problem was understood.