What is Overfitting?

The plain definition

A model that has overfitted has learned the training set rather than the pattern behind it. Training error keeps falling. Validation error stops falling and starts to rise. The gap between the two is the signature.

The mechanism is straightforward. Real data contains signal and noise. A model with enough capacity will fit both, because fitting noise also reduces training loss. Noise does not recur in new data, so every unit of noise the model absorbed is a unit of error it carries into production.

How to detect it honestly

You need data the model has never touched. The standard split is three-way: train to fit weights, validation to make choices about architecture and hyperparameters, and test touched exactly once, at the end, to estimate real-world performance.

The discipline that gets violated is the test set. Every time you look at test performance and change something in response, you leak information from it, and its value as an unbiased estimate decays. Teams that tune against the test set for six weeks routinely ship models that underperform their reported numbers by a wide margin. The failure is procedural, not technical.

For time-series and any forward-running process, random splits are invalid — they let the model train on the future. Use forward-chaining validation, where each fold trains on data up to time t and validates on t+1. And for grouped data, split by group: if the same patient, customer, or device appears in both train and validation, you are measuring memorisation of that entity rather than generalisation to new ones.

What actually fixes it

More data is the most reliable fix and usually the least available. Noise averages out as sample size grows; signal does not.

Less capacity. Fewer layers, fewer parameters, shallower trees. Cheap to try and often sufficient — a smaller model that generalises beats a larger one that memorises.

Regularisation. L2 weight decay penalises large weights. L1 pushes weights to exactly zero, which also selects features. Dropout randomly disables units during training so the network cannot rely on any single path. In tree ensembles, limit depth and increase the minimum samples per leaf.

Early stopping. Track validation loss each epoch and keep the weights from the best epoch rather than the last. Nearly free and remarkably effective.

Data augmentation. Generate plausible variants — flips, crops, and colour shifts for images; paraphrase and back-translation for text; jitter and warping for sensor traces. This expands effective dataset size without new labelling.

Cross-validation does not prevent overfitting but gives a far more stable estimate of generalisation on small datasets, which stops you fooling yourself about whether a change helped.

Underfitting, and why the tidy story is incomplete

The opposite failure is underfitting: both training and validation error are high because the model lacks the capacity or training time to capture the pattern. The classic advice is to find the sweet spot on the bias-variance trade-off between the two.

That framing is useful but no longer complete. Very large overparameterised networks exhibit double descent: as capacity grows past the point of interpolating the training data, test error rises as classical theory predicts, and then falls again. This is why models with billions of parameters generalise at all. The practical takeaway is not to abandon regularisation but to stop assuming that a model with more parameters than data points must be overfitting — measure it on held-out data rather than reasoning from parameter counts.