What is Hyperparameter Tuning?

Parameters versus hyperparameters

Parameters are learned. The weights in a neural network are fitted by gradient descent, and there may be billions of them.

Hyperparameters are chosen. Learning rate, batch size, number of layers, dropout rate, weight decay, tree depth, number of estimators. Nobody derives them from the data; they are set before training starts and they materially change the outcome. Tuning is the search for good values.

The methods, and which are worth using

Grid search tries every combination on a predefined lattice. It is exhaustive, trivially parallel, and scales terribly — five hyperparameters at four values each is 1,024 training runs. Reserve it for two or three parameters at most.

Random search samples combinations at random from ranges. Counterintuitively this beats grid search on the same budget, and the reason is worth understanding: usually only two or three hyperparameters genuinely matter, and random search explores many distinct values of those, while grid search wastes most of its budget re-testing the important parameters at the same few lattice points. Random search is a strong default.

Bayesian optimisation builds a probabilistic model of the objective and picks the next configuration expected to be most informative. It reaches good configurations in fewer trials, which pays off when each trial is expensive. Optuna and similar libraries make it practical.

Successive halving and Hyperband exploit a simple observation: bad configurations usually look bad early. Start many trials with a small budget, kill the worst half, give the survivors more budget, repeat. On deep learning workloads this is often the highest return per GPU-hour available.

Neural architecture search automates architecture design itself. It is real, it works, and the compute cost places it outside the reach of most teams. Use a published architecture.

Where to spend the effort

Hyperparameters are not equally important, and treating them as such wastes budget. For neural networks, learning rate dominates everything else — an order of magnitude wrong will prevent convergence entirely, and it is worth tuning alone before touching anything else. Then batch size, then regularisation strength, then architecture width and depth.

For gradient-boosted trees, the sensitive knobs are learning rate together with number of estimators (they trade off directly), then max depth, then subsampling rates and minimum samples per leaf.

Search over log scales, not linear ones. Learning rates should be sampled across 1e-5 to 1e-1 logarithmically, because the interesting variation is multiplicative. Sampling linearly between 0.00001 and 0.1 puts almost every sample in the same uninteresting region.

Overtuning: the failure hiding inside the practice

Every hyperparameter decision made by looking at validation performance leaks a little information from the validation set into the model. Run a thousand trials and pick the best, and some of that winning margin is luck — the configuration that happened to suit the noise in that particular split. Reported validation performance becomes optimistic, and the gap shows up in production.

Defences are procedural. Keep a test set that is used once, at the very end. On smaller datasets, use nested cross-validation so the tuning loop is evaluated on data it never selected against. Prefer a simpler configuration when its performance is within noise of a complex one. And treat a modest tuning gain with suspicion: if the top fifty configurations are within half a percent of each other, you have found a plateau rather than a peak, and picking the exact maximum is not meaningfully better than picking a robust point on it.

Finally, keep perspective on where the returns are. On most applied projects, better features and cleaner training data outperform hyperparameter tuning by a wide margin. Tuning is polish, applied after the fundamentals are right.