What is Neural Network?

The plain definition

A neural network is a stack of layers. Each layer takes a list of numbers, multiplies them by a matrix of weights, adds a bias, and passes the result through a non-linear function. Stack enough of these and the whole thing becomes a very flexible function approximator: given examples of inputs and correct outputs, it can learn a mapping between them.

The biological metaphor is where the name comes from, and it is where most explanations go wrong. A neural network does not think, reason, or understand. It is arithmetic — a large pile of multiplications arranged so that a gradient can be computed and followed downhill.

How training actually works

Training has three moving parts. A loss function measures how wrong the current predictions are. Backpropagation computes, for every weight in the network, the derivative of that loss with respect to that weight. An optimiser (usually Adam or SGD) nudges each weight a small step in the direction that reduces loss.

Repeat that loop over batches of training data for a few thousand to a few million steps and the weights settle into a configuration that makes good predictions. Nothing else happens. There is no insight, only a long descent down an error surface.

The non-linearity matters more than people expect. Without it — if every layer were purely a matrix multiply — a hundred stacked layers would collapse algebraically into a single matrix, and the network could only learn straight lines. The non-linear activation (ReLU, GELU, tanh) is what buys the ability to model curves, interactions, and conditional logic.

When a neural network is the wrong choice

This is the part most introductions skip. Neural networks are strong on perceptual data — images, audio, free text — where the useful features are buried in raw signal and nobody can write them down by hand.

On tabular business data (rows of numbers and categories from a database), gradient-boosted trees such as XGBoost or LightGBM usually match or beat a neural network while training in seconds instead of hours, needing far less data, and producing feature importances you can actually explain to a stakeholder. If your problem is churn prediction on 40 columns of CRM data, a neural network is very often the more expensive way to get a slightly worse answer.

Neural networks also want data volume. With a few hundred labelled examples, a deep network will memorise rather than generalise — see overfitting. Regularisation helps; it does not conjure signal that is not there.

The vocabulary you will hear

Epoch — one full pass over the training set. Batch size — how many examples are averaged before each weight update. Learning rate — how big each step is; the single most important hyperparameter to get right. Depth — number of layers. Width — units per layer.

Vanishing gradients — in very deep networks the derivative can shrink toward zero as it propagates backward, so early layers stop learning. Residual connections (the core trick in ResNets and in every transformer) exist mainly to give the gradient a clean path back.