What is Transformer?

The problem it solved

Before transformers, sequence models were recurrent: they read one token at a time, carrying a hidden state forward. That created two hard limits. Information from early tokens degraded as it was repeatedly overwritten, so long-range dependencies were weak. And because step t required the output of step t−1, training could not be parallelised across the sequence.

The 2017 transformer architecture removed recurrence entirely. Every token attends to every other token directly, in one operation, and the whole sequence is processed in parallel. The long-range dependency problem largely disappeared, and — decisively — training could now saturate GPU clusters. Nearly every capability advance since has been downstream of that one change.

How attention actually works

Each token is projected into three vectors: a query (what am I looking for), a key (what do I offer), and a value (what I will contribute). To compute a token's new representation, you take the dot product of its query with every other token's key, which yields a relevance score for each; softmax those into weights; and take the weighted sum of the corresponding values.

The result is that each token's representation becomes a blend of the tokens that matter to it. In the sentence 'the invoice that the supplier disputed was paid', attention lets 'was paid' consult 'invoice' directly, with no intermediate steps to erode the signal. Multi-head attention runs several of these in parallel with different projections, so different heads can specialise — syntax, coreference, factual association.

Two supporting pieces are necessary. Positional encoding injects word order, since attention alone is order-blind. And residual connections plus layer normalisation keep gradients healthy through dozens of stacked layers.

The cost, and what it means for you

Attention compares every token to every other token, so compute and memory scale with the square of sequence length. Doubling context roughly quadruples the attention cost. This is why long-context models were slow to arrive and why they remain expensive, and it is the reason a large body of work exists on approximations — sparse attention, sliding windows, linear attention, and the memory-layout optimisations in FlashAttention.

For anyone building on top of these models, the practical consequence is that context length is a real budget, not a free parameter. Sending an entire document collection in the prompt is both slower and more expensive than retrieving the three relevant passages, which is precisely the argument for retrieval-augmented generation.

Three variants are worth distinguishing. Encoder-only models such as BERT are built for understanding tasks — classification, extraction, embeddings. Decoder-only models generate text and underpin every modern large language model. Encoder-decoder models suit transformation tasks such as translation.