What is Transfer Learning?

The plain definition

Training a large model from random initialisation requires enormous data and compute. Transfer learning sidesteps that. Someone else trains a model on a very large general corpus — hundreds of millions of images, or trillions of tokens of text — and that model learns broadly useful internal representations. You then start from those weights and adapt them to your specific problem.

The intuition is that low-level structure is shared. A vision model's early layers learn edges, textures, and gradients, which are the same whether the final task is identifying tumours or grading avocados. Only the later, more task-specific layers need to change. In language, the same logic applies to syntax, coreference, and factual association.

The four strategies, cheapest first

Feature extraction. Freeze the pretrained network entirely, use its output vectors as inputs to a small classifier such as logistic regression or a gradient-boosted tree. Fast, cheap, works with a few hundred examples, and a surprisingly strong baseline that teams skip too readily.

Full fine-tuning. Unfreeze all weights and continue training on your data at a low learning rate. Highest ceiling on accuracy, highest cost, and the highest risk of catastrophic forgetting, where the model loses general capability while over-adapting to your narrow set.

Parameter-efficient fine-tuning. Freeze the base weights and train a small number of new parameters — LoRA adapters are the dominant form. You get most of the benefit of full fine-tuning at a small fraction of the memory and storage cost, and you can host many task-specific adapters against one shared base model. For language models this is now the default choice.

In-context learning. Change nothing about the weights; put examples in the prompt. Zero training cost, instant iteration, but capped capability and a recurring inference cost that scales with prompt length. See prompt engineering and fine-tuning for where the line between these sits.

When transfer fails

Transfer learning is not free lunch, and its failure mode has a name: negative transfer, where starting from pretrained weights produces worse results than starting fresh. It happens when the pretraining domain is genuinely unlike the target domain. Natural-image models transfer poorly to certain scientific imaging modalities — seismic volumes, some medical scans, spectrograms — where the statistics of the signal share little with photographs.

The other trap is evaluation contamination. If your test set overlaps with the pretraining corpus, your measured performance is fiction. This is a live concern with large language models scraped from the public web: any benchmark, any published dataset, and quite possibly your own documentation may already be in the training data. When it matters, build a held-out evaluation set from data created after the model's cutoff, or from private data that was never published.

Why it changed the economics

Transfer learning is the reason applied machine learning became accessible to organisations that are not research labs. A decade ago a serious vision system meant a large labelled corpus and a GPU cluster. Now a well-chosen pretrained backbone plus a few thousand labelled examples reaches production quality on a single machine.

The practical consequence for project planning: your first question should not be 'what architecture should we train' but 'what has already been trained that is close to this, and what is the cheapest adaptation that clears our accuracy bar'. Working up from feature extraction is almost always faster than working down from full training.