What is Fine-Tuning?

The plain definition

Fine-tuning starts from a model that already works and continues training it on your examples at a low learning rate. It is the most direct form of transfer learning, and it changes the weights — which is what distinguishes it from prompting, where the model is unchanged and merely instructed.

What fine-tuning is genuinely good at is behaviour: reliably producing a specific output structure, adopting a house style, using domain vocabulary correctly, or performing a narrow classification task with high consistency. What it is bad at is knowledge. Teaching a model new facts by fine-tuning is inefficient, hard to update, and impossible to cite. Facts belong in retrieval.

LoRA and why the economics changed

Full fine-tuning updates every weight, which requires memory for the weights, their gradients, and optimiser state — historically putting large models out of reach for most teams, and producing a complete model copy per task.

Parameter-efficient methods changed that. LoRA freezes the base model and trains a pair of small low-rank matrices injected alongside the original weights. The trainable parameter count drops by orders of magnitude, memory requirements collapse, and the resulting adapter is a small file. QLoRA goes further by holding the frozen base in quantised form, bringing sizeable models within reach of a single GPU.

The operational consequence is as important as the cost saving: you can serve many task-specific adapters against one shared base model, swapping them per request. That turns fine-tuning from a heavyweight commitment into something closer to configuration.

The decision, and how to not waste the effort

Work up the ladder rather than starting at the top. First try a careful prompt with a few examples — prompt engineering is free and instant. If the gap is missing knowledge, add retrieval. Only fine-tune when prompting plateaus on a behaviour you need and you have the data to teach it. Fine-tuning is justified when you need consistency prompting cannot deliver, when a smaller fine-tuned model can replace a larger general one and cut inference cost, or when you want to shorten prompts by baking instructions into weights.

Data quality dominates volume here to an unusual degree. A few hundred carefully curated, consistent examples routinely beat tens of thousands of noisy ones, because the model will faithfully learn any inconsistency in your labels. Every example should demonstrate exactly the behaviour you want, in exactly the format you want.

Two failure modes to plan for. Catastrophic forgetting — hard fine-tuning on a narrow set degrades general capability; mitigate with lower learning rates, fewer epochs, and mixing in some general data. And maintenance — a fine-tune is pinned to a base model version, so provider upgrades mean re-running the job. Keep the training data and pipeline reproducible, and keep an evaluation set that can tell you whether the new version is actually better.