Every team building on language models eventually hits the same wall: the model, as it arrives from the provider, is not quite right for the job. It hallucinates details about your product, ignores your formatting rules, or answers in a tone that does not fit your brand. At that point the conversation turns to the three standard remedies, prompt engineering, retrieval-augmented generation, and fine-tuning, and it tends to turn to them in a confused way, because the three are usually presented as competing options on a single spectrum of effort.
They are better understood as different tools for different failure modes. Prompt engineering changes the instructions a model receives, retrieval-augmented generation (RAG) changes the information it has access to, and fine-tuning changes the model itself. The reason teams choose wrong is rarely that they misjudge the techniques; it is that they have not diagnosed which kind of failure they actually have. This post is organized around that diagnosis.
Three interventions, three layers
Consider what actually reaches the model on a single API call: a prompt containing instructions and content, processed by a set of trained weights. There are only three places to intervene.
Prompt engineering edits the instructions. Clearer task descriptions, output format specifications, worked examples (few-shot prompting), explicit constraints on what to do when unsure. It requires no training and no infrastructure, takes effect on the next request, and its cost is developer iteration time plus the extra tokens the instructions consume on every call.
RAG edits the information. At request time, the system searches a document collection for material relevant to the user's query, using embeddings to find semantically similar passages, and inserts what it finds into the prompt before the model sees it. The model then answers from the supplied passages rather than from whatever its training data happened to contain. Our RAG explainer covers the pipeline in full.
Fine-tuning edits the model. Training continues on your own examples, typically a few hundred to a few thousand demonstration pairs, so the desired behavior moves from the prompt into the weights. Parameter-efficient methods, LoRA being the standard, make this affordable by training small adapter matrices instead of the full model; the mechanics are covered in our quantization, LoRA, and distillation explainer.
Diagnose by failure mode, not by ambition
The selection question becomes tractable when framed as: when the model fails, what is it missing?
It is missing knowledge. The model does not know your internal documentation, your product catalog, this quarter's pricing, or anything that post-dates its training data. Wrong answers here are confident fabrications or honest refusals. This is RAG's territory, and it is worth being direct about why fine-tuning is the wrong reflex: models do not reliably absorb facts from fine-tuning runs, the original RAG work was motivated exactly by the limits of knowledge stored in parameters, and weights-encoded knowledge cannot be updated without another training run or audited for provenance. Retrieval keeps knowledge current by re-indexing documents, and lets every answer point at its source.
It is missing instructions. The model has the knowledge and the capability but was not told, or not told clearly enough, what you want. Symptoms: output that is right in substance but wrong in shape, inconsistent handling of edge cases, verbosity where you wanted brevity. This is prompting's territory, and it extends further than most teams expect. A substantial fraction of what gets attributed to model limitations is instruction quality, which is why the honest advice is to exhaust prompting before reaching for anything heavier: iteration is free, and every later technique assumes a well-prompted baseline anyway.
It is missing behavior. The model, even well-instructed and well-supplied, does not consistently do the thing: it drifts from a strict output schema, cannot hold a specialized voice, or underperforms on a narrow task that general training did not emphasize. When examples in the prompt fix it but only at the cost of hundreds of tokens per request, that is the signal that the behavior belongs in the weights. This is fine-tuning's territory: consistency, style, format reliability, and narrow-task skill.
The costs, honestly
Each approach carries a different cost shape, and the shapes matter more than the absolute numbers.
| Prompt engineering | RAG | Fine-tuning | |
|---|---|---|---|
| Upfront cost | hours of iteration | embedding pipeline + vector store | dataset preparation (the real cost) + training run |
| Per-request cost | higher input tokens | retrieval latency + context tokens | often lower (instructions move into weights) |
| Keeping it current | edit the prompt | re-index documents | re-train |
| Failure visibility | immediate | retrieval quality is measurable | regressions surface in evaluation only |
| Skill required | prompt design | search + data pipeline | ML training + evaluation |
Two of these rows decide most real cases. The keeping-it-current row is why knowledge belongs in RAG: documents change weekly, and re-indexing is cheap while re-training is a project. The per-request row is why high-volume narrow tasks eventually justify fine-tuning: a 600-token instruction-and-examples preamble on every one of a million daily requests is a real bill, and moving it into the weights pays for the training run quickly. The dataset caveat deserves emphasis, though: the training run itself is often tens to hundreds of dollars with parameter-efficient methods, but assembling a few thousand high-quality demonstration pairs is days to weeks of skilled work, and dataset quality dominates the outcome.
There is also a quieter cost to fine-tuning that only shows up later: a custom model is yours to maintain. Base models improve on a fast cadence, and every upgrade of the underlying model invalidates your adapter, which means re-training and re-evaluating. Teams that fine-tuned casually in 2024 spent 2025 discovering this treadmill.
The economically interesting case for fine-tuning
One scenario deserves separate mention because it changes the budget conversation entirely: fine-tuning a small model to replace a large one on a narrow task. A general-purpose frontier model handling a high-volume classification, extraction, or routing-adjacent task is usually overkill, and small language models in the 1-10B range, fine-tuned on a few thousand task examples, routinely reach or exceed frontier quality on that one task at a small fraction of the serving cost.
Here fine-tuning is not fixing a failure at all. It is an optimization: trading upfront training effort for a permanently cheaper unit economics on a workload you have already validated at volume. The precondition is real: you need the volume to amortize the work, and you need evaluation data to prove the small model actually matches quality on your distribution rather than on a benchmark's. That precondition is why this is a scale-stage move, not a first move.
Composition, and the order of operations
The three approaches compose, and production systems of any maturity typically run more than one. A support assistant might carry prompt-engineered task instructions and guardrails, retrieve from a continuously updated help-center index, and, at sufficient scale, run on a model fine-tuned for the product's domain vocabulary. Nothing about the techniques conflicts, because each occupies its own layer.
What should be sequential is the investment. Prompting first, always: it is free to iterate, it establishes the baseline every other technique is measured against, and it frequently ends the conversation. RAG second, when the diagnosis is missing knowledge, which for products built on private or fast-moving information is nearly guaranteed. Fine-tuning last, when a well-prompted, well-grounded model still exhibits behavior problems, or when the volume-driven optimization case above applies. Teams that invert this order pay for training runs to fix problems a paragraph of instructions would have solved.
One more decision interacts with all three: which model sits underneath. A stronger base model often needs less adaptation, prompting further, following instructions more reliably, and using retrieved context more faithfully, which shifts the whole calculus, while different requests within one product may deserve different models entirely. That selection problem is its own topic, covered in how to choose the right AI model and, for the per-request version, model routing. The adaptation techniques in this post assume a model choice; it is worth making that choice deliberately rather than inheriting it.
A worked diagnosis
To close, three miniature case studies, each one paragraph, mapping symptom to remedy.
A legal-tech assistant invents clause numbers when asked about specific contracts. The knowledge (the contracts) exists but not inside the model, and it changes with every new client. Diagnosis: missing knowledge. Remedy: RAG over the contract store, with citations surfaced to the user, and no amount of fine-tuning would have been an acceptable substitute.
An e-commerce team's product-description generator writes good copy in the wrong voice, and keeping it on-brand requires a 900-token style guide plus four examples in every prompt. The behavior is achievable but expensive to specify repeatedly. Diagnosis: missing behavior, at volume. Remedy: fine-tune on a few thousand approved descriptions, drop the preamble, and keep a short prompt for the per-product facts.
A data team's SQL helper produces queries that are usually correct but formatted inconsistently and occasionally destructive. Diagnosis: missing instructions, the model was never told the house dialect rules or forbidden operations. Remedy: a tighter prompt with explicit constraints and two examples, plus structured outputs for the response envelope. Total cost: an afternoon.
The pattern across all three: the remedy follows from the diagnosis, and the diagnosis is usually not subtle once the question is asked as "what is the model missing?" rather than "which technique is best?"