Almost every time someone tells me "we need fine-tuning," what they actually need is better context. Fine-tuning is the first word out because it sounds like the serious solution, the one that "teaches" the model. But in product I've changed very few behaviors with fine-tuning and a huge number with a good prompt or with well-built RAG. The rule that has worked for me is to start cheap and reversible, and only climb a rung when the previous one proves to be capped. This article is that decision tree, with the trade-offs I accept on each branch and the signals that tell me when I've chosen wrong.
First I ask: is this a knowledge problem or a behavior problem?
This is the fork that orders everything else. If the model fails because it doesn't know something —your domain data, internal documentation, facts after its training cutoff— it's a knowledge problem, and knowledge is injected through context, not baked into weights. There, RAG is the default answer.
If the model knows what it needs but consistently responds with the wrong format, tone, or structure, it's a behavior problem. And behavior is a candidate for fine-tuning… but only after prompting falls short, which is later than people think.
Confusing these two branches is the most expensive mistake I see. Nobody fixes the model not knowing your product catalog with fine-tuning: you train it today and tomorrow the catalog changed. And nobody fixes the model refusing to return clean JSON with RAG: no matter how many documents you feed it, the problem is shape, not data.
Prompting: the rung I almost always underestimate
My starting point is always prompting, because it's the only change I ship in minutes and revert in seconds. Before touching anything else, I squeeze clear instructions, representative few-shot examples, and an explicit output structure. At JXBS, a good chunk of what looked like it needed a fine-tuned model was solved with three well-chosen examples in the prompt and a strict output schema.
The cost of this branch is context: every example you add takes tokens you pay for on every call, and a 20-example prompt becomes expensive and slow at scale. That's exactly the symptom that maybe it's time to climb a rung: when you need so many examples that the prompt is half your bill, the pattern is already there and it may be worth baking in.
💡 If you haven't tried to solve it with prompting for at least an afternoon, you don't have the data to justify fine-tuning. "We tried a prompt and it didn't work" isn't prompting, it's an anecdote.
RAG: for knowledge that changes and must be citable
When the problem is knowledge, RAG almost always wins for a reason beyond the technical one: traceability. A RAG system can tell you which document the answer came from. A fine-tuned model gives you the answer from inside its weights, with no source, and when it hallucinates you have nowhere to look.
// Knowledge lives outside the model and updates without retraining
const context = await searchChunks(question, { topK: 5 });
const answer = await llm.complete({
system: 'Answer only with information in <context>. If it is not there, say so.',
context,
question,
});
The trade-off I accept with RAG is operational: now I maintain an ingestion pipeline, embeddings, a vector index, and a chunking strategy. It's more infrastructure than a prompt. In exchange, I update knowledge by changing documents, not retraining, and in a fast-moving domain that's worth gold.
Fine-tuning: the last rung, and I know why I'm climbing it
I reach fine-tuning only when three things hold at once: the behavior I want is stable and repeated, prompting gives it to me but at an unsustainable token cost, and I have enough quality real examples to train on. If any of the three is missing, I don't climb.
What I buy with fine-tuning is a model that does what I want with a short prompt, cheaper and faster per call. What I pay is rigidity: every behavior change is a retrain, the dataset becomes an asset you have to version and care for, and I've introduced an artifact that can go stale silently. Fine-tuning doesn't eliminate RAG either: the most powerful thing I've built combines a model fine-tuned on how to answer with RAG for what to know.
The mistake of skipping rungs
The antipattern that's cost me most is starting at the end. Teams that build fine-tuning for a problem a prompt would have solved, and end up with an expensive-to-maintain model that performs no better than the baseline they never measured. Starting cheap isn't being conservative: it's generating the evidence that justifies the next step. When you finally do fine-tuning after exhausting prompting and RAG, you know exactly what you're buying and why. That clarity is the difference between an engineering decision and a purchase driven by hype.
