Prompting vs RAG vs fine-tuning
Your model gives wrong or off-style answers. Should you fix the prompt, add retrieval or fine-tune? Learn what each approach changes, a simple decision order, and why fine-tuning is the last step for most apps.
A common beginner question is: "My model doesn't know my data or doesn't behave how I want. Should I fine-tune it?" Usually the answer is no, not yet. There are three main ways to improve an AI feature, and they fix different problems. Choosing the wrong one wastes weeks.
This lesson gives you a way to choose, with simple examples. It uses the idea of an exam: prompting is giving clear instructions, RAG is an open-book exam, and fine-tuning is months of extra coaching.
The three tools
| Approach | What it changes | Analogy | Typical use |
|---|---|---|---|
| Prompting | The instructions and examples in each request | Briefing a smart new employee | Tone, format, rules, a few examples |
| RAG | The facts the model sees, fetched per question | Open-book exam with the right pages open | Company documents, menus, policies, anything that changes |
| Fine-tuning | The model's weights, through extra training on examples | Coaching until a skill becomes a habit | A consistent style or format, a narrow task done cheaply at scale |
They are not rivals. Real systems often combine them: good prompts, RAG for facts, and sometimes a fine-tuned model on top.
Start with prompting
Prompting is the cheapest and fastest to try. You can iterate in minutes, and a better model often fixes problems for free. Before you consider anything else, you should have:
Before you look further
- Clear instructions with role, task, constraints and output format (see *Anatomy of a good prompt*).
- 3 to 5 good examples in the prompt (few-shot), covering normal and tricky cases.
- A test set of real inputs with a score, so you can tell if changes help (see *Evaluating AI apps (evals)*).
- Tried at least one stronger model, in case the problem is model capability.
Long prompts cost more tokens on every call, and examples take context space. That is the limit of prompting: if you need to teach something that does not fit in a prompt, or you pay for thousands of tokens of instructions on every request, look at the other two.
Use RAG when the problem is knowledge
A model does not know your private documents, yesterday's prices or a policy updated this morning. It was trained on data up to a cutoff date. If the failure is "it doesn't know X" or "it made up X", the fix is to give it X at question time: retrieve the relevant chunks and put them in the prompt (see *Retrieval-augmented generation (RAG) end to end*).
RAG has real advantages:
| Advantage | Why it matters |
|---|---|
| Fresh data | Update the documents, not the model. A price change is a database edit. |
| Citations | You can show which document an answer came from, so users can verify |
| Access control | You can retrieve only what this user is allowed to see |
| No training needed | Works with any model and costs far less to set up |
| Easy to fix | If an answer is wrong, you can inspect which chunks were retrieved |
Fine-tuning is a poor way to teach facts. Models trained on facts can still forget or misstate them, and you cannot easily remove or update a fact later. Use RAG for knowledge.
Use fine-tuning when the problem is behaviour
Fine-tuning means training an existing model a little more on your own examples, each an input and the ideal output. It is good at changing how the model behaves, not what it knows:
Good reasons to fine-tune
- A consistent brand voice or writing style that is hard to get with prompts alone.
- A strict output format that must be followed every time.
- A narrow, repetitive task (classify support tickets into 12 categories, extract fields) where a small fine-tuned model can match a big one at lower cost and delay.
- Replacing a very long prompt of rules and examples with learned behaviour, to save tokens at high volume.
- Running a small model locally (see *Running open models locally with Ollama*) that must do one job well.
What fine-tuning needs: a few hundred to thousands of high-quality example pairs (the quality matters more than the count), a held-out test set, time to run and compare experiments, and a plan to repeat the work when a better base model arrives. Many cloud providers offer hosted fine-tuning for some of their models, and open models can be tuned with libraries from the Hugging Face ecosystem. Cost, supported models and methods change often, so check the provider's current documentation.
One common technique is LoRA, which trains a small set of extra weights instead of the whole model. It makes tuning open models possible on modest hardware, for example a free Google Colab GPU for small models.
A decision guide
| Symptom | Likely fix |
|---|---|
| Answers are the wrong format or tone | Better prompt, add examples |
| Model says "I don't know" or invents facts about our data | RAG |
| Information changes weekly | RAG (never fine-tune for changing facts) |
| Need citations or per-user permissions | RAG |
| Prompt is huge and the bill is high at scale | Prompt caching first; then consider fine-tuning a smaller model |
| Task is narrow, we have 1,000+ labelled examples and big volume | Fine-tune a small model, compare with your eval set |
| Model is not smart enough even with good prompts | Use a stronger model before anything else |
| Style must match a specific person or brand | Examples in prompt first; fine-tune if that is not enough |
A practical order: better prompt, then better model, then RAG for knowledge, then fine-tuning for behaviour, measuring with evals at each step. Stop as soon as the quality is good enough.
Worked example: Brew & Bean
Brew & Bean wants a WhatsApp-style assistant. The first version answers menu questions wrongly. Steps a good engineer takes:
1. Put the menu and rules in the system prompt, add 3 example conversations. Evals show prices are now right, but the bot cannot answer questions about the monthly offers PDF.
2. Add RAG over the offers PDF and the FAQ. Evals show correct, cited answers. Offers can now change without code changes.
3. Months later, the shop sends 50,000 order-status messages and wants every reply in a fixed friendly format at low cost. They collect 2,000 reviewed replies and fine-tune a small model. They compare it on the eval set against the prompted large model, and only switch if the quality holds.
Notice that fine-tuning came last, for a narrow high-volume task, and was justified by a test.
Training data is a privacy risk
Do not fine-tune on customer data you have no right to use. Personal data in training examples can leak back out in answers. Remove or mask personal data and follow your obligations under the DPDP Act.
How this comes up in interviews
A very common question is "We want the model to answer from our company documents. Would you fine-tune it?" A strong answer: no, use RAG for knowledge because it is fresh, cheap, citable and permission-aware; consider fine-tuning only for style or a narrow high-volume task, after prompting and evals show a gap. Mention that you would measure each option on a test set rather than guess.
I am building [describe the AI feature]. The model currently fails like this: [describe 3 real failures with examples].
My data: [documents, labelled examples, how often it changes, how many requests per day].
Recommend the order in which I should try prompting, a stronger model, RAG and fine-tuning. For each step say what evidence in my failures supports it, how I would measure success on a test set, and what would make me stop.What to expect: Check that the answer relies on your failure examples and not a generic list. Push back if it recommends fine-tuning for facts that change.
Try it: Diagnose three failures
For each situation, say whether the first fix should be prompting, RAG or fine-tuning, and why. (a) A college helpdesk bot gives last year's exam dates. (b) A bot's answers are correct but too long and chatty, and you want exactly two sentences. (c) You must label 200,000 support emails a month into 15 categories cheaply and fast, and you have 5,000 hand-labelled emails.
Key takeaways
- Prompting changes instructions, RAG changes the facts in front of the model, fine-tuning changes the model's behaviour.
- Try in order: better prompt, better model, RAG for knowledge, fine-tuning for behaviour.
- Use RAG for private, changing or citable information; fine-tuning is a poor way to teach facts.
- Fine-tuning suits consistent style, strict formats and narrow high-volume tasks, and needs good example data and a test set.
- Measure every option with evals; stop when quality is good enough.
- Mask personal data in training examples.
Check yourself
Finishing the quiz marks the lesson as done.
1. A company policy document changes every month and users need answers with citations. What is the best approach?
2. What is fine-tuning best at?
3. Which should you usually try first?
4. You pay for a 3,000-token instruction block on every one of 100,000 daily requests for a narrow task. What might help after prompting is already tuned?
5. Why is fine-tuning on raw customer chats risky?
Saved on this device. Create a free account to keep it on every device.
Practise this with Sam
A real mock interview, out loud, with an honest scorecard. Free to start.