When to fine-tune, when to use RAG, and when to write a better prompt

At 10:47 p.m., a support engineer is staring at a refund answer on a second monitor while a customer waits in a hotel lobby. The answer sounds polished, but it cites a policy that expired in March. The right call in the fine-tuning vs RAG debate is usually simpler than teams make it: write a better prompt first, use RAG for changing or private facts, and fine-tune only when the model needs to behave differently every time.

That's the AI decision framework I use when I build production systems. The model is one station on an assembly line. Before replacing it, check whether the input is wrong, the work instructions are vague, or the station truly needs new training.

Start with the failure, not the technology

When an LLM produces a bad answer, teams often jump straight to architecture. Vector databases, adapters, context windows, training runs - all of it on the table before anyone writes down what actually failed. That is how a prompt problem becomes a six-week project.

I start with the output. Was the answer factually wrong? Was the tone wrong? Did the system ignore a required format? Did it lack a document that existed somewhere else? Those are different defects, even when the customer sees the same thing: "the bot got it wrong."

A useful diagnosis has four buckets:

• Instruction failure: the model had the right information but did not follow the task.
• Knowledge failure: the model needed current, private, or specialized information.
• Behavior failure: the model repeatedly missed a style, format, or decision pattern.
• Product failure: the surrounding workflow supplied bad inputs or gave the model too much authority.

The conventional advice is to add retrieval whenever an answer is wrong. That is not always sound. If a billing assistant already has the correct invoice in its context but still invents a discount, another database will not fix the instruction hierarchy. You need a tighter prompt, a clearer output schema, or a validation step.

In my own multi-model products, I've found that the first useful artifact is a small failure set. Pull 20 to 50 real examples, mark the expected answer, the source of truth, and the failure type. You do not need a grand research program. You need enough examples to see whether the defect repeats.

The assembly line analogy helps here. Do not replace a machine because one package had the wrong label. Find out whether the label file, operator instruction, or machine calibration caused the miss. Your choice among prompting, RAG, and fine-tuning should follow that diagnosis.

Write a better prompt when the model already knows enough

Use prompt engineering when the model has the needed facts and the problem is task execution. This is common with classification, extraction, rewriting, routing, and structured responses. GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro can all handle these tasks when the request is specific and the surrounding context is clean.

A weak prompt says, "Review this support ticket and respond helpfully." A production prompt says what counts as a refund request, which policy takes priority, what the model must never promise, and what JSON fields the application expects. It also gives examples of borderline cases. Completely different instructions, even if the goal sounds the same.

Improve the prompt when you see failures such as:

• The model follows one instruction but misses a later constraint.
• The answer changes because the requested format is vague.
• The model receives relevant facts but does not distinguish evidence from a user claim.
• The same input produces different decisions because the boundary case is undefined.

A good prompt is not a wall of prose. It is an operating procedure. State the task, define the inputs, name the decision rules, provide a small example set, and specify what to do when evidence is missing. Structured output through JSON Schema or a tool definition can remove a large class of formatting errors.

Try a prompt-only test before adding infrastructure. Run the same evaluation set against the current prompt and two revised versions. Track accuracy, refusal rate, format validity, latency, and token cost. If the revised prompt solves the failure without making the request much longer, stop there.

There is a practical limit. Prompts do not create knowledge. They cannot make a model know an internal pricing table from last night or reliably follow a specialized workflow it has never seen. But teams often reach that limit too early because their prompts describe goals instead of decisions.

What I'm telling you is simple: if the model has the evidence and still behaves badly, tune the instructions before you tune the model.

Use RAG for facts that live outside the model

Retrieval-augmented generation, or RAG, is the right fit when the answer depends on information that is private, recent, large, or frequently updated. The model does not need to memorize the material. It needs access to the right pieces at request time.

Consider a hotel operations assistant used by a property in Singapore. The answer to "Can I check in at 11 a.m.?" may depend on the reservation, the property's current occupancy, a loyalty tier, and a local operating policy. Fine-tuning cannot safely encode those facts because they change. RAG can retrieve the reservation record and current policy, then pass both to the LLM.

RAG is a data pipeline, not a magic search box. A basic implementation often includes:

• Source selection: identify which systems are allowed to answer the question.
• Ingestion: pull documents, records, and policy updates on a defined schedule.
• Chunking: split material into units that preserve enough meaning to stand alone.
• Retrieval and ranking: find relevant passages, then remove weak matches.
• Grounded generation: instruct the model to answer from supplied evidence.
• Citation or trace storage: retain the sources used for the response.

Postgres with pgvector can be enough for an early system. Pinecone, Weaviate, or another vector service may make sense at larger scale, but the database choice will not rescue poor source data. A 2024 retrieval study from the University of Waterloo and collaborators found that retrieval quality depends heavily on chunking and query formulation, which matches what I see in production: the right document is useless if the retriever selects the wrong paragraph.

Use RAG when you need:

• A current product catalog or policy manual.
• Customer-specific account or transaction data.
• Internal documents that should not be baked into model weights.
• Traceable answers where a reviewer can inspect the source.

But do not use RAG to compensate for unclear behavior rules. Retrieved text can tell a support bot what the cancellation policy says. It cannot, by itself, tell the bot when to escalate an angry customer or how to format the case record.

Fine-tune when behavior must become consistent

Fine-tuning changes model weights so the model becomes more likely to produce a desired response pattern. It is useful when you have many high-quality examples and the behavior repeats across requests. It is not a convenient way to upload a handbook.

Suppose a legal operations team has 8,000 reviewed intake records. Each record maps a messy email to a strict internal category, confidence score, and escalation route. The categories are stable. The output format is fixed. A fine-tuned model may learn that mapping more reliably and with fewer prompt tokens than a general model receiving long examples on every request.

Fine-tuning is a candidate when:

• The desired behavior is stable over time.
• You have a representative, labeled dataset.
• Prompting with examples has reached a measurable ceiling.
• Lower latency or lower input-token cost matters.
• The model must follow a house style across many workflows.

OpenAI's fine-tuning guidance has consistently stressed dataset quality and evaluation rather than raw example count. That matters. One hundred clean examples covering real edge cases can teach more than 10,000 repetitive records. A fine-tuning run can also preserve bad habits, amplify label mistakes, or make the model confidently wrong in cases absent from the training set. If you've ever watched a confident model fail the exact case you thought you'd covered, you already know how much that stings.

Use retrieval for facts and fine-tuning for habits. That distinction is not perfect, but it is a useful operating rule. A fine-tuned support model can learn to ask for an order number before discussing an account. RAG should supply the current order details and refund terms.

Parameter-efficient methods such as LoRA and adapters can reduce the cost of experimentation because they update a smaller set of parameters instead of the full model. They still require the same discipline: a held-out test set, versioned data, rollback support, and monitoring after release.

Fine-tuning also creates maintenance work. When the policy changes, you must decide whether to retrain, add retrieval, or change the prompt. If the answer changes every week, training the model every week is usually the wrong assembly-line design.

Use the AI decision framework before you build

I use a decision table before selecting a tool. The table forces the team to describe the problem in operational terms rather than picking the most fashionable technique.

Then score the proposed approach against four production constraints:

• Freshness: how quickly must new facts become available?
• Control: how much authority can the model have?
• Cost: what are the input tokens, retrieval calls, training runs, and retries?
• Evidence: can you show why the response was produced?

A prompt-only approach is often best for a product description classifier. RAG fits a customer service assistant that answers from current account data. Fine-tuning fits a document router that must apply a stable internal taxonomy to thousands of similar requests.

Here is a concrete case. At 8:15 on a Monday morning, a finance assistant receives an invoice from a new supplier. If the problem is that it cannot identify the supplier's current payment terms, retrieve the approved vendor record. If it knows the terms but returns invalid JSON, repair the prompt and schema. If it repeatedly assigns the wrong internal category after thousands of labeled examples, test fine-tuning.

The mistake I keep watching teams make is combining all three before establishing a baseline. That creates a system nobody can debug. Build the smallest version that can prove the diagnosis, then add one capability at a time.

Measure the assembly line in production

The right architecture is not the one that wins a demo. It is the one that gives you a clear answer when performance drops on a Saturday night.

Create an Eval Loop before changing the model. Store the input, retrieved documents, prompt version, model version, output, reviewer label, latency, and cost. Then divide results by failure type. A single accuracy score hides too much. A refund bot can score 95 percent overall while failing every urgent escalation, which is exactly the part that creates operational risk.

Track at least these measures:

• Grounded accuracy: does the answer match the approved source?
• Instruction compliance: does it follow the requested workflow and format?
• Retrieval recall: did the system bring back the evidence needed to answer?
• Cost per successful task: what did the complete request cost, including retries?
• Time to update: how long does it take for a policy change to reach production answers?

Run offline tests with a fixed dataset before every prompt, retrieval, or model change. Then sample live traffic, with privacy controls, to catch failures your test set missed. For a high-risk workflow, add a human approval gate instead of giving the LLM unrestricted authority.

And budget for the whole system. A longer prompt raises input cost. RAG adds embedding, storage, retrieval, and reranking work. Fine-tuning adds dataset preparation, training, evaluation, hosting, and refresh costs. The cheapest model call is not the cheapest successful transaction.

So the deeper lesson is this: handoffs cause more trouble than diagrams suggest. Give every component an owner. Version the source documents. Keep a rollback path. Log what the model saw, not just what it said, with sensitive data handled properly.

Start with a 50-case test set this week. Label each case prompt, knowledge, behavior, or product failure. Try the least expensive fix first, then rerun the same cases. The assembly line gets better when each repair teaches you which station actually failed.

FAQ

Is RAG better than fine-tuning for most LLM applications?

RAG is usually better for current, private, or user-specific information because the system can update the source without retraining the model. Fine-tuning is better for stable behavior, repeated formatting, or a specialized response pattern supported by high-quality examples.

Can prompting replace fine-tuning?

Prompting can replace fine-tuning when the model already understands the task and the required behavior can be stated clearly. If many tested prompt revisions still miss a stable pattern, fine-tuning becomes worth testing.

Does RAG teach an LLM new knowledge?

RAG gives the LLM relevant information at request time. It does not change the model's weights, so the model may still misunderstand the retrieved material unless the prompt, source quality, and evaluation process are sound.

How much data is needed for fine-tuning?

There is no universal threshold. A small, clean dataset that covers real edge cases can beat a large repetitive dataset, but you need enough held-out examples to measure generalization and detect regressions.

What should I try first when an AI feature fails?

Classify the failure before choosing a solution. Test a clearer prompt and structured output first, then add RAG for missing or changing facts, and consider fine-tuning only after you have stable labeled examples and a measured behavior gap.

Take the 50 worst real outputs from your system and classify each one before touching the architecture. The model is rarely the whole product. The assembly line around it decides whether the product works.