RAG is a data-quality problem, not a search problem

A support engineer opens a refund policy PDF at 9:14 on a Saturday morning. A customer wants to know whether a hotel booking made with points can be refunded. The answer is somewhere on page 12, but the file contains three policy versions, a scanned table, and a footnote that expired last June.

That is a RAG problem, and it is not mainly a search problem. Retrieval augmented generation works only when the system can access current, complete, well-structured, well-owned information. Vector search can rank the wrong material perfectly.

I've built live multi-model AI products, and I keep seeing the same pattern: teams tune chunk size, embeddings, and reranking while leaving the source files untouched. RAG quality usually improves more from fixing data quality than from changing the search stack.

The retrieved answer cannot outrun the source data

RAG adds external information to a model's prompt at response time. A typical flow looks simple:

• A user asks a question.
• The system searches a document store.
• Relevant passages enter the model context.
• The model writes an answer grounded in those passages.

The model does not repair a broken knowledge base as part of that process. If the source contains an obsolete price, a duplicated procedure, or a missing exception, retrieval has fewer good choices. The model may produce fluent prose, but fluent prose is not evidence that the answer is correct.

The original RAG paper by Patrick Lewis and colleagues, published in 2020, described a model combining parametric memory with non-parametric memory retrieved from an external source. That external source is the part many teams treat as a neutral warehouse. It is not neutral. It is a product dependency with owners, version history, access rules, and failure modes.

Consider a hotel support assistant working from these files:

• `refund_policy_2022.pdf`
• `refund_policy_final.pdf`
• `refund_policy_final_v2.pdf`
• A 2024 operations memo in a shared drive
• A help center page updated in January 2026

All five documents may discuss refunds. Only one may govern the answer for a specific booking type. A vector database cannot infer authority from a filename called `final`. The model cannot safely infer it from confident wording, either.

Here's the part most people miss: relevance and truth are separate properties. A passage can score highly on semantic similarity and still be wrong for the customer's date, region, account tier, or transaction type.

What does RAG actually retrieve?

RAG retrieves text units, not business truth. A chunk is a passage created by splitting a document, and the split can strip out the condition that gives the sentence its meaning.

For example, this sentence may appear in one chunk:

> Cancellations are refundable within 24 hours.

The next chunk may begin with:

> This applies only to reservations made directly through the mobile application and excludes promotional rates.

A search system that returns only the first chunk has found related text but not a complete rule. The failure happened during document preparation, before embeddings or vector search entered the picture.

Data quality has several dimensions

Teams often describe data quality as a vague cleanup task. That framing makes it hard to assign ownership and even harder to measure. For a RAG system, I use a more practical definition: data is useful when it is accurate, current, complete, attributable, and shaped for the question the product must answer.

Those dimensions create different failure modes.

• Accuracy asks whether the claim is factually correct.
• Freshness asks whether the claim still applies today.
• Completeness asks whether related conditions and exceptions are present.
• Authority asks whether the source is approved for this decision.
• Structure asks whether tables, headings, dates, and relationships survived ingestion.

A benefits assistant may retrieve an accurate 2023 health plan document and still give the wrong answer to an employee checking their coverage in March 2026. A sales assistant may retrieve the correct discount percentage but miss the rule that applies only to annual contracts. A manufacturing assistant may find a calibration procedure while ignoring the equipment model called out in the document header.

The conventional advice is to improve retrieval with better embeddings. Sometimes that is right. But teams often use embeddings as a hiding place for unresolved ownership questions. No embedding model can decide whether a policy approved by finance should outrank a draft written by marketing.

Imagine a financial services product pulling from a Postgres database, a SharePoint archive, and a customer-facing FAQ. The database says a transfer limit is $10,000. The FAQ says $5,000. A compliance memo says the limit depends on account age. The issue is not that the vector index has poor semantic understanding. The issue is that the company has not defined which record governs which customer.

So the deeper lesson is simple: before you ask whether the system retrieved the right passage, define what "right" means. That definition belongs to the product and data owners, not only to the machine learning team.

Chunking can destroy the evidence

Chunking is the process of splitting source material into pieces that can be embedded and retrieved. It looks like an implementation detail until a legal exception lands in a different chunk from the rule it modifies.

A fixed 500-token split is easy to implement. It is also blind to document structure. It may divide a heading from its table, a question from its answer, or a safety warning from the operating step it qualifies. Recursive character splitters, used in many frameworks, improve the situation by respecting separators, but they still do not understand every business document.

A better ingestion process starts with the shape of the source:

• Preserve document titles, section headings, page numbers, and effective dates.
• Keep a table with its column headers and nearby explanatory notes.
• Attach the parent heading to each child chunk.
• Keep a policy rule and its exceptions together when the question depends on both.
• Store the source URL, version identifier, and access permissions with every chunk.

Here is a concrete example. A hospital operations team loads a 70-page PDF about refrigerator temperature monitoring. The target question is: "What should staff do when the reading reaches 9 degrees Celsius?" The answer is split across a table, a warning box, and an escalation section. A generic splitter returns the table row but not the requirement to quarantine the contents and notify the supervisor.

The model may then answer with a plausible monitoring instruction instead of an incident response. That is not hallucination in the narrow sense. The system retrieved incomplete evidence and asked the model to fill the gap. If you've watched a demo fall apart on the third question, you already know how quickly this erodes user trust.

Parent-child retrieval can help. The system searches smaller child chunks for precision, then sends the parent section or surrounding window to the model. A neighboring-window strategy can do something similar. Neither approach excuses poor source preparation. More context is useful only when the added context is authoritative and relevant.

Test chunking with adversarial documents, not just clean Markdown. Use scanned PDFs, Excel exports, slide decks, HTML tables, and pages with repeated headers. In my experience, the ugly document is where the product's real quality shows up. The polished demo file is mostly theater.

Metadata is part of retrieval, not decoration

Vector similarity answers a narrow question: which text is semantically close to the query? Production products usually need more than that. They need to restrict the search by time, customer, geography, product, permission, or workflow state.

Suppose an employee asks, "How much parental leave do I have?" A semantic search may surface policies from the United States, Canada, and the United Kingdom. The most similar wording is not necessarily the applicable policy. The system needs metadata such as country, employment type, policy status, and effective date before it ranks passages.

Useful metadata often includes:

• `effective_from` and `effective_to`
• Product family and version
• Region, department, or customer segment
• Source owner and approval state
• Security classification and access scope
• Document type, such as policy, contract, guide, or incident record

This is where relational systems remain useful. Postgres with pgvector can store embeddings beside ordinary columns, letting a query combine vector similarity with SQL filters. Pinecone, Weaviate, and other vector databases support metadata filtering in different ways. The tool matters less than whether the product records the fields it needs.

A query for an aircraft maintenance procedure should not search every procedure ever written. It should filter for the aircraft model, component, maintenance manual revision, and active status before the system compares language similarity. A query for a customer's contract should filter by account identifier and contract date before it searches clauses.

The mistake I keep watching teams make is treating metadata as optional because the initial dataset is small. Small datasets become large faster than the code gets redesigned. Once a team has indexed 100,000 documents without ownership or effective dates, cleanup becomes a migration project with production risk.

Build the metadata contract before ingestion. For every source, write down who owns it, how freshness is measured, which fields are mandatory, and what happens when a document expires. If a source cannot answer those questions, it is not ready to become evidence for an automated answer.

Authority and freshness need explicit controls

A RAG system should know more than what a document says. It should know whether the document is allowed to answer the question.

That requires an authority model. A public help page may be suitable for a general product question but not for a contractual dispute. An internal runbook may be useful for an engineer but inappropriate for a customer. A draft policy may be searchable for review while remaining unavailable to the production assistant.

One practical approach is to assign each source an authority class and update path:

• Approved operational policy
• Customer-specific agreement
• Internal procedure
• Reference material
• Draft or superseded content

Then make retrieval respect those classes. A customer contract can outrank a general FAQ for that customer's account. An active product manual can outrank an archived manual for a current device. A draft can be excluded unless the user has a review role.

Freshness also needs a policy. "Updated recently" is not the same as "valid." A document may carry a new upload date because someone changed its formatting. Another document may be old but still active because the policy owner renewed it explicitly.

For each source, record at least:

• The date the content became effective
• The date it expires or requires review
• The person or team responsible for approval
• The event that triggers re-indexing
• The action taken when review is overdue

Consider a retailer's return policy. On November 1, the business changes the holiday return window. If the content pipeline runs only once a month, the assistant may answer incorrectly for weeks. If the source system emits an event when the policy changes, the index can update within minutes. The important design choice is not a faster embedding model. It is connecting policy change to retrieval refresh.

At Ziggyloo and Zentoko, I pay close attention to this boundary because a multi-model system can make source confusion harder to spot. One model may produce a polished answer from an old passage, while another produces a cautious answer from a current one. Model agreement does not prove source validity. Two models can repeat the same bad evidence.

Measure the whole RAG path

RAG evaluation should separate retrieval quality from answer quality. If the final response is wrong, you need to know whether the correct document was absent, buried, incorrectly filtered, or ignored by the model.

A useful evaluation set contains real questions and expected evidence. Pull questions from support tickets, sales calls, operations incidents, and compliance reviews. Annotate the source that should answer each question, the conditions that must appear, and the situations where the system should refuse.

Track measures such as:

• Retrieval recall at top 5: whether the required evidence appears in the first five results
• Retrieval precision: how much of the returned material is useful
• Citation correctness: whether the cited passage supports the claim
• Answer completeness: whether conditions and exceptions survived
• Abstention quality: whether the system declines when evidence is missing
• Freshness compliance: whether expired content is excluded

These measures expose different problems. Low recall may point to poor chunking, missing synonyms, weak metadata, or an absent source. Good recall with poor citation correctness may indicate the model is stitching together unrelated passages. Good citations with incomplete answers may mean the chunks omit tables or exceptions.

Use named test cases. "Can I return a laptop?" is weak. "Can a New York customer return a refurbished laptop purchased through the education store on December 20, 2025?" is useful because it tests geography, product state, channel, and date.

Log the retrieved chunk IDs, source versions, filters, scores, model name, prompt version, and final citations. Without that record, a production failure becomes a screenshot and a guess. I've spent enough time debugging distributed systems to know that memory is not an observability strategy.

The eval loop should run whenever ingestion logic, metadata filters, prompts, models, or source documents change. A document cleanup can improve one question and break another. Treat the corpus as changing software, not as a static upload.

Build the data-quality operating system

Once RAG supports real users, data quality becomes an operating process. It cannot stay a one-time preprocessing script owned by whoever built the prototype.

Start with a source inventory. For every collection, record its owner, format, update frequency, authority level, access policy, and known defects. Then choose a small set of high-value user questions and trace each answer back to its source. This is faster than trying to clean every document at once.

A practical rollout can follow this sequence:

• Select one business workflow, such as refund support or device troubleshooting.
• Identify the approved sources for that workflow.
• Remove or quarantine duplicates and superseded versions.
• Define mandatory metadata and validation checks.
• Build representative questions from actual user interactions.
• Measure retrieval and answer behavior before changing the model.
• Add an owner review when source freshness or authority fails.

Automate the cheap checks. Reject empty files, unreadable scans, missing titles, impossible dates, and duplicate hashes. Flag documents with no owner or no effective date. Compare new content against the prior version so a small formatting change does not trigger needless review.

Then add human review where the cost of a wrong answer is high. A restaurant recommendation can tolerate some uncertainty, a medication instruction or wire transfer procedure cannot. The product should make this risk visible instead of hiding it behind a friendly answer.

Here's the thing about the default startup advice to ship the MVP fast: for RAG, the failure mode is often a minimum viable corpus. A demo that answers familiar questions from handpicked documents but has no process for change, conflict, or missing evidence. That shortcut moves the hard work into production, where every mistake has a user attached to it.

Use the simplest retrieval stack that gives you control. OpenAI embeddings, Cohere rerankers, Elasticsearch, Postgres with pgvector, and managed vector databases can all be reasonable choices. None removes the need for source ownership, versioning, permissions, and evaluation.

What I'm telling you is this: RAG is a data product with a language model at the edge. The search layer matters, but it is not the foundation. The foundation is trustworthy evidence.

FAQ

Is RAG a search problem or a data-quality problem?

RAG is both, but data quality is usually the larger source of failure. A search system cannot reliably retrieve current, complete, authoritative answers when the source data is stale, duplicated, incomplete, or poorly labeled. Better ranking helps only after the corpus gives the system good evidence to rank.

How do I improve RAG retrieval quality?

Start by removing duplicates, assigning document owners, recording effective dates, preserving related conditions, and adding metadata filters. Then measure retrieval recall and answer accuracy before changing embeddings or ranking models. Test with real questions that include dates, regions, products, account types, and exceptions.

Does better vector search fix bad documents?

No. Better vector search can rank relevant passages more effectively, but it cannot determine whether a passage is current, complete, or approved. Bad documents remain bad evidence, even when a sophisticated embedding model finds them quickly.

What should a RAG evaluation measure?

Measure whether the correct source appears in the retrieved results, whether the answer is supported by that source, whether important conditions are preserved, and whether the system refuses questions that lack sufficient evidence. Store retrieval details so you can separate an indexing failure from a generation failure.

Should I use a vector database for every RAG system?

No. A relational database with pgvector, Elasticsearch, a managed vector database, or a hybrid setup may all work. Choose based on filtering, permissions, update patterns, scale, and operational skill. The storage choice will not compensate for missing source ownership or weak evaluation.

Take one production question your assistant currently answers. Trace it from user query to retrieved chunks, source version, and final claim. If you cannot identify why that source was allowed to answer, fix that path before tuning vector search.

The model is not the product. The evidence pipeline around the model is the product. Ship the data-quality controls first, and retrieval has a chance to become something users can actually trust.