Fix the corpus: a field guide to RAG data quality

At 11:47 p.m., I was sitting at my kitchen table with three browser tabs open: a customer policy in SharePoint, an older PDF on a shared drive, and a chatbot confidently quoting the wrong refund window. RAG was not the problem. The corpus was.

RAG, or retrieval augmented generation, works only when the system can find the right source, recognize its authority, and pass clean context to the model. That makes data quality the main engineering problem, not a cleanup task you knock out before the real work starts.

If your RAG application is returning wrong answers, inspect the documents before you touch the prompt or swap the model. Your corpus is the load-bearing wall.

Treat the corpus as a product surface

Here's the thing: teams almost always describe RAG as a search problem. That framing is too narrow. Search can only rank the material it receives, and a language model can only reason over the context it is given. If the corpus contains duplicate policies, expired contracts, broken tables, or missing ownership data, retrieval becomes a fast way to deliver confusion at scale.

The popular advice is to "just add RAG" when a model lacks company knowledge. That skips the hard part entirely. A model connected to an untidy document store is not informed, it is selectively misinformed, which is much harder to detect because the answer often sounds polished.

The original Retrieval-Augmented Generation paper by Patrick Lewis and colleagues, published in 2020, described a system that combines a language model with an external memory. The paper made the architecture clear. It did not remove the need to decide which documents belong in that memory, how they should be represented, or when they should be retired.

I see this in production constantly. A support assistant receives a current return policy from Zendesk, an old version from Google Drive, and a training document that says "ask a supervisor." The retriever finds all three. The model blends them into one answer. No prompt can reliably repair contradictory source material.

Build a corpus inventory before you build a retrieval pipeline. Record:

• The document source, owner, creation date, and last review date
• The business process or question the document supports
• The authority level and status, such as draft, active, archived, or superseded
• The access rules that apply to the original document
• The expected refresh frequency and deletion policy

This is not paperwork for its own sake. It tells you what the system is allowed to retrieve and what it must refuse to use.

A useful inventory also reveals where a business has no source of truth. If sales has one pricing sheet, finance has another, and the public website has a third, the RAG team cannot solve that disagreement with embeddings. Someone has to decide which document governs. That is not an engineering decision.

Find the bad data before you embed it

A vector database does not improve a document. Pinecone, pgvector, Weaviate, and similar tools store and retrieve representations of the material you provide. If you embed a scanned contract with no usable text layer, the index faithfully stores a poor representation. Expensive obedience.

Start with a source-by-source audit. I usually sample a fixed number of documents from every system rather than reviewing only the files people nominate as important. Nominees are almost always the cleanest files in the company. The problems are hiding in the exports nobody remembers creating.

For each sample, check whether:

• Text extraction preserves headings, tables, footnotes, and page order
• Dates, prices, product names, and identifiers survive conversion
• The document contains repeated headers, navigation menus, or legal boilerplate
• The source includes stale versions that conflict with active material
• A human can identify the owner and effective date without opening another system
• The access permissions in the index match the source permissions

A practical audit catches failures that embedding metrics will never reveal. On one common support corpus, a PDF export might convert "30 days" into "30 days from shipment" in one file and "30 days from delivery" in another. Both strings are semantically close. They are operationally different.

Use simple quality scores at the document level. Score extraction, freshness, authority, completeness, and access metadata from zero to two. A document scoring four out of ten should not enter production just because its vector is available.

The National Institute of Standards and Technology AI Risk Management Framework, published in 2023, places data quality, documentation, and traceability inside the management of AI risk. That is useful language for hiring managers because it connects corpus work to an existing control process rather than treating it as some obscure search concern.

A good audit produces a disposition for every source:

• Keep and index after cleanup
• Keep outside retrieval until an owner reviews it
• Merge with a newer source
• Archive and exclude from production search
• Delete when retention rules permit it

The mistake I keep watching teams make is treating every document as equally valid because it came from an internal system. Internal does not mean current. A file can be official, accessible, and wrong for the question being asked today.

Normalize the source before you chunk it

Chunking gets too much attention because it is easy to demonstrate. Teams adjust chunk size from 500 tokens to 700, run a few questions, and declare progress. But chunking a damaged source only creates smaller damaged sources.

Normalize first. Preserve the structure that carries meaning, and remove the structure that exists only for presentation. A PDF page number may help a human locate a passage, while a repeated "Confidential" banner adds nothing to retrieval. A table may look fine on screen but become a column soup after extraction.

The right normalization process depends on the source:

• Convert HTML into sections while removing menus, cookie notices, and repeated footers
• Extract PDF tables into structured rows when the question depends on numeric values
• Preserve headings and parent sections so a passage retains its subject
• Run OCR on scanned documents, then sample the output against the original pages
• Store the raw file and the cleaned representation so you can trace every answer

A real example: an employee travel policy. The visible PDF says, "Hotel reimbursement is limited to the nightly rate in the city schedule." A naive parser places the city schedule on a later page, stripped of the policy heading. Retrieval returns the rate table but not the sentence that defines when it applies. The model then answers a narrow question with a number and no condition. (If you've watched a demo fall apart on the third question, you already know how fast this happens.)

Normalization should also standardize dates, identifiers, and units where the business meaning permits it. Do not silently convert a contract date from local time to UTC if the agreement uses a local calendar date. Do not turn "$1,000 per quarter" into "$1,000 monthly" because a parser dropped the phrase after a line break.

Consider a file from a warehouse in Rotterdam. The source uses commas for decimal values and lists temperatures in Celsius. A general cleanup script that assumes American number formatting can change "4,5 kg" into "45 kg." The result may look clean in a text viewer and still be materially false.

Keep provenance attached to every cleaned section. At minimum, carry the source URI, document version, page or section location, effective date, owner, and access label. That provenance lets a reviewer inspect the answer instead of trusting a floating paragraph in a vector store.

A clean representation should remain reversible. If a reviewer asks why the system cited a sentence, you should be able to move from answer to chunk, from chunk to normalized document, and from normalized document to the original source. Without that trail, debugging is just guesswork.

Chunk for decisions, not for token counts

A chunk is not a storage unit. It is a unit of evidence. The best boundary is usually the smallest passage that can answer a realistic question without borrowing essential conditions from a neighboring passage.

This is where many RAG systems lose accuracy. A fixed 800-token splitter cuts directly through a heading, a numbered exception, or the final row of a table. The resulting chunks are similar enough to retrieve but incomplete enough to mislead.

Use the document structure to guide the split. A policy section, an API endpoint description, or a troubleshooting procedure often makes a better unit than an arbitrary character count. If a section is too large, split it at subheadings or complete steps, then attach the parent heading to each child chunk.

Useful metadata includes:

• Document type and business domain
• Product, region, customer tier, or department
• Effective date and expiration date
• Source system and version identifier
• Permissions and sensitivity level
• Parent heading, section path, and page number

Metadata supports filtering before semantic ranking. If someone asks about the Canadian refund policy, do not ask the embedding model to distinguish Canada from the United States if a region field can remove the irrelevant documents first.

Consider hybrid retrieval as well. Dense vectors help with meaning and paraphrase. Traditional keyword search handles exact identifiers such as "SLA-204," "VAT," or a specific product code. PostgreSQL with pgvector and full-text search can support both in one system for many teams, and the right choice depends on scale and operational needs, not what's trending in the Discord.

I once watched a demo fail on the third question because the first two used broad concepts. The third asked for an exact clause number. The vector search returned a related explanation but not the clause. That was not a model failure. It was a retrieval design that had no path for exact terms.

Test chunking against question types rather than abstract retrieval scores. A troubleshooting answer may need the error message, the affected service, and the resolution steps in one passage. A compliance question may need the rule plus its exception. A pricing question may need the value, currency, date, and customer segment together.

Once you frame it this way, chunk size becomes an outcome of the evidence you need, not a number you copy from a tutorial. Measure whether the retrieved passage contains enough information to support the answer without invention.

Build an Eval Loop that exposes corpus defects

You cannot improve data quality on a vague feeling that the chatbot "sounds better." Create a small Eval Loop before changing the pipeline. Twenty-five carefully chosen questions can reveal more than a hundred casual conversations if each question points to a known source and expected answer.

Include questions that test different failure modes:

• Direct lookup: "What is the current cancellation fee for the Oslo plan?"
• Conditional policy: "Can a customer cancel after shipment but before delivery?"
• Version conflict: "Which travel allowance applies after July 1, 2026?"
• Exact identifier: "What does error code SLA-204 mean?"
• No-answer behavior: "Does the policy cover personal device damage?"
• Access control: "Can a contractor retrieve the internal escalation guide?"

For every question, record the expected source, acceptable answer, required conditions, and a refusal or uncertainty rule. Then measure retrieval recall separately from answer quality. If the correct passage never appears in the top five results, prompt work is a distraction.

The BEIR benchmark, published in 2021, showed that retrieval performance varies substantially across information-retrieval tasks. That matters in applied systems because a retriever that performs well on general web questions may behave very differently on internal policies, support tickets, or technical logs.

Track at least these metrics:

• Recall at five: whether the correct source appears in the top five retrieved results
• Citation accuracy: whether the cited passage supports the claim made
• Answer completeness: whether conditions and exceptions are included
• Abstention accuracy: whether the system declines questions outside the corpus
• Freshness accuracy: whether the system uses the active version

Test the corpus after every ingestion change. A new parser can improve one document family and damage another. A metadata filter can increase precision while silently blocking material that certain users need.

A useful failure record should answer four questions:

• What did the user ask?
• What did retrieval return?
• What source should have been returned?
• Which pipeline change would prevent the failure from recurring?

Do not label every failure as hallucination. If retrieval returned an outdated contract, the model may have answered faithfully from bad evidence. That distinction matters because the fix belongs in source management or retrieval, not in model selection.

So the deeper lesson is simple: an eval loop is not just a quality gate for the model. It is a diagnostic tool for the corpus, parser, chunker, filters, and permissions that surround it.

Operate the corpus like a living system

A corpus decays. Policies change. Product names shift. People leave. SharePoint folders accumulate five versions of the same procedure and nobody deletes the old ones, because deletion feels risky. The result is a retrieval system that grows less trustworthy as it grows larger.

Give the corpus an operating rhythm. Every source should have an owner who can answer three questions: what belongs here, how often does it change, and how do we know the index reflects the source?

Set explicit rules for ingestion:

• Reject files with failed extraction or missing ownership metadata
• Quarantine documents with conflicting effective dates
• Re-index changed sources and remove deleted or expired versions
• Preserve an audit trail from source file to retrieved chunk
• Alert when ingestion volume, extraction failure, or duplicate rate changes sharply

Do not treat deletion as an edge case. If a document is removed from the source system, it should disappear from retrieval within a defined period. That period should be part of the product requirement, especially for legal, financial, health, or security content.

Permissions deserve their own test cases. A retrieval system must apply the source user's access rights before generation, not after the model has already seen the text. Filtering the final answer is too late. The sensitive passage has already entered the context window.

Here's the part most people miss: teams often spend money in the wrong order. They tune embedding models while leaving document ownership undefined. They buy a larger context window while allowing stale policies into the index. They add a reranker while ignoring failed OCR. More machinery cannot compensate for missing control.

Instrument the ingestion path before you optimize retrieval. Log source changes, parser failures, duplicate documents, empty chunks, permission mismatches, and unanswered evaluation questions. A small dashboard will tell you where the corpus is breaking.

Then make one owner responsible for the next correction. Not a committee. One person with authority to remove a source, change its metadata, or require a review.

Start with one source and prove the loop

The fastest way to improve RAG data quality is not to clean the entire company. Pick one source tied to a real user question and make the full path visible.

Take the support policies used by a Saturday-night refund assistant, the one that has to answer a customer's refund question correctly with no human in the loop. Export 50 documents from Zendesk and the shared drive. Mark every document as active, expired, draft, or unknown. Compare the text extraction with the originals. Build 25 questions around refund windows, exceptions, regions, and order status.

Then run the questions through the existing system and inspect each failure. You will likely find several different causes hiding under the same user complaint:

• The answer source was never ingested
• The right source was present but ranked too low
• An expired version outranked the active policy
• A table lost its row labels during parsing
• The answer needed an exact term that vector search missed
• The user lacked permission to see the source

Fix one failure class at a time. Re-run the eval loop after each change. Keep the original failures on record, because a system that passes only the latest test has not proved much.

For a hiring manager evaluating an AI product leader, this work is genuinely revealing. Strong candidates can explain the model choice. Better candidates can explain how they would identify a stale document, measure retrieval recall, preserve permissions, and assign ownership after launch. (And yes, that gap in what candidates can explain is still the clearest signal I've found.)

What I'm telling you is that corpus work is product work. It defines what users can ask, what the system can answer, how it handles uncertainty, and how quickly the team can recover when a policy changes.

Start tomorrow with one source, not the whole company. Export 50 documents, score their extraction, authority, freshness, completeness, and permissions, then run 25 questions against the current index. The results will tell you more than another afternoon of prompt tuning.

The model is not the product. The corpus, retrieval path, and evidence trail are the product. Fix those load-bearing parts, and the generation layer finally has something solid to work with.