How I build a self-certifying review loop for production AI

At 11:47 p.m., a support workflow in my kitchen was still running. A customer had asked for a refund, the agent had found the order, and the draft answer sounded perfectly reasonable. It was also wrong. The order was outside the refund window.

That is the production AI problem in miniature. An agent can produce fluent work while quietly failing the task. My approach is to build an AI review loop that checks evidence, repairs specific failures, and refuses to certify an answer when the available evidence is weak. That is what makes agentic AI safer to operate and gives self-correcting AI a real production shape.

The loop is not a second chatbot saying, "Looks good." It is an assembly line with inspection stations, measurable checks, and a reject bin. I have used this pattern while building multi-model AI products, and it has changed how I think about reliability: the agent is not finished when it writes an answer. It is finished when the system can explain why the answer passed.

Start with a review contract, not a reviewer prompt

The mistake I keep watching teams make is adding a reviewer model before defining what "correct" means. They send the original prompt and the generated answer to another model, then ask whether the response is good. That is not evaluation. It is a second opinion with expensive punctuation.

I start with a Review Contract. It describes the output, the evidence required, and the conditions that force a rejection. The contract is a testable interface between the worker agent and the review loop.

For a refund assistant, the contract might say:

• The answer must identify the order ID from the customer's message or account record.
• The refund decision must match the current policy version stored in Postgres.
• Any dollar amount must come from the order record, not from model memory.
• The answer must include a handoff when policy data is missing or contradictory.

That contract is intentionally boring. Boring is useful here. A reviewer can test "amount matches order record" with a database comparison. It cannot reliably test "sounds helpful" without drifting into taste.

I separate checks into four classes because each class fails differently:

• Grounding checks compare claims with retrieved records.
• Policy checks compare actions with a versioned rule set.
• Format checks validate JSON, fields, links, and required disclosures.
• Risk checks decide whether a human must approve the result.

A model can help write the test cases, but code should own checks that have a crisp answer. If the agent says a refund is $84.20 and Stripe says $48.20, no language model needs to debate the matter.

In one production design, I store each contract with a version number and a test fixture set. When policy changes, the old results remain traceable. That matters during an incident, because "the model changed" is not an explanation. You need to know which contract, policy record, prompt, and model produced the decision.

Give every claim an evidence trail

A self-certifying system needs more than a pass or fail value. It needs an evidence trail that another process can inspect. I call this the Evidence Ledger. It records what the agent claimed, which source supported the claim, and which check accepted or rejected it.

For a travel support workflow, the ledger might contain:

• Claim: "Your reservation can be canceled without a fee."
• Source: reservation record 78142 and policy version 2026-04.
• Check: cancellation window compared with the booking timestamp.
• Result: pass, with a timestamp and trace ID.

The ledger does not need to expose private customer data to every reviewer. It does need stable references, hashes, timestamps, and enough context to reproduce the decision. I usually keep the full payload behind access controls and pass a redacted evidence packet to the review model.

Here's the part most people miss. Teams often treat RAG as a search problem, then blame the reviewer when it surfaces the wrong document. The retriever can only be as good as the records it is allowed to see. A policy table in Postgres, indexed with pgvector or paired with ordinary filters, should not compete with a stale PDF copied into a general-purpose vector store.

The assembly line needs the right parts at the right station. I use deterministic filters before semantic retrieval when the data has a clear field, account ID, product region, policy version, or effective date. Semantic search is useful for finding relevant language, but it should not decide which customer record belongs to the request.

At my own platforms, I have seen how quickly a plausible answer can hide a retrieval error. A model may cite a policy that sounds relevant but expired six months ago. The review loop must inspect source freshness and authority, not just whether the answer contains a citation.

A practical evidence packet includes:

• The original user request with sensitive fields masked.
• The retrieved records and their source types.
• The model output in structured form.
• The exact contract and policy versions used.
• The checks that passed, failed, or were skipped.

That packet turns a vague complaint into a repairable defect. You can ask whether retrieval failed, extraction failed, policy matching failed, or the response writer ignored valid evidence. Those are different stations on the line, so they need different repairs.

Separate review into independent stations

One reviewer prompt is a bottleneck. It asks one model to judge facts, policy, tone, formatting, and risk in a single pass. The model then produces a confident paragraph, and the system treats that paragraph as truth. I would not approve that design for a payment workflow, a medical intake workflow, or a support agent that can change an account.

I split review into stations with narrow jobs. Each station returns structured output, not prose. A station can be a code check, a database query, a small classifier, or a model call. The right choice depends on how precise the test needs to be.

Consider a claims-processing agent in a Denver office at 8:15 a.m. It extracts a claim number, summarizes an incident, checks coverage, and drafts a response. The review line might run like this:

• Schema station: confirms every required field exists and uses the correct type.
• Evidence station: matches claim facts against the case record.
• Coverage station: applies the effective policy version to the incident date.
• Risk station: routes uncertain or high-value cases to a human queue.
• Writing station: checks that the final answer does not overstate the decision.

The stations do not all need the same model. GPT-4.1 can handle a long structured context, Claude can be useful for document comparison, and a smaller model can classify a narrow risk category at lower cost. Whisper can transcribe a phone recording, but a transcription is not proof that an extracted claim detail is correct. The station after transcription still needs to compare the detail with the source audio or case record.

Use separate trace spans for every station. OpenTelemetry can carry the trace ID through the worker, reviewer, database calls, and human handoff. LangSmith, Braintrust, or a plain Postgres table can store evaluation results. The tool matters less than the ability to answer four questions: what ran, what did it see, what did it decide, and what happened next?

A station should have one owner and one failure meaning. If a single reviewer returns "mostly acceptable," you lose that clarity. "Evidence check failed because source 2 was expired" is actionable. "Quality score 0.71" is not.

Make correction targeted and bounded

Self-correcting AI is often described as an agent reviewing its own work until it feels satisfied. That sounds clever and behaves badly. Without limits, the system can loop, rewrite a correct answer into a wrong one, or spend $40 trying to improve a sentence that already passed.

Correction should act on a named failure. If the schema station fails, repair the schema. If the evidence station fails, retrieve again with the missing entity or ask for the missing record. If policy matching fails, stop and request a human decision. Do not send the entire output back with the instruction to "try harder."

I model each repair as a typed action:

• `RETRIEVE_AGAIN` with a missing field and a new filter.
• `REWRITE_FIELD` with the failed claim and approved evidence.
• `ASK_USER` with the exact information required.
• `ESCALATE` with the reason and risk level.
• `ABORT` when the system cannot establish a safe basis for action.

Each action has a retry limit. My default is one targeted repair for low-risk work and zero automatic repairs for actions involving money movement, account closure, or irreversible changes. Not because models never improve on a second attempt, but because the blast radius rises when the system keeps acting without stronger evidence.

Here is a concrete case. A billing agent drafts, "We have issued your $120 refund." The ledger shows only that a refund was requested. The evidence station rejects the claim. The repair should change "issued" to "requested" or route the case to a payment-status check. It should not ask a general reviewer to make the sentence sound more cautious.

Track correction yield separately from pass rate. If 70 percent of failures are repaired on one attempt, that tells you something useful. If the same error consumes four retries, the prompt is not your only problem. The contract, retrieval filters, or source data may be wrong.

And instrument the retry path before tuning the prompt. Teams optimize token cost to the third decimal while ignoring a retry storm caused by one malformed field. I have made that mistake myself (the bill is rarely the first symptom, latency and queue growth usually show up earlier). By the time costs spike, the damage is already compounding.

Certify the result with a decision policy

A review loop becomes self-certifying when it can make a bounded decision about its own output. Certification does not mean the model claims it is right. It means the system has satisfied a defined set of checks and can show the evidence behind that decision.

I use a decision policy with explicit outcomes:

• CERTIFIED: all required checks passed, evidence is current, and risk is within the automatic-action limit.
• REPAIRABLE: one or more checks failed, but a named repair action is safe and available.
• REVIEW_REQUIRED: the result may be valid, but a human must approve it because of risk, ambiguity, or missing authority.
• REJECTED: the system cannot establish a trustworthy basis for the output.

The policy should be stricter than the writer. If an agent creates a product description, a minor style problem may be repairable. If it changes a customer's subscription, missing evidence should stop the action.

So the deeper lesson is that "human in the loop" is not a complete safety design. A human who receives 2,000 vague alerts per day is not a meaningful control. The queue needs reason codes, evidence links, suggested next actions, and a confidence signal that reflects the checks, not the model's tone.

For a support operation, I would route a case to a person when:

• The customer record and policy record disagree.
• The action changes money, access, or legal status.
• The agent has used its maximum repair count.
• A required source is unavailable or older than the contract allows.

The reviewer should be able to approve, edit, reject, or send the case back to a named station. Those actions become new labeled examples. Over time, you learn which contracts generate noise and which failures are worth automating.

A certification record should contain the final output hash, contract version, model identifiers, source references, station results, repair count, and human decision when one exists. That record is the production equivalent of a test report. Keep it long enough to investigate incidents and sample quality over time.

Operate the loop as a product system

A review loop is not finished when the pipeline runs in staging. Production changes the input distribution, the source records, the costs, and the users' patience. You need a release process for the loop itself.

Before changing a model or prompt, run a fixed evaluation set containing ordinary cases, boundary cases, and known failures. Include a refund request made at 11:59 p.m. on the final eligible day. Include a customer with two orders and one similar product name. Include a policy updated between the original request and the review. These are the cases that expose weak joins between stations.

Measure more than answer accuracy. I watch:

• Certification rate by workflow and contract version.
• Repair rate and repair yield.
• Human escalation rate and queue age.
• Cost per certified result, including retries.
• Time from request to certification.
• False certification rate from sampled human audits.

The false certification rate matters most. A system that rejects too much is expensive and frustrating. A system that certifies wrong work is dangerous. You need both numbers to understand the tradeoff.

Use canary releases for model changes. Send a small share of traffic through the new version, compare station failures, and keep the old path available. Store replayable fixtures so a production incident can run against a candidate repair without touching a live customer.

What I'm telling you is that the strongest teams I've worked with did not inspect only the final product. They inspected the process that made it. AI products need the same discipline, except the input varies more and the failure can look polished.

With Zentoko, a multi-brand AI marketing platform, the same principle applies to generated campaign work: a fluent draft is not enough. The system needs to check brand rules, source claims, required fields, and approval status before the draft moves forward. With Ziggyloo, a children's AI tutor, the review problem shifts toward age-appropriate guidance, grounded lesson context, and safe escalation. The domain changes. The assembly-line logic does not.

Start small. Pick one workflow where a wrong answer has a clear cost. Write the Review Contract, create 30 real or carefully anonymized fixtures, and build four stations around the failures you can already name. Then run the loop for a week before adding more autonomy.

FAQ

What is an AI review loop?

An AI review loop is a process that checks an AI output against defined requirements, repairs specific failures, and either certifies, escalates, or rejects the result. In production, it should record evidence, model versions, retry actions, and final decisions rather than returning only a score.

How is self-correcting AI different from ordinary retry logic?

Ordinary retry logic runs the same operation again, often after a timeout or technical error. Self-correcting AI identifies the failed requirement, changes the relevant action, and limits the number of repairs so the system does not loop without stronger evidence.

Should a second language model review the first one?

Sometimes, but not by default. Use code and database checks for exact requirements, then use a second model for ambiguous work such as comparing long documents or judging whether a response overstates the evidence. A second model cannot compensate for missing or stale source data.

How many review steps should an agentic AI workflow have?

Use as many stations as the workflow needs to separate failure types, not a fixed number. A refund workflow may need schema, record, policy, risk, and writing checks. Start with the smallest set that catches known failures, then add stations when production evidence shows a gap.

What should I measure first?

Measure certification rate, false certification rate, repair yield, human escalation rate, latency, and total cost per certified result. The most important early question is whether the system is confidently approving wrong work, so sample certified outputs with human reviewers.

The practical next step is simple: choose one production workflow this week and write down what would make an output certifiable. Turn those conditions into checks, attach evidence to every claim, and give each failure one bounded repair path.

The model is only one station on the line. The product is the line that can prove when the work is good enough to move forward.