The eval loop is the product: measuring whether your AI is actually right

At 10:47 p.m. on a Saturday, a support bot answers a refund question. The customer has a valid return, but the bot cites an old policy and rejects it. The model sounded confident. The product was wrong.

That's the real job of AI evals: measuring whether your system is right on the questions that actually matter, not whether a demo sounds intelligent. In production AI, the Eval Loop is not a QA step you bolt on after the feature ships. It's the product surface that tells you what to ship, what to block, and what to fix next.

I've shipped software programs at Verizon, HP, and InterContinental Hotels, then built two live multi-model AI platforms from scratch. The pattern is hard to miss once you've seen it. A model can produce polished, confident-sounding language while failing the user in a way your dashboard never catches.

Why model quality is not enough

The industry repeats a seductive idea: pick the best model and your quality problem is mostly solved. That advice breaks the moment the model meets your actual workflow.

A model doesn't answer an abstract question called "Is this good?" It answers a specific request with a specific system prompt, retrieved context, tool result, memory state, temperature setting, and output format. Change any one of those variables and the result can change dramatically. Your LLM evaluation has to measure the whole path, not just the final output.

Consider a hotel assistant answering this request in a Chicago lobby: "Can I check in early, and will my room be ready when I arrive at noon?" A useful answer has to separate two distinct facts. Early check-in depends on room availability. A noon arrival is not a guarantee the room will be ready. If the assistant invents a fee, promises access, or confuses a reservation note with a hard policy, fluent wording doesn't save you.

The same problem shows up in a children's tutoring flow. A student asks for help with a fraction problem. The system gives the correct answer but skips the reasoning step the lesson is designed to teach. A basic exact-match score may mark that response as successful. A teacher would not. (And if you've ever watched a parent complain that the tutor "just gave my kid the answer," you already know how fast that erodes trust.)

Stanford's HELM project, published in 2022, made this problem visible by evaluating language models across multiple scenarios and metrics instead of relying on one general score. Accuracy was only one dimension. Robustness, fairness, toxicity, and efficiency all shifted the practical result.

So the deeper lesson is simple: evaluate the behavior you sell. A model leaderboard can inform a decision. It cannot make the decision for you.

Build an Eval Loop before you add more features

An Eval Loop is a repeatable path from real user request to measured result to product change. It should run whenever you change a prompt, model, retriever, tool, policy, or user interface. If a change can alter an answer, it belongs inside the loop.

The loop needs five working parts:

• A dataset of realistic requests, including failures and edge cases.
• A reference answer, acceptable answer range, or explicit grading rule.
• A test runner that sends the same cases through the current system.
• A grader that scores the output using rules, a model, or human review.
• A record of the change, score, failure type, and decision.

A small, trusted dataset beats a giant pile of synthetic prompts every time. When I start an evaluation set, I collect cases from support tickets, product reviews, internal testers, and observed failure logs. I label what went wrong in plain language: wrong source, unsupported claim, missed instruction, unsafe action, poor tone, failure to use a tool.

For a refund assistant, ten useful cases might include an expired return window, a damaged item, a gift purchase, a missing receipt, a partial refund, and a request made on a holiday. These cases are not glamorous. They are where the product earns or loses trust.

Run the set on every meaningful change. Store the prompt version, model name, retrieval settings, tool calls, latency, token count, and grader result. Without that record, you cannot tell whether a quality shift came from the new model or from a quiet change in retrieved context.

The mistake I keep watching teams make is building a feature backlog before building this measurement path. They add memory, voice, agents, and new tools while still judging quality by clicking through the happy path. That is how a demo becomes a system nobody can safely change.

Choose the right measurement for the job

There is no single score for LLM evaluation. You need several measures because different failures carry different costs.

For a support answer, useful measures may include:

• Factual correctness: does the answer match the current policy?
• Grounding: can each important claim be traced to approved material?
• Resolution rate: did the user get a usable answer without escalation?
• Deflection quality: did the system avoid creating extra work for a human agent?
• Policy compliance: did it avoid making promises or taking actions it cannot support?

For a tutoring assistant, the measures change. You may care about whether the answer preserves the student's chance to reason, follows the lesson objective, gives an appropriate hint, and avoids revealing the final answer too early.

For an agent that modifies a database, a successful final message is not enough. You need to inspect the action trace. Did it call the right tool? Did it ask for confirmation before a charge? Did it modify one record or fifty? Did it recover when the API returned a timeout?

Use deterministic checks where they fit. JSON schema validation catches malformed output. Exact matching works for fixed labels. SQL assertions can verify that a tool call changed the intended record. A human reviewer may still be needed for tone, nuance, and educational quality.

Model-based grading has a place, but don't treat it as a judge from the sky. Give the grader a narrow rubric, examples of acceptable and unacceptable answers, and a pass threshold. Then compare its decisions against human labels. OpenAI's 2023 work on model-assisted evaluations described the usefulness of automated grading while also showing why grader reliability and calibration matter.

Track disagreement. If the grader says pass and a human says fail, that is not annoying noise. It's evidence that your rubric is incomplete or your grader is weak.

A practical scorecard can use pass rates for hard requirements and a separate quality rating for subjective output. Do not average away a safety failure. A response can be warm and well written while still making an unauthorized financial commitment.

Test the failures your demo avoids

A good test set is not a collection of easy questions. It is a map of the system's blast radius.

Start with real incidents. Suppose a travel assistant in Denver receives this request: "Move my flight to tomorrow, use the cheapest option, and charge the card on file." A safe system has to check whether the user is authorized, retrieve live flight data, show the price, and request confirmation before purchase. A response that claims the change is complete without a successful transaction is a serious failure, even if the prose is perfect.

Now add adversarial variations. Use misspellings, incomplete details, contradictory instructions, stale documents, long conversation history, and attempts to override system rules. Include users who ask the same question three different ways. People do that. I do that when a website makes me suspicious.

Your production AI test set should include:

• Common requests that represent normal volume.
• High-cost errors where a wrong answer creates money, safety, or legal exposure.
• Boundary cases where the system must refuse, escalate, or ask a clarifying question.
• Regression cases from previous failures.
• Distribution shifts such as new policy language, new products, or a new customer segment.

For a retrieval system, test whether the right source was retrieved before you judge the final answer. If the policy document never entered the context window, blaming the response alone sends you toward the wrong fix. You may need better chunking, metadata, document versioning, or access rules.

For an agent, test partial failure. Let the payment tool time out after authorization. Return an empty search result. Give the calendar tool a conflicting event. The question is not whether the agent can complete the ideal workflow. The question is what it does when the workflow breaks in the middle.

Here's the part most people miss: production logs are useful here, but only if you remove private data and preserve the failure structure. Sample cases by intent, customer type, channel, and severity. Otherwise your evaluation set will quietly represent the loudest users rather than the most important risks.

Make evaluation part of the shipping process

An eval that runs once is a report. An eval that runs on every meaningful change is product infrastructure.

Put the evaluation command in the same development path as the application. A pull request that changes a system prompt should trigger the relevant test suite. A model switch should run the full regression set. A retrieval change should run grounding tests, not only output-quality tests.

Set thresholds before you look at the results. For example:

• No release if any critical safety case fails.
• No release if grounded factual accuracy falls more than two percentage points.
• No release if tool-call errors rise above the agreed limit.
• Manual review when the grader and reference answer disagree.

The exact numbers depend on the product. What matters is that the team makes the decision before a favorite feature arrives with a shiny demo.

Record failures by category, not only by score. A 92 percent pass rate hides whether the eight failures are harmless wording issues or eight unauthorized account changes. A failure taxonomy gives engineers a queue they can act on.

I also separate offline and online evaluation. Offline tests tell you whether a proposed change behaves on known cases. Online monitoring tells you whether real users are bringing new cases. Production feedback should flow back into the offline set after review, with sensitive information removed.

This is the operational rhythm:

• Capture a real failure.
• Reproduce it as a stable test case.
• Fix the prompt, data, tool, model, or interface.
• Run the full regression suite.
• Ship only when the fix improves the target without breaking another behavior.

That last step matters more than most teams realize. AI fixes often move the problem rather than solve it. A shorter answer may reduce hallucination but increase escalation. A stricter refusal rule may reduce risk but block legitimate requests. Measure the tradeoff instead of arguing from anecdotes.

Once you frame it this way, evals stop being a gate at the end of development. They become the control surface for product decisions.

Give your team a budget for uncertainty

Every AI product has uncertainty. Pretending otherwise is where the real risk hides.

A production AI system should know when it's outside its tested range. That doesn't require a magical confidence score, it requires observable signals and explicit responses.

Create a Context Budget for the information the system is allowed to rely on. Track which documents, memories, tool outputs, and user claims entered the prompt. A large context window does not guarantee useful context. It may contain stale policy text, duplicate passages, or conflicting instructions.

For each important workflow, define what happens when uncertainty appears:

• Ask the user for the missing fact.
• Show the source and let the user verify it.
• Hand the request to a person.
• Refuse the action while preserving a useful next step.
• Complete only the reversible part of the task.

At my own platforms, I've learned that multi-model routing does not remove the need for measurement. It increases it. If one model handles classification, another writes the response, and a third checks the result, you now have several failure points and more latency to explain.

Watch the practical signals alongside quality:

• Pass rate by intent.
• Escalation rate by failure type.
• Tool-call success rate.
• Unsupported-claim rate.
• Median and tail latency.
• Cost per resolved request.
• Retry rate after a failed or low-quality response.

Do not optimize cost before you understand error cost. A cheaper model that creates one extra human escalation per hundred requests may cost more than the model you replaced. The arithmetic belongs in the same review as the quality score.

What I'm telling you is this: teams treat evaluation as a compliance expense, then spend far more repairing trust after users find obvious failures. Measurement is not the tax on shipping AI. It's how you decide whether the system is ready to be trusted at all.

FAQ

What are AI evals?

AI evals are repeatable tests that measure whether an AI system produces acceptable results for defined tasks. They can check factual accuracy, groundedness, safety, tool use, format, latency, cost, or user outcomes.

How is LLM evaluation different from testing a normal software feature?

Traditional software tests often compare a fixed input with a fixed output. LLM evaluation usually needs an acceptable range of answers, source checks, human or model grading, and tests for behavior across varied wording. The system can change when the model, prompt, context, or tool result changes.

What should I measure in production AI?

Measure task success, critical error rate, unsupported claims, escalation rate, tool-call accuracy, latency, cost, and retry behavior. Break results down by workflow and failure type so a strong average does not hide a serious problem.

Do I need human reviewers for AI evals?

Not for every case. Use deterministic checks for schemas, labels, citations, and tool results, then use calibrated model graders or human reviewers for nuanced judgments. Compare automated grades with human labels regularly so you know where the grader is unreliable.

How large should an AI evaluation dataset be?

Start with a small set of trusted cases that cover normal requests, high-cost errors, boundary conditions, and known failures. Expand it from reviewed production incidents. Coverage and case quality matter more than an impressive test count.

Take one workflow this week. Pull 50 real, anonymized requests from its logs. Label the acceptable behavior, mark the dangerous failures, and run the set before you touch the prompt or swap the model.

That first suite will be imperfect. Good. It gives the team a shared instrument instead of another opinion. The model is not the product. The measured behavior around the model is the product.