The eval loop is the product: why I build evaluations before features in any AI app

At 10:42 on a Tuesday morning, Priya sent me a screenshot from the support assistant her team was building. The bot had answered a refund question with total confidence, cited a policy that did not exist, and told the customer to wait seven business days. The actual policy allowed three.

That is the moment AI evals stop being an abstract engineering concern. An evaluation is a repeatable way to ask whether an AI system gave the right answer for a specific situation. I build the Eval Loop before I build most features because the loop tells me which features deserve to exist.

I have watched this pattern across two live multi-model platforms and the technical programs I shipped at large companies. A polished feature can hide a weak product for weeks. A small set of honest tests exposes the weakness before users have to do it for you.

The feature is not the product

The usual product sequence sounds reasonable: collect requests, design the feature, connect a model, test before launch. That order works when software follows fixed rules. It breaks the moment the system writes, summarizes, classifies, searches, or acts on probabilities.

Conventional wisdom says ship the smallest useful version and learn from users. I agree with the learning part. I disagree with treating evaluation as something you bolt on after the first version ships. In an AI product, the first version is already making judgments. If you have not defined how to judge those outputs, users become your test suite.

That is an expensive way to learn.

A chatbot answering billing questions carries a different standard from a writing assistant. The first may need exact policy adherence. The second may need useful suggestions without flattening the writer's voice. A tutor may need to guide a learner without handing over the answer. "The response sounds good" is not a product requirement.

Before I add a feature, I want to answer four questions:

• What user situation is this feature meant to improve?
• What does a successful answer or action actually look like?
• Which errors would cause a user to lose trust?
• How will I detect a regression after changing the prompt, model, retrieval data, or workflow?

Those questions turn a vague feature request into something buildable. They also give a product manager a way to say no. That matters. AI teams can spend a month polishing a capability that has no reliable pass condition.

So the deeper lesson is simple: the feature creates an output, but the eval loop creates the product standard. Without the loop, you are shipping behavior you cannot explain or control.

Build the test set before the interface

When I start a new AI workflow, I do not begin with the screen. I begin with examples.

Some are ordinary. A customer asks whether a hotel cancellation fee applies. A learner asks for a hint on a fractions problem. A marketing manager asks for four versions of a product description in a defined voice. These map the work the system must actually perform.

Other examples are deliberately uncomfortable. The customer asks about a policy that changed yesterday. The learner pastes an answer from a homework site and asks for confirmation. The marketing manager supplies an incomplete brief with a prohibited claim. Real users do this constantly, usually while busy and mildly annoyed.

A useful starting test set covers several kinds of cases:

• Normal cases that represent the common workflow.
• Boundary cases that sit close to a policy or capability limit.
• Adversarial cases that try to trigger unsafe, fabricated, or off-task behavior.
• Historical failures pulled from logs, support tickets, and review notes.

I write the expected result in plain language before I write a scoring script. "Uses the current three-day policy, does not invent a citation, and offers the next action" is more useful than "high quality." It gives a reviewer something concrete to inspect.

This is where LLM evaluation gets more nuanced than traditional unit testing. There may not be one perfect sentence. A response can use different wording and still be correct. So I separate the test into claims that can be checked independently.

For a support response, those claims might be:

• It identifies the correct policy version.
• It answers the user's actual question.
• It avoids unsupported details.
• It gives an appropriate next step.

That structure also helps when one model grades another. I can ask a judge model to score each claim, but I do not treat the judge as an oracle. I compare its decisions against human ratings on a sample, inspect disagreements, and keep a small set of known answers for calibration. (If you have ever watched a demo fall apart on the third question, you already understand why this calibration step is not optional.)

OpenAI's Evals framework, Anthropic's public work on model evaluations, and tools like LangSmith or Braintrust all point toward the same operating habit: save examples, run them repeatedly, inspect changes over time. The tool matters less than the discipline.

And yes, a spreadsheet is a perfectly respectable place to begin. I have built systems with far more infrastructure than a spreadsheet that were less useful because nobody agreed on what counted as a pass.

Treat failures as product evidence

A failed evaluation is not a red mark against the team. It is information about the product contract.

Suppose an AI assistant fails 18 percent of refund questions. That number tells you something, but not enough. You need to group the failures. Maybe the assistant retrieves an old policy. Maybe it understands the policy but stumbles on questions that include dates. Maybe the answer is correct but buried in a long paragraph the customer cannot act on.

Each cause points to a different product decision. Better retrieval will not fix a missing policy. A longer prompt will not fix a workflow that sends the wrong account context. A new model may improve wording while leaving the actual failure completely untouched.

This is why I track failure categories rather than one overall score. For each test run, I want to see:

• Accuracy on the business claim.
• Grounding in the supplied source material.
• Whether the response follows the requested format.
• Whether the system refuses or escalates when it should.
• Cost and latency for the completed interaction.

The last two often get ignored. They should not. An answer that is accurate but takes 45 seconds may still fail the support experience. An answer that costs too much at scale may force a product change even when the model is performing well.

Here's the part most people miss: teams fix the most visible failure rather than the most damaging one. A typo in a generated email gets attention because it is easy to see. A subtle false statement about eligibility may affect fewer test cases but carry far more risk.

Prioritize failures on a practical combination of frequency, user harm, and recovery cost. A frequent minor formatting issue can wait. A rare answer that invents a legal policy should block launch.

This is also where emotional honesty helps. Teams often feel a low evaluation score means their idea is failing. Sometimes it does. More often, the score says the idea needs a narrower promise.

A general-purpose assistant may perform poorly across twenty tasks but perform well as a refund-policy assistant for one product line. Narrowing the first release is not admitting defeat, it is giving the system a job it can actually do.

With the multi-model platforms I have built, I have had to make this choice repeatedly. One model may be better at structured extraction. Another may produce warmer explanations. A third may cost less for routine requests. The evaluation set lets me compare those tradeoffs against the same user situations instead of picking a winner based on a memorable demo.

Put the Eval Loop inside AI product management

AI product management changes when evaluation is part of planning rather than a gate at the end.

A standard roadmap might contain "add document upload" or "support another model." Those are implementation tasks. I prefer to write the desired behavior first: "For a customer who uploads a current policy, answer questions using the policy's rules and flag when the document is incomplete." The feature is then one possible way to improve the pass rate.

That shift affects the whole team. Designers can show uncertainty and escalation states. Engineers can instrument retrieval and retries. Product managers can decide whether a failure is acceptable for the intended audience. Support teams can contribute real examples instead of hearing about quality after the first release.

Set a small release contract before work begins. Mine usually includes:

• A test set with enough normal and difficult cases to surface the main risks.
• A pass definition for each important behavior.
• A maximum acceptable rate for severe failures.
• A plan for reviewing new failures after release.

Do not wait for a perfect benchmark. Start with 25 cases if those are the ones you can review carefully. Then add cases from real use. A test set should grow when the product meets a new user type, source document, language, or failure mode.

Keep the evaluation run close to the code. When a prompt changes, run the set. When the retrieval index changes, run the set. When you swap GPT-4o for another model, run the set. When you change a timeout and add retries, run the set again.

That last point matters more than it sounds. Teams often tune prompt tokens to save a few cents, then create a retry storm that multiplies the bill. Cost and latency belong in the same review as answer quality, because users experience the whole path, not just the generated paragraph.

I also keep a Context Budget for every workflow. It is a plain record of what information the model receives, what gets dropped, and how much room remains for the answer. If a system fails only when a customer includes a long email thread, the problem is likely context selection rather than reasoning.

Once you frame it this way, the roadmap becomes less about adding model tricks and more about reducing known failure rates. That is a healthier conversation for founders, recruiters, and hiring managers assessing an AI product leader. Ask what the candidate measures, how they choose test cases, and what they do when a score drops after a release.

Strong AI product management does not require pretending the model is predictable. It requires building a process that makes unpredictable behavior visible enough to manage.

A practical starting plan for your next AI feature

If your team has no evaluation practice today, do not begin by buying a platform. Begin with one workflow that matters and one person who will review failures every week.

Use this sequence:

• Write down the user promise in one sentence.
• Collect 20 to 30 real or carefully reconstructed examples.
• Mark the claims, actions, and refusals that matter in each example.
• Run the current system and label every failure by cause.
• Fix one failure category, then rerun the full set.
• Save the results so future model or prompt changes can be compared.

A simple scorecard can live in Postgres, a Git repository, or a shared document. Later, connect it to LangSmith, Braintrust, Humanloop, an internal harness, or your own scripts. The first win is not automation. The first win is shared judgment about what counts as correct.

You will also need a review rhythm. I like a short weekly review while a workflow is changing quickly. Look at new failures, disagreements between human and automated graders, cost shifts, and sessions where users abandon the interaction. Those signals tell you whether the product is becoming more dependable or just more impressive in demos.

What I'm telling you is that the first test set will be incomplete. Your labels will disagree with each other. Someone will discover that the "obvious" success case depends on a detail nobody wrote down. Good. That discovery is the work.

The people building AI products right now are carrying an unusual burden. You are asked to move fast while the system can produce fluent nonsense, hide uncertainty, and shift behavior whenever a prompt or model changes. That is hard because it is hard. Not because you are doing it wrong.

Start with one promise. Test it honestly. Let the failures decide what you build next.

FAQ

What are AI evals?

AI evals are repeatable tests that measure whether an AI system behaves correctly for defined user situations. They can assess factual accuracy, policy adherence, format, safety, tool use, cost, latency, and whether the system asks for help when it should.

How is LLM evaluation different from normal software testing?

Normal software tests often compare a fixed output with an expected output. LLM evaluation usually allows multiple valid responses, so teams score claims, behavior, and risks rather than exact wording. Human review remains useful, especially for checking automated graders and ambiguous cases.

When should an AI team build evaluations?

Build the first evaluations before the feature is complete and before the interface is polished. A small test set can reveal that the promise is too broad, the source data is incomplete, or the chosen model is a poor fit while those decisions are still inexpensive to change.

What should an AI product manager measure?

Measure the behaviors that affect the user's decision, along with severe failure rates, escalation quality, latency, and cost. A single overall score can hide the one error that damages trust, so keep results broken down by failure type and user situation.