The three numbers I watch on every AI product I ship

At 9:17 on a Tuesday morning, Sarah opened the support assistant she had helped launch the week before. Marcus asked it a simple refund question. The answer arrived in 2.8 seconds, sounded confident, and cited a policy that did not exist.

That is the moment AI metrics stop being a dashboard exercise. On every AI product I ship, I watch three numbers: quality, latency, and cost. These AI product KPIs tell me whether the product is useful, usable, and able to survive real usage without quietly setting money on fire.

I've shipped systems across large companies and built live multi-model AI products of my own. The pattern is consistent. A model can look impressive in a demo while one of those numbers is already telling you the product is in trouble.

The dashboard has three gauges

I think of an AI product as a car with three gauges. Quality tells you whether the engine is taking you to the right place. Latency tells you whether the ride feels usable. Cost tells you whether you can afford to keep driving.

The metaphor is not perfect. No metaphor survives contact with a retry storm. But it gives a product team a shared language for talking about tradeoffs without hiding behind model rankings or a glossy demo.

Here are the three numbers I put on the dashboard:

• Quality: the percentage of responses that meet the actual user need.
• Latency: the time a user waits for a complete, useful response.
• Cost: the total spend required to produce that response.

The order matters less than the connection between them. A cheap answer that is wrong is not efficient. A perfect answer that arrives after the user has left is not useful. A fast answer that burns through your gross margin is a pricing problem wearing an engineering costume.

The mistake I keep watching teams make is treating these as separate departments. Product owns quality. Engineering owns latency. Finance discovers cost during a quarterly review. That division is tidy on paper and disastrous in production.

The three gauges describe one system.

Quality is not a vibe

Quality is the number teams argue about most and measure least. People run ten prompts through a new model, admire the answers, and say, "This feels better." I've done versions of that myself. It's a very human way to confuse novelty with progress.

A useful quality metric starts with a defined task. For a support bot, you might measure whether the answer gives the correct refund window, cites an approved policy, and asks for human review when the case falls outside policy.

That creates a small evaluation set. Not a giant research project. A spreadsheet with real examples is enough to begin.

Your first Quality Set should include:

• Common questions that represent most sessions.
• Difficult cases where the source material is incomplete or contradictory.
• Unsafe requests that need refusal or escalation.
• Recent failures copied from production, with identifying details removed.

Score each response against a simple rubric. A response can be correct, partly correct, wrong, or unsafe. You can use a language model as a judge for some comparisons, but keep human review in the loop for the cases that affect money, safety, or trust.

For one product, I might track "percentage of answers that pass all required checks." For another, I might track "task completion rate." Those are different measures because the user is trying to accomplish a different thing.

So the deeper lesson is this: quality is not the model's general intelligence. Quality is whether the product completed the job you promised.

What should you measure when quality is hard to define?

Measure the user's next meaningful action. If a customer uses an AI assistant to draft a reply, track whether they accept or edit the draft. If they use it to search policies, track whether they find the right policy and stop asking the same question.

Those signals are not perfect. A user can accept a bad answer. But they give you a starting point that is closer to value than a model score on a public benchmark.

I also separate two numbers that often get blended together:

• Answer quality: whether the generated content is correct and useful.
• System quality: whether retrieval, routing, tools, memory, and fallback behavior worked as intended.

A wrong answer may come from the model. It may also come from stale data in Postgres, a failed vector search, a truncated context window, or a tool that returned an empty result. If you only score the final text, you may spend a week tuning prompts while the real fault sits upstream.

That is why I build an Eval Loop early. It is a repeatable set of examples, scoring rules, and production failures that lets me compare a change against the previous version. The loop does not need to be elegant. It needs to run every time the prompt, model, retrieval logic, or tool behavior changes.

Latency is part of the product promise

People describe latency as an infrastructure concern. Users experience it as doubt.

When an AI assistant takes six seconds to answer, the user starts wondering whether it heard them. At ten seconds, they may click again. At twenty seconds, they may open a new tab and solve the problem another way. The backend may still be working perfectly. The product has already lost the moment.

I watch at least four latency points:

• Time to first token, which tells me when the interface starts responding.
• Time to first useful content, which matters more than a meaningless loading animation.
• Time to complete response, which affects whether the task can finish.
• Tail latency, such as the 95th or 99th percentile, which shows what slower users actually experience.

The average can lie. A system with a two-second average might still send one in twenty users into a twelve-second wait. Those users are not comforted by your average.

Streaming can make a response feel faster, but it does not repair a slow system. If the first sentence appears quickly and the useful answer arrives much later, you have improved perception without improving completion time. Sometimes that is worthwhile. Just name the tradeoff accurately.

Here's the part most people miss: latency has a budget. If your product promise is "answer a customer question in under five seconds," every part of the request gets a slice of that budget.

A rough budget might look like this:

• 300 milliseconds for authentication and request setup.
• 700 milliseconds for retrieval and database work.
• 2.5 seconds for model generation.
• 1 second for tool calls, validation, and the response path.

The numbers will vary. The discipline is what matters. Once you write down the budget, a slow feature becomes diagnosable instead of mysterious.

Instrument the path before you tune the prompt. Record model selection, prompt size, retrieval time, tool duration, output length, retries, and total request time. With multiple models in the system, tag every request so you can compare providers honestly.

A faster model may produce a lower quality answer. A smaller context may reduce time while removing the policy detail needed for correctness. The dashboard should make that exchange visible.

And give yourself permission to set different latency targets for different jobs. A live sales assistant may need a response in two seconds. A nightly document classifier may be fine at two minutes. One universal target usually creates the wrong pressure somewhere (and yes, I've watched teams optimize for the wrong job because nobody wrote down which one mattered more).

Cost is not just the model invoice

The third number is where most teams get surprised. They calculate the input and output token charge, then discover that the real cost includes retries, retrieval, moderation, image generation, storage, observability, and support for the cases that fail.

Model pricing still matters. OpenAI, Anthropic, Google, and other providers publish different rates for input, output, cached context, and sometimes batch work. But the invoice is only one part of the product's cost structure.

I track cost per successful task, not only cost per request. A request that fails and retries three times should not look cheaper than a request that finishes once. A response that requires a human agent to repair it carries a support cost even if the API call was inexpensive.

My Cost Telemetry usually includes:

• Model spend by feature, route, tenant, and model.
• Average input and output tokens per successful task.
• Retry rate and the spend attached to retries.
• Tool, retrieval, storage, and moderation costs.
• Human review or support work caused by failed responses.

Here's the conventional wisdom I want to push back on: optimize token cost first. That is often the wrong first move. Teams trim a prompt by a few hundred tokens and then allow a broken tool call to trigger four expensive retries. The small saving is visible on the dashboard. The larger leak is not.

Watch the full cost path.

For a multi-model product, route by task rather than reputation. A small, fast model may handle classification or extraction. A larger model may handle a complicated customer request. A vision model may be appropriate for an image task, while a text model should never see that work at all.

Do not route purely by price. Route by the quality and latency target the task requires. Then calculate whether the route meets your margin goal.

A practical unit metric is:

`cost per successful task = total AI and support spend / completed user tasks`

That number is much harder to game than cost per call. It also gives product and finance a sentence they can share without needing a seminar on tokens.

If your AI feature is free, cost still matters. Free users consume capacity, affect rate limits, and shape the economics of future pricing. You do not need a perfect forecast on day one. You do need to know what one successful task costs before usage becomes a surprise.

Make the three numbers useful to a team

A dashboard does not improve a product by existing. Someone has to look at it, discuss the tradeoffs, and decide what changes next.

I keep the review small and regular. Once a week, I look at a sample of real interactions alongside quality, latency, and cost. I want to see the same request through all three gauges.

For each meaningful product change, ask:

• Did quality move for the better or worse?
• Did the response get faster for typical and slow users?
• Did cost per successful task rise, fall, or stay flat?
• Which users or tasks experienced the biggest change?

Do not wait for perfect instrumentation. Start with ten to twenty representative tasks, a stopwatch at the interface, and a cost estimate from provider logs. Then improve the measurement as the product earns more usage.

I also write down thresholds before a release. A feature may ship only if quality stays above 90 percent on the evaluation set, p95 latency stays below five seconds, and cost per successful task stays under the planned ceiling. Those are the gauges I check before I hand the keys to anyone.

The exact thresholds belong to the product. The habit belongs to every AI product.

This approach also helps with something easy to overlook: team morale. When an answer fails, people can feel as if the entire system is broken. A shared dashboard turns a vague failure into a specific conversation. Quality fell on policy questions. Latency rose after retrieval was added. Cost increased because the fallback route ran too often.

That clarity is a form of kindness. It gives engineers something they can fix and gives product leaders a way to protect users without blaming the person who happened to be on call.

What I'm telling you is not to build a giant observability platform before launch. It is to make the product's tradeoffs visible while they are still small enough to change.

A practical starting point for your next release

Before you ship the next AI feature, create one page with these fields:

• The user task the feature must complete.
• Ten representative examples and the quality rubric.
• The target response time and the p95 limit.
• The cost per successful task you can support.
• The fallback behavior when the model, tool, or data fails.

Then test the feature under ordinary pressure. Ask the same question twice. Remove a source document. Force a timeout. Send a longer input. Try a request that should be escalated. Watch what happens to all three numbers.

You will learn more from those tests than from another hour of model comparison.

If your product is already live, start with the last twenty failed or abandoned interactions. Mark what went wrong, measure the wait, estimate the spend, and group the failures by cause. That small review often surfaces the next engineering task faster than a roadmap workshop.

And if your numbers are messy, that does not mean you are behind. It means you are looking at the real product now, rather than the version that behaved nicely in a demo.

A good AI product is not one that wins a benchmark and stays quiet. It is one that gives users a correct enough answer, quickly enough to use, at a cost the business can carry.

Watch quality, latency, and cost together. The model is only one part of the dashboard. The product is the whole instrument panel.

FAQ

What are the most important AI metrics for an early product?

Start with quality on representative user tasks, p95 latency, and cost per successful task. These cover whether the product works, feels usable, and can support continued usage.

How do I measure AI quality without a large data science team?

Create a small evaluation set from real or carefully written examples, define pass and fail criteria, and review a sample by hand. You can use an LLM judge for routine comparisons, but keep human review for high-risk cases and unexpected failures.

Should I optimize AI quality, latency, or cost first?

Fix failures that make the product untrustworthy before chasing speed or lower spend. After the product meets its quality bar, use latency and cost budgets to choose models, context size, retrieval methods, and fallback routes.

What is the best cost metric for an AI product?

Cost per successful task is usually more useful than cost per API request. It includes failed calls, retries, supporting services, and the spend required to complete the user's actual job.

You are not supposed to have all three numbers perfect on launch day. You are supposed to know which one is hurting users or the business, and make the next fix visible. That is enough to start.