Why your AI demo works and your AI product does not

At 9:07 on a Tuesday morning, a support bot answered a refund question perfectly in a product demo. The prompt was clean, the customer record was easy to find, and the model gave a confident answer in under five seconds. Your AI demo can work because you control the question. Your AI product has to survive the question you did not expect.

That is the last-20% problem. The first 80% proves that an LLM can produce a useful result. The final 20% determines whether an AI product is reliable enough for real customers, real data, real latency, and real consequences. Closing that gap requires evaluation, better data boundaries, failure handling, and a product contract that says what the system will do when it is uncertain.

I have watched this pattern across large technical programs and while building multi-model AI products. The demo is rarely the hard part. The hard part is making the same system behave acceptably on a Saturday night, with a missing field, a strange request, a provider timeout, and a user who has zero patience for your architecture diagram.

The demo is a controlled experiment

An AI demo is a small stage with excellent lighting. The builder chooses the input, prepares the data, resets the conversation, and already knows which answer should appear. That is not dishonest. It is useful. A demo answers one narrow question: can the model perform this task under favorable conditions?

An AI product answers a different question: can the full system deliver an acceptable result across the messy range of situations real users create?

Consider a hotel concierge assistant tested in a conference room in Denver. The team asks, "What time does the pool close?" The retrieval system finds the current hotel policy, the model responds clearly, and everyone nods. A guest in Phoenix asks, "Can I bring my service animal to the rooftop pool after midnight?" Now the system needs location, policy version, exception handling, and a safe way to say it does not know. The product is not just a language model answering a question. It is a chain of decisions with several places to fail.

Here's the thing: conventional wisdom says to ship the MVP fast and improve it with user feedback. For ordinary software, that can work. For AI products, a fast demo often hides the fact that nobody has defined acceptable failure. You ship the happy path, users discover the edge cases, and your team spends the next quarter explaining why the system was "mostly right." Mostly right is not a product requirement.

Write down the difference between demo conditions and production conditions:

• Demo input is selected; production input is adversarial, incomplete, and repetitive.
• Demo data is fresh; production data has duplicates, stale records, and permission gaps.
• Demo latency is observed once; production latency is measured across traffic spikes and retries.
• Demo failure is edited out; production failure becomes a support ticket, refund, or lost account.

The demo proves possibility. The product must prove repeatability.

The last 20% is a reliability problem, not a prompt problem

When an AI product starts failing, teams often rewrite the prompt. Sometimes that helps. More often, the prompt is being asked to fix a problem caused by missing context, ambiguous business rules, weak retrieval, or an unhandled tool error.

Here is a support example. A company builds an assistant that classifies incoming tickets as billing, technical, or account access. On a sample of 100 carefully selected tickets, it reaches 92 correct classifications. The team refines the system prompt and gets to 94. That sounds like progress, but production reveals that forwarded email chains, screenshots, and account cancellations are common. The real distribution looks nothing like the test set.

The model did not suddenly get worse. The test was too polite.

LLM reliability has several dimensions, and a single accuracy score hides most of them. A response can be factually correct but too slow. It can be safe but uselessly vague. It can cite a real document that does not apply to the customer's region. It can call the right tool with the wrong account identifier. Each is a different product failure.

I use a Guardrail Surface to describe the places where the system needs an explicit boundary. It covers the input, retrieval, model output, tool calls, state changes, and user-facing response. At each boundary, ask what can be checked before the next step runs.

For a refund assistant, that surface might include:

• Confirming that the order belongs to the authenticated customer.
• Checking that the refund policy is current and region-specific.
• Blocking a refund tool call when the amount exceeds an approval limit.
• Requiring a human handoff when the model cannot identify the order.
• Logging the source documents and tool arguments used for the decision.

The mistake I keep watching teams make is treating reliability as a personality trait of the model. Reliability is a property of the whole path. If a model has a 2% chance of producing an unacceptable answer at one step, a workflow with several dependent model calls can produce a much larger failure rate than the demo ever suggested.

Build an Evaluation Loop before adding more features

Evaluation is not the final QA phase. It is the mechanism that tells you whether a change made the product better or merely different.

Create an Eval Loop with four parts: a representative test set, explicit scoring rules, automated runs, and review of failures. The test set should come from real product behavior as soon as you have it. Before launch, build examples from support logs, domain experts, policy documents, and deliberately difficult cases.

A useful evaluation set contains more than successful questions. Include cases where the system should refuse, ask for clarification, use a tool, cite a source, or hand off to a person. If every test expects an answer, the model learns that answering is always the right move.

For a travel booking assistant, test cases might include:

• A customer asking to change a flight after the ticket has been partially used.
• Two passengers with similar names on the same reservation.
• A request that contains a passport number in an uploaded image.
• A provider response that times out after the booking tool has accepted the request.
• A customer asking for a policy that changed last month.

Score each case against the behavior that matters. A simple rubric can track factual accuracy, correct tool selection, policy compliance, citation quality, latency, and handoff behavior. Store the model version, prompt version, retrieved documents, tool arguments, and final response. Without those fields, a failed result becomes a screenshot instead of evidence.

Public research supports this shift. The Stanford HELM project, published in 2022, evaluated language models across multiple scenarios and metrics rather than relying on a single benchmark score. The NIST AI Risk Management Framework, released in 2023, also treats measurement and monitoring as ongoing work, not a one-time certification.

Run evaluations on every meaningful change:

• A new model or provider.
• A new retrieval index or chunking method.
• A prompt change that alters tool use.
• A policy update or new customer segment.
• A change to timeout, retry, or fallback behavior.

Then review failures by category. If 40% of failures trace back to stale policy documents, another prompt rewrite is not the next move. Fix the document pipeline.

Context is a budget, and bad context spends it badly

A model can only reason over the context you provide. More context does not automatically produce a better answer. It can bury the relevant fact, mix old and new policies, and drive up cost and latency at the same time.

Treat context as a Context Budget. Every retrieved passage, conversation turn, tool result, and instruction competes for attention. The model may support a large context window, but your product still needs to decide what belongs there and in what order.

Take a technical support assistant for a router manufacturer. A customer reports that a device drops connection every evening. The retrieval layer returns 12 documents because they all contain the phrase "connection drops." Four are for older firmware, three describe a different hardware revision, and one contains an internal troubleshooting note customers should never see. The model has plenty of text. It does not have clean evidence.

A better retrieval path filters by product version, region, document status, and access level before ranking passages by semantic similarity. Postgres with pgvector can support a practical early system when the data volume and query patterns fit. Pinecone and other vector stores can help as scale and operational needs grow. The storage choice matters less than the rules around freshness and access.

Your retrieval tests should ask:

• Did the system return the document that contains the answer?
• Did it exclude documents that are obsolete or unauthorized?
• Did it preserve the source and version for citation?
• Did the model answer only from the returned evidence?

A useful production trace records the query, filters, retrieved document IDs, scores, final context, and answer. That makes a wrong answer diagnosable. Otherwise, you are staring at a response and guessing whether the problem came from embedding quality, metadata, ranking, or generation.

So the deeper lesson is this: retrieval-augmented generation is not mainly a search problem. It is a data-quality and access-control problem with a search component attached. The model cannot repair a knowledge base that shows it the wrong policy.

Production behavior needs its own design

A demo usually has one request and one response. Production has queues, retries, duplicate clicks, partial tool success, provider outages, rate limits, and users who refresh the page because nothing happened.

A model call that takes four seconds in development can take 20 seconds when the provider is under load. A retry can double your cost. Worse, a retry after a successful payment or booking call can create a duplicate transaction. These are not model-quality issues. They are system design issues. Full stop.

Teams optimize prompt token cost to the third decimal and then absorb a four-times bill from an unbounded retry storm nobody instrumented. Instrument the retry path before you tune the prompt.

Build a failure policy for each external dependency:

• Set a timeout that matches the user experience, not the provider's maximum.
• Use bounded retries with jitter and an idempotency token for state-changing actions.
• Return a clear fallback when the model or retrieval service is unavailable.
• Separate read-only model calls from actions that change money, access, or records.
• Record provider, model, latency, token usage, status, and retry count.

Imagine a claims assistant used by an insurance team in Columbus, Ohio. It extracts fields from a scanned form, calls a coverage service, and drafts a response. The extraction succeeds, the coverage service times out, the workflow retries. If the system does not preserve the original request ID, the second attempt may create a second case or overwrite a human correction. If you've ever had to explain a duplicate transaction to a user at 11 PM, you already know how fast this becomes a trust problem.

This is where Cost Telemetry becomes part of product quality. Track spend per completed task, not just total API usage. A low token bill means little if users abandon the workflow after a slow response or if support staff must verify every output by hand.

Measure operational signals alongside answer quality:

• Completion rate by workflow type.
• P50, P95, and P99 latency.
• Retry rate and timeout rate.
• Human handoff rate.
• Cost per successful task.
• Unacceptable output rate from sampled production reviews.

Once you frame it this way, reliability becomes visible. You can decide whether a slower answer with strong evidence is better than a fast answer that needs correction. The answer depends on the task, but the decision should come from measured behavior, not instinct.

Turn the AI capability into a product contract

The final gap is often a product management gap. The team knows what the model can do, but users do not know what the product promises.

A product contract states the job, the evidence standard, the limits, and the recovery path. It does not need legal language. It needs operational clarity. "Answers questions about internal policy" is too broad. "Finds the current travel reimbursement rule, cites the source, and sends uncertain cases to finance" is testable.

For each workflow, define:

• The user problem and the action the system supports.
• The facts the system must get right.
• The actions it may take without approval.
• The situations where it must ask, refuse, or hand off.
• The evidence and audit trail a reviewer can inspect.

I learned this the hard way on multi-model products. Adding another model can improve a narrow task, but it also adds routing logic, different latency, different failure modes, and another provider dependency. A model router is not automatically an architecture improvement. It is a new surface that needs tests.

The same applies to agents. An agent that can browse, call APIs, write records, and continue across turns may look impressive in a demo. The product question is narrower: which actions should be autonomous, and which require confirmation? Give autonomy to low-risk, reversible work first. Put approval gates around money movement, account changes, deletion, and sensitive data.

Your release review should ask questions that a demo cannot answer:

• What happens when the user provides incomplete information?
• What happens when the model is confident and wrong?
• What happens when the source data conflicts?
• What happens when the provider is down?
• Who owns the failure after launch?

What I'm telling you is this: define the contract, build the evaluation set from that contract, instrument the path that carries real user value, then make feature decisions based on failure evidence, not on how polished the demo feels.

FAQ

Why does my AI demo work but my production system fail?

A demo uses selected inputs, prepared data, and a known expected answer. Production includes incomplete requests, stale records, provider errors, repeated conversations, and users who behave in ways your test script never covered. The gap is usually caused by missing evaluation and failure handling rather than one bad prompt.

What is the last-20% problem in an AI product?

The last-20% problem is the work required to move from a convincing capability demonstration to dependable product behavior. It includes representative evaluation, retrieval quality, access controls, latency, retries, cost, monitoring, human handoff, and clear limits.

How should I evaluate LLM reliability?

Build a test set from real or realistic user tasks, including cases where the correct behavior is to refuse, ask a question, cite evidence, or hand off. Score factual accuracy, tool use, policy compliance, latency, citation quality, and unacceptable-output rate. Run the set whenever you change the model, prompt, retrieval system, or workflow.

Do I need a vector database for a production AI product?

Not always. Postgres with pgvector can be a practical choice when your data volume, filtering needs, and operational model fit it. A dedicated vector service may make sense at larger scale, but no database fixes stale documents, weak metadata, or missing access rules.

What should I fix before adding another AI feature?

Fix the highest-volume or highest-risk failure category in your current workflow. Check traces, review production samples, and measure cost per successful task before expanding scope. A smaller system with a strong evaluation loop usually teaches you more than a larger system with an attractive demo.

Start with 50 real tasks from your product. Label the acceptable behavior for each one, including refusal and handoff cases. Run the current system against them, inspect every failure, and fix the largest category before you add another feature.

The model is not the product. The system around the model is the product, and the last 20% is where users decide whether to trust it.