AI latency is a product experience problem

At 9:12 on a Tuesday morning, a support agent in Phoenix clicks "draft reply" and watches a spinner sit still while a customer waits on the phone. The model may need six seconds, but the customer experiences one thing: the product stopped responding.

AI latency is not just an infrastructure metric. It is a product experience problem. You reduce LLM latency by designing the workflow around waiting, streaming useful output, splitting long tasks, and measuring the moments that users actually feel.

I have shipped enough software to know that a fast model cannot rescue a badly designed interaction. A slow model can still feel good when the product explains what is happening and gives the user something useful early.

Latency is a product promise

Traditional software answers a request in a predictable window. A database query takes 80 milliseconds. A page loads in 1.4 seconds. An API returns a status code. LLM applications are different because generation happens over time, and the model may call tools, retrieve documents, retry a request, or wait for another service.

That creates several types of delay:

• Time to first token: how long the user waits before seeing any generated output.
• Time to useful result: how long until the answer contains something the user can act on.
• Total completion time: how long the full response or workflow takes.
• Interaction delay: how long the system takes to respond after the user changes direction.

These numbers do not mean the same thing. A legal research assistant may stream a useful outline after two seconds and finish its citations in 18 seconds. A travel booking agent that displays partial prose for 18 seconds is still frustrating if the user only needs to know whether a reservation exists.

Nielsen Norman Group's long-standing response-time guidance gives a useful starting point: around 0.1 seconds feels immediate, around one second keeps the user's flow intact, and around ten seconds tests patience. LLM products often blow past those boundaries, so you cannot pretend the old rules disappeared. You need to design around them.

The mistake I keep watching teams make is treating p95 latency as the whole user experience. A p95 chart might say the system responds in 11 seconds. It does not tell you whether users saw useful content at second two, whether the interface looked broken, or whether the workflow needed 11 seconds in the first place.

What I'm telling you is simple: latency is part of your product promise. If the interface implies "instant answer," six seconds feels broken. If it says "I'm checking three sources and will return a reviewed answer," the same six seconds can feel responsible.

Measure the moments users feel

AI application performance needs more than one stopwatch. I use a latency budget that follows the user's experience from click to outcome.

For a customer-support copilot, that budget might look like this:

• 300 milliseconds to acknowledge the click and disable duplicate submissions.
• 1.5 seconds to show the source documents or case facts being inspected.
• 2 seconds to stream a draft opening sentence.
• 8 seconds to produce a complete suggested reply.
• 20 seconds to finish policy checks and mark the answer as ready for review.

The exact numbers will vary. The structure matters because it separates visible progress from invisible work.

Instrument each stage with a request ID. Record model name, region, prompt size, output tokens, tool calls, queue time, retry count, time to first token, time to first useful token, and total completion time. Store these values with the product event that initiated the request, not just in a provider dashboard.

A provider dashboard can tell you that a model call took 7.4 seconds. Your product telemetry needs to tell you that the user waited 3.1 seconds before seeing a draft, then abandoned the workflow at 9 seconds because a policy lookup had not finished.

I call this the Perceived Latency Budget. It is not a claim about what the server did. It is a working model of what the user experienced.

Track these metrics by workflow, not only by endpoint:

• Median and p95 time to first useful output.
• Percentage of requests that produce visible progress within two seconds.
• Abandonment rate at five, ten, and 20 seconds.
• Retry rate caused by timeouts or malformed tool calls.
• Completion rate for workflows that use one model call versus several.

Once you frame it this way, optimization gets more practical. You may discover that shaving 400 milliseconds from retrieval matters less than moving a policy check after the first draft appears. You may also find that a "fast" model creates more retries, making the full workflow slower and more expensive.

Stream something useful, not just something early

Streaming is often treated as a checkbox. Turn on server-sent events, print tokens as they arrive, and call the product responsive. That works only when the first tokens help the user understand or act.

A streamed paragraph that begins with "Sure, I can help you with that" is not progress. It is animated waiting.

A better pattern is to stream in stages. Start with a short status message, then expose the first useful artifact, then continue with detail. In a support tool, the user might see "Checking refund policy" followed by "The order is eligible for a refund through September 24." The polished email can arrive after that.

For an internal analyst asking Postgres and an LLM to explain a revenue change, stream the query status and the first verified finding separately. Do not make the person wait for a five-paragraph narrative before revealing that the Northeast region fell 12% week over week.

Streaming has limits. It does not reduce model compute. It reduces the silence before output and gives the interface a chance to maintain trust. If the system later rejects its own draft because a tool returned conflicting data, early text can become harmful rather than helpful.

Use an output contract that separates provisional material from approved material. Labels like "draft," "checking," and "ready to send" are not decoration. They tell the user what can be trusted now.

Good streaming behavior usually has four properties:

• The first visible update explains what the system is doing.
• The first content block gives the user a usable fact, decision, or next step.
• Later updates improve the result rather than repeat the same status.
• The interface makes clear when the answer is still being checked.

At InterContinental Hotels, a front-desk workflow might need a room change confirmed against live inventory before it can promise anything. A useful interface could show the requested room type immediately, then stream the availability check, then present a confirmation. It should not invent a confident answer while the reservation system is still silent. If you've ever watched a demo fall apart on the third question, you already know how quickly that silence becomes distrust.

So the deeper lesson is that streaming is a communication design problem. Token velocity is not the goal. Useful progress is.

Split long workflows into product stages

Long AI workflows become painful when the system treats every task as one giant request. A customer asks for a market brief, and the application retrieves documents, summarizes them, checks dates, creates charts, reviews citations, and formats a PDF before showing anything. The user sees a spinner. The system sees a pipeline.

Break the pipeline into stages that can stand on their own. Each stage should have an output, a timeout, and a recovery path.

Consider a Monday morning workflow for a sales manager in Chicago who asks for a brief on five target accounts. The product can:

• Confirm the accounts and return an initial scope in under two seconds.
• Retrieve public filings and internal notes in parallel.
• Generate a rough account summary as each source set completes.
• Run factual checks on dates, amounts, and named executives.
• Produce the polished brief as a background job.

The manager can start reading while the later stages run. If one source fails, the product can mark that section as incomplete instead of throwing away the entire job.

This is where queues and background workers matter. A web request should not hold open a fragile connection while a 90-second workflow performs six model calls. Store workflow state in a durable system like Postgres, publish jobs through a queue, and let the interface poll or subscribe to progress updates.

Use parallel work only when the dependencies allow it. Retrieval from three independent sources can happen together. A final synthesis must wait for those results. Parallelism can cut wall-clock time, but it also increases load, rate-limit pressure, and the number of partial failures you must handle.

Here's the part most people miss: the default startup advice is "make the MVP fast." For AI products, that often produces a demo with one oversized prompt and no recovery points. The first useful version should instead have a small number of explicit stages, because those stages become the load-bearing structure for retries, cancellation, and human review.

Give users control over long work. Let them cancel a job, change its scope, or leave the page and return later. If a workflow takes longer than a normal interaction, it has become a job, not a chat response.

Reduce LLM latency at the system level

You can reduce LLM latency by changing more than the model. The model call is one part of the path, and often not the slowest part.

Start with the request itself. Large prompts take longer to process and cost more. Remove duplicate instructions, compress repeated history, retrieve only the documents needed for the current task, and avoid sending a full transcript when a structured state object will do.

Then inspect the surrounding services:

• Put application servers near the model provider region when possible.
• Cache stable retrieval results and repeated policy documents.
• Reuse connections to Postgres, Redis, and external APIs.
• Set separate timeouts for retrieval, generation, and validation.
• Cancel abandoned requests so they do not keep consuming capacity.
• Route simple classification or extraction to a smaller model.

A multi-model application can also use a fast model for intent detection and drafting, then a slower model for a narrow review step. That is different from asking the slow model to do every task. Keep the expensive reasoning where it changes the outcome.

Be careful with retries. A request that times out after 20 seconds and then retries twice has created a 60-second user experience, not a 20-second one. Exponential backoff is useful for transient failures, but a user-facing workflow also needs a retry ceiling and a visible fallback.

At a Verizon operations desk, a network alert assistant might need to classify thousands of events quickly, while only a small number deserve a deeper explanation. The fast path can group alerts and suppress duplicates. The slow path can analyze probable causes for the handful that an engineer actually opens.

This is a Latency Control Surface: the set of product and system decisions that determine how much waiting a user experiences. It includes model choice, prompt size, retrieval scope, tool order, queue policy, caching, and interface behavior.

Do not optimize these parts in isolation. A smaller prompt may lower generation time but remove the evidence needed for accuracy. A faster model may increase review work. A cache may serve stale policy data. Measure the outcome, not just the component.

Design for waiting without making users babysit the job

A waiting interface needs a contract. Tell the user what will happen, show what has happened, and provide a way out.

For a real-time AI system, that contract might include a visible status line, a progress phase, an estimated next event, and a cancel action. Do not use a fake percentage that advances from 10% to 90% and then sits there. Users notice, and they remember.

Use language that maps to actual work:

• "Searching 14 approved documents."
• "Comparing the answer with refund policy."
• "Draft ready. Citation check still running."
• "This source did not respond. Continue without it?"

A progress message should be specific enough to reduce uncertainty without exposing internal implementation details that mean nothing to the user. "Running agent loop" is a system label. "Checking inventory and rate rules" explains the work.

Give people a useful escape hatch. They may not need the finished report. They may need the first three findings. They may want to edit the question while the system gathers data, or they'd prefer an email notification when the job is complete.

The interaction should also preserve state. If someone leaves a browser tab while a background workflow runs, returning to a blank chat is a product failure. Save the job, its partial results, its errors, and its current stage.

And do not make every workflow real time. A real-time interface is appropriate when the user is actively making a decision. A background job is better when the output is a report, a batch classification, or a scheduled review.

The problem is rarely the spinner itself. The problem is that the product has no honest answer to "What is happening, and what can I do while I wait?"

Build a latency review into product development

Treat latency as a release criterion, not a cleanup task after the model works. Before launch, define the slowest acceptable experience for each important workflow.

Write a small test set with real paths. Include a short prompt, a long conversation, a tool timeout, a missing document, a provider error, and a user who cancels halfway through. Run it against the same model and infrastructure your customers will use.

Review these questions before shipping:

• Does the user see a meaningful update within two seconds?
• Can the user identify whether the answer is draft or complete?
• Does a slow tool block work that could happen in parallel?
• What happens after the first timeout?
• Can the user leave and return without losing progress?
• Does the system stop work when the user cancels?
• Is the retry path visible in telemetry and in the cost report?

Use p50, p95, and abandonment together. A median response time can look healthy while a meaningful group of users waits through repeated retries. Slice the data by workflow, geography, model, prompt length, and time of day.

When I ship multi-model AI products, I care about the handoff between system stages as much as the model's raw response speed. A 300-millisecond model can sit behind a two-second queue and a five-second retrieval service. The user does not experience model latency, they experience the full chain.

Start with one workflow. Instrument every stage. Watch five real sessions, not only a dashboard. Then remove one source of waiting before adding another feature.

The model is not the product. The waiting experience around the model is part of the product, and it deserves the same design attention as the answer itself.

FAQ

What is AI latency?

AI latency is the time between a user action and a useful response from an AI system. For LLM applications, measure time to first token, time to first useful output, and total completion time because each affects the experience differently.

How can I reduce LLM latency?

Reduce prompt size, limit retrieval scope, cache stable data, run independent tool calls in parallel, use smaller models for simple tasks, and set clear timeouts. You can also reduce perceived latency by streaming useful output and moving long work into background jobs.

Does streaming make an AI model faster?

No. Streaming usually does not reduce the model's total compute time. It reduces the silence before the user sees output, which can make the product feel faster when the streamed content is useful and clearly marked as provisional.

What is a good latency target for an AI application?

There is no single target for every workflow. Aim to acknowledge the user immediately, show meaningful progress within about two seconds, and define a separate completion target for each job type. A report may take 30 seconds, while a classification or autocomplete action should feel much faster.

Should every AI workflow be real time?

No. Use real-time interaction when the user is actively making a decision or refining a request. Use a queued background job when the work involves several tools, long documents, batch processing, or a completion time that users do not need to watch.

Pick one workflow this week and draw its full timeline from click to completed result. Add timestamps for queueing, retrieval, model calls, tool calls, retries, and validation. Then redesign the first ten seconds so the user gets a useful signal before the slowest step finishes.