The startup metric almost nobody instruments: retry storms

At 2:13 a.m., Sarah refreshed the support dashboard from her kitchen table. One customer question had failed, then failed again, and then the worker queue began feeding the same request back into the system like a waiter returning to the same table with no order. A retry storm can turn a small provider hiccup into a serious reliability and cost problem, and most startups don't see it until the bill arrives.

I've shipped production systems at large companies and built two live multi-model AI platforms. The pattern is consistent: teams instrument request latency, model errors, and token usage, but they leave retries in the dark. That missing signal is where reliability work quietly becomes a margin problem.

A retry storm is a multiplication problem

The basic story sounds harmless. Your application calls a model. The request times out. Your code tries again. Reasonable. A retry gives a temporary network fault a chance to recover without making the user stare at an error message.

The trouble begins when several layers retry the same operation. The browser may retry. Your API may retry. A queue worker may retry. The model gateway may retry. One customer action can become eight or sixteen provider calls before anyone notices.

Here's the part most people miss: a retry storm is not just a high error rate, but a feedback loop where failure creates more traffic, more traffic creates more delay, and more delay creates more failure.

A simple example:

• One request costs $0.02.
• A timeout triggers three application retries.
• A queue redelivers the job twice.
• The same request now costs as much as $0.12 before a customer sees an answer.

At 10,000 requests, that difference is $1,000. At 100,000 requests, it is $10,000. The exact numbers vary by model and prompt size, but the shape of the problem does not.

Teams often treat retries as a reliability setting. That is too narrow. Retries affect:

• AI cost per successful user outcome
• Provider rate limits
• Queue depth and worker saturation
• Response latency
• Duplicate side effects, such as repeated emails or database writes
• Customer trust when answers arrive late or arrive twice

I learned this before AI systems became the center of the conversation. In large technical programs, a service can appear healthy while the recovery path quietly consumes its capacity. The main request path gets the dashboard. The exception path gets a log file nobody opens until something is on fire.

And AI systems make the problem less forgiving. A model request may carry a long system prompt, retrieved documents, conversation history, and an image. Retrying that request repeats the full workload. You are not always repeating a cheap ping.

Why do AI applications retry so often?

AI applications retry because model providers, network connections, queues, and internal services can all return temporary failures. Engineers add retries to protect the user from those failures, but they often configure each layer without seeing the total retry count.

Common triggers include:

• HTTP 429 responses caused by rate limits
• HTTP 500 and 503 responses from a provider
• Client-side timeouts set below the model's actual response time
• Broken streaming connections
• Queue visibility timeouts that expire while a model is still working
• JSON parsing failures after a model returns malformed structured output

Not every failure deserves another attempt. A malformed input, rejected policy request, or invalid tool call may fail the same way every single time. Retrying it only increases cost and delays the real fix.

The metric you need is not retry count

A dashboard that says "12,000 retries" is better than silence, but it does not tell you whether the system recovered or simply burned money. You need to connect retries to outcomes.

I call this the Retry Budget. It is the maximum extra work you are willing to spend recovering one user request. The budget can be expressed as a count, a time limit, or a cost ceiling.

For example, a team might decide:

• No more than two retries for a transient provider error
• No more than 20 seconds of total recovery time
• No more than three model calls for one user action
• No retry after a validation error or policy rejection

The important part is not the exact number. It is making the decision visible before an incident forces you to make it at 3 a.m.

Track these fields for every model operation:

• Request ID and parent user action ID
• Model provider and model name
• Attempt number
• Error class and HTTP status
• Start and end time for every attempt
• Input and output token counts
• Estimated cost per attempt
• Final outcome: success, failure, fallback, or abandoned

The parent action ID matters more than most teams expect. Without it, your logs show ten separate requests. With it, you can see that ten requests came from one button click.

Your core dashboard should answer five questions quickly:

• What percentage of successful responses required a retry?
• How many total model calls came from each user action?
• Which error types create the most repeated work?
• What is the cost of retries as a percentage of total model cost?
• How long does a retried request take compared with a first-attempt success?

That is observability with a purpose. You are not collecting every possible event because dashboards look impressive. You are tracing the path from a user action to the final result.

The conventional advice says to retry transient errors automatically. That advice is incomplete. A transient error is only safe to retry when you have bounded the number of attempts, added delay, and confirmed that repeating the operation cannot cause a harmful duplicate.

The mistake I keep watching teams make is tuning the prompt before measuring the recovery path. They spend an afternoon shaving 200 tokens from a request, then allow the same request to run six times during a provider slowdown. The arithmetic is not subtle.

Build observability around the failure path

Start with a single request trace. Not a giant platform project. Pick one real workflow, like a support bot answering a refund question on a Saturday night with no human in the loop.

Follow that workflow through:

• The browser or client request
• Your API endpoint
• The orchestration service
• The model provider
• Any retrieval or tool call
• The queue and worker, if present
• The response returned to the customer

For each hop, record whether the operation was attempted once or repeated. A trace should make it obvious when a failed provider call led to a second model call, a repeated retrieval query, and a queue redelivery.

Then add Cost Telemetry. This is not a finance report. It is request-level information that connects model usage to a product event. When a team can see that one failed checkout explanation generated five calls to a large model, engineers can fix the mechanism instead of debating the monthly invoice.

Cost Telemetry should include:

• Cost per attempt
• Cost per successful response
• Cost per user action
• Retry cost by provider and model
• Retry cost by endpoint or workflow
• Cost from abandoned requests

A useful ratio is simple:

retry cost rate = retry spend / total model spend

Track it by hour during incidents and by week during normal operation. A low weekly average can hide a damaging 45-minute event, and the average is never the whole story.

You also need to separate a retry storm from a healthy retry pattern. A small number of second attempts may be a sensible trade for better completion rates. The warning signs are different:

• Retry volume rises faster than user traffic
• Average attempts per request climbs above its normal range
• Queue depth grows while successful responses fall
• Provider 429 responses increase after your retries begin
• Token spend rises without a matching increase in completed actions
• Several services report errors for the same request ID

Set alerts on ratios, not only totals. A ten percent retry rate might be normal for one workflow and alarming for another. Alert when retry cost exceeds a defined share of spend, or when the average number of attempts crosses the retry budget.

(If you've ever watched a clean demo fall apart on the third question, you already know the emotional version of this problem. Production failure feels personal because someone is waiting. Instrumentation gives the team a place to stand before blame takes over.)

Design retries so failure stops spreading

Once you can see the pattern, change the mechanics. Keep the first half of the system calm. The recovery path should not behave like a second traffic source.

Use exponential backoff with jitter. Exponential backoff increases the wait between attempts. Jitter adds a random variation so thousands of workers do not retry at the same instant. A basic schedule might wait 500 milliseconds, then 1 second, then 2 seconds, with a small random offset at each step.

Cap the delay and the attempt count. A retry policy without a ceiling is a request for an incident. The ceiling should be explicit in code and visible in telemetry.

Classify failures before retrying:

• Retry temporary capacity and network failures.
• Retry a stream disconnect only when the operation can safely resume or restart.
• Do not retry invalid requests.
• Do not retry a policy rejection.
• Do not retry malformed output without recording why the output failed validation.
• Route persistent provider failures to a bounded fallback or a clear user message.

Be careful with fallback models. A fallback can protect availability, but it can also double your spend if the primary request continues running in the background. Cancel work when cancellation is supported. Otherwise, measure both calls and count the outcome honestly.

Use idempotency for operations with side effects. If an AI agent can send a message, create a ticket, or update a record, assign one operation ID and make the downstream action safe to repeat. A successful model response followed by a lost network connection is not the same as a failed operation.

This is where many AI teams discover that their retry code is not really retry code. It is a second execution path with different timeouts, different logging, and no owner. Treat it like production code, because it is production code.

For queue workers, align the visibility timeout with the longest expected model operation plus a safety margin. If a worker takes 90 seconds to process a request but the queue makes the message visible again after 60 seconds, another worker may start the same job. That is a duplicate execution problem wearing a queue configuration costume.

And do not let every layer own a retry policy. Choose one layer to coordinate retries for each operation. Other layers should either pass the error through or use a very small, documented transport retry. The goal is not zero retries. The goal is one understandable recovery plan.

Make retry storms part of product planning

Retry storms are usually treated as infrastructure incidents, but they reach the product quickly. A customer who waits 40 seconds for an answer does not care whether the delay came from a timeout, queue redelivery, or provider rate limit. They experience one broken promise.

The product decision is often simple: what should happen when recovery takes too long? You might return a partial answer, switch to a smaller model, save the request for later, or tell the user that the operation did not complete. A clear failure is often kinder than a hidden loop.

Write the behavior down for each important workflow:

• Maximum total wait time
• Maximum model attempts
• Allowed fallback models
• Whether partial output is safe to show
• Whether a user can safely press the button again
• What support staff can see when a request fails

Run a controlled test before a real incident. Add a provider timeout in a staging environment. Return 429 responses from a mock gateway. Delay a stream. Let a queue message approach its visibility limit. Then watch what your system does.

Measure more than whether the final request succeeds. Record total attempts, total tokens, total cost, queue age, and user-visible time. A test that ends with a green status but spends five times the normal amount is not a successful reliability test.

In the multi-model platforms I've built, the hard part has rarely been choosing another model. The hard part has been making the system explain why it moved from one model to another, why it tried again, and who paid for the extra work. That explanation is what lets a small team act quickly.

So the deeper lesson is this: AI reliability is not only about getting an answer. It is about controlling the work performed while the system is trying to get one.

Start with one endpoint this week. Add an attempt number, a parent action ID, error classification, and cost per attempt. Set one alert for retry cost and one alert for attempts per successful action. Then force a timeout and read the trace like a customer would.

You do not need a perfect observability platform to begin. You need enough visibility to tell the difference between recovery and repetition. The model is only one part of the product. The recovery path is where your reliability and cost discipline become real.

FAQ

What is a retry storm in an AI application?

A retry storm happens when failed or slow AI requests trigger repeated calls faster than the system can recover. Multiple layers may retry the same operation, causing provider load, queue growth, higher latency, and unexpected model cost.

How do I measure retry storm risk?

Track attempts per user action, retry cost as a share of total model cost, retry rate by error type, and the time from the first attempt to the final outcome. Parent action IDs let you connect separate provider calls to one customer event.

Should every AI error be retried?

No. Temporary network failures, rate limits, and some provider capacity errors may deserve a bounded retry with exponential backoff and jitter. Invalid inputs, policy rejections, and persistent validation failures usually need a direct error or a different workflow.

What is a reasonable retry limit for an LLM request?

There is no universal number, but two or three total attempts is a practical starting point for many synchronous workflows. Set the limit with a time and cost ceiling, then adjust it using completion rate, latency, and retry cost from real traffic.

When the dashboard finally reveals a retry storm, you may feel embarrassed that it was there at all. Don't. Almost every team learns this after a queue grows at the wrong hour or a bill lands with an uncomfortable number.

You are not failing because retries exist. You are ready to improve the system when you can see what each retry protects, what it costs, and when it should stop. Make the recovery path visible, then give it a budget.