The first week after launching an AI product: what broke, what surprised me, and what I fixed

At 7:14 on a Monday morning, I was standing in my kitchen with one hand on a coffee mug and the other refreshing a production log. The AI product had launched over the weekend. The demo had worked. Real users had immediately found the parts I had not tested.

The first week after launching an AI product is not a victory lap. It is a systems inspection. My AI product launch lessons were simple: the model was rarely the only thing that broke, user behavior was less predictable than any test set, and the fastest fixes came from better evidence rather than better prompts.

I have shipped technical systems at Verizon, HP, and InterContinental Hotels, and I now run live multi-model products. The pattern is familiar. Production does not politely confirm your assumptions. It removes them.

The launch plan ended when real users arrived

Before launch, I had a checklist. Authentication worked. Billing worked. The main AI workflow returned an answer. The application handled a model timeout in my local tests. I had even tried several awkward prompts, which is the software equivalent of checking the front door while forgetting the basement.

Then a user in Austin submitted three requests within a minute. The first completed. The second timed out. The third appeared to hang because the browser had lost its connection while the server continued processing. From the user's point of view, the product had failed twice. From my server's point of view, it had completed one request and was still working on another.

That distinction matters. An AI product has at least two states: what the backend is doing and what the user believes is happening. Those states can separate through a timeout, a dropped browser connection, a slow provider response, or a retry button pressed three times.

Here's the part most people miss. The conventional advice is to launch a minimum viable product and learn from users. That is sensible for a static workflow. For a generative AI product, the first release also needs a minimum viable operating model. You need to know which request was made, which model answered, how long it took, what it cost, and what the user saw.

The first things I checked were not model quality scores. They were:

• Request volume by hour and by workflow.
• Completion, timeout, and cancellation rates.
• Median and p95 latency, not just the average.
• Provider errors and application errors separately.
• Duplicate requests from repeated clicks or retries.
• Whether a user received an answer after the browser had disconnected.

This became my first production habit: do not call a request successful because the model returned text. Call it successful only when the user received a usable result and the system recorded what happened.

The distinction sounds obvious after the fact. It was not obvious at 7:14 on Monday morning.

What broke first was not the model

The first visible failure was a timeout. The deeper failure was that the timeout path had no clear owner.

My application called an AI provider, waited for a response, and returned the result to the browser. That path looked clean in a diagram. In production, the provider took longer on a large request, the browser timed out sooner, and a retry created a second provider call. One user action had become two paid calls.

I found the duplicate pattern in application logs and confirmed it through provider usage records. The system had no idempotency token, so it could not tell whether the retry represented a new task or the same task arriving again. The fix was not a clever prompt. I created a request record before calling the model, assigned it a unique identifier, and made repeated submissions return the existing job state.

I also changed the user experience. A long request now showed a definite processing state, a request identifier, and a clear message when the browser could safely reconnect. If the task failed, the user saw a retry action that reused the original request where appropriate.

That small change reduced ambiguity across the whole stack. Support could search for the request. I could trace it through Postgres, the queue, and the provider log. The user no longer had to guess whether clicking again would help or make the problem more expensive.

Google's Site Reliability Engineering guidance treats an error budget as a way to balance reliability and release speed. I applied the same idea at a smaller scale. I set practical thresholds for failed requests and p95 latency, then paused feature work when those thresholds were crossed.

Here is what I now inspect for every production failure:

• Was the request accepted by my application?
• Was it sent to the provider once or more than once?
• Did the provider return an error, or did my code mishandle a valid response?
• Did the user receive a result, an error, or no visible state at all?
• Did the system charge for work that the user never received?

The mistake I keep watching teams make is treating retries as harmless. A retry is a new product event, a new cost event, and sometimes a new data event. Instrument it before you tune anything else.

The surprising failures came from reasonable user behavior

A product team can imagine unusual users. It is harder to imagine ordinary users doing ordinary things in a way the system never saw during testing.

On Tuesday afternoon, a user pasted a full email thread into a field designed for a short description. The request was valid. The input was useful. It exceeded the Context Budget once the system instructions, retrieved records, and output allowance were included.

The model did not crash. The application received a response that was truncated halfway through a sentence. My test expected either a complete answer or an error. Production gave me a third state: a plausible beginning that looked finished until someone read it carefully.

I added input length checks, a visible warning, and a server-side token estimate before the provider call. I also changed the workflow so the system could summarize a long source before using it for the final task. That introduced another check: the summary itself needed an evaluation, because compression can remove the one detail that makes the answer correct.

Another surprise came from a user who pasted a spreadsheet column containing blank rows, duplicate records, and a formula error. The model followed the data well enough to produce a polished answer, but the answer inherited the spreadsheet's bad assumptions. The output sounded confident because the source looked structured.

This is why I do not treat user feedback as a list of feature requests. Feedback is evidence about the boundaries of the system. "The answer was weird" can mean a context overflow, an ambiguous input, a missing record, a rendering problem, or a valid answer that violated the user's expectation.

During the first week, I classified reports into four groups:

• Bad input: the user supplied something the workflow could not safely interpret.
• Bad retrieval: the system selected incomplete or stale information.
• Bad generation: the model produced an incorrect or unsupported answer.
• Bad product behavior: the answer may have been fine, but the interface hid the state or made the next action unclear.

That classification changed my fixes. I stopped rewriting prompts for every complaint. Sometimes the right answer was a file limit, a confirmation step, better retrieval filters, or a clearer empty state.

NIST's AI Risk Management Framework, published in 2023, puts measurement and context around AI risks rather than treating model output as the entire problem. The first week made that practical for me. Reliability is partly model behavior, but it is also input handling, state management, data quality, and user understanding.

The eval loop had to include real failures

Before launch, I had a test set made from expected use cases. It included normal requests, a few difficult requests, and some obvious failure cases. It did not include the strange but legitimate inputs that arrived during the first week.

So I built a small Eval Loop from production evidence. Every confirmed failure became a test case with the original input, the expected behavior, the observed output, and a label explaining the failure. I removed personal information before storing examples and kept the set versioned with the application.

This was not a large research project. A spreadsheet and a short script were enough to start. The script sent each case through the current workflow and recorded whether the result met the expected standard. For cases where there was no single correct answer, I checked specific criteria: factual support, required fields, refusal behavior, and whether the answer stopped when the source was insufficient.

One case came from a user in Denver who asked for a recommendation based on an old account record. The system correctly retrieved the record but failed to mention that it was six months old. A prompt change could have helped, but the stronger fix was to pass the record date as explicit metadata and require the answer to disclose stale information.

Another case came from Zentoko, my multi-brand AI marketing platform. A brand request used a tone instruction that conflicted with a shared system rule. The output was not unsafe, but it was inconsistent. I separated global policy from brand-specific instructions and added a test showing which instruction should win when the two disagree.

The first week also revealed that output grading needed more than one score. I tracked:

• Task completion: did the answer perform the requested job?
• Evidence support: can each important claim be traced to supplied data?
• Format compliance: did the result contain the fields and structure required by the workflow?
• User recovery: could the user understand and correct a failure?

I reran the eval set after each change. A prompt that improved one example but damaged three others did not ship.

The default startup advice is to ship the MVP fast. For AI products, that advice is incomplete. Ship the smallest useful product, but build the smallest repeatable test loop beside it. Otherwise every fix is a guess, and guesses compound quickly. And yes, "build the eval loop first" sounds obvious until you are three days post-launch with six support tickets open and no way to tell which prompt change actually helped.

The cost surprise was caused by workflow shape

I had estimated the cost of one model call. That was not the cost of a user completing a task.

A real workflow could include a classification call, a retrieval step, a generation call, a safety check, and a retry. A long input could increase token usage at every stage. A user could edit the request and run it again. The unit that mattered was not "one completion." It was "one successful job."

I added cost telemetry at the request level. Each job recorded the provider, model, input tokens, output tokens, latency, retry count, and estimated cost. I grouped those records by workflow so I could compare the cheap-looking path with the path users actually followed.

The surprising result was not that one model cost more. It was that failed jobs consumed meaningful spend. A timeout followed by a retry could cost more than a successful job. A validation step that rejected an answer after generation still incurred the generation cost. A large context sent to a fast model could be more expensive than a shorter request sent to a slower model.

This is where a context budget became useful. I set a limit for the total information a workflow could send to the model, then assigned room for instructions, retrieved material, conversation history, and the response. The budget was not only about provider limits. It was about predictable cost and stable response quality.

I also added three operational controls:

• A per-user rate limit for repeated requests.
• A maximum retry count with a visible failure state.
• A spend alert based on daily workflow cost, not only total account usage.

Teams often optimize prompt token cost to the third decimal and then absorb a large bill from a retry storm nobody instrumented. I have made that mistake in less dramatic forms. The fix is to measure the full path from click to completed job.

Do not wait for finance to explain your AI bill. Make cost part of the product record while the volume is still small enough to understand.

The fixes that mattered were mostly boring

By Friday, I had a list of ambitious improvements. I wanted better model routing, richer memory, more polished streaming, and a larger evaluation set. Some of those changes were useful. None mattered as much as the boring repairs.

I added structured logs with a correlation identifier. I wrote clearer error messages. I made empty results distinct from failed results. I stopped exposing raw provider errors in the interface. I created a simple support view that showed the request status without requiring a database query by hand.

I also wrote a short launch-week runbook. It answered practical questions:

• Where do I look when a user says the product is stuck?
• How do I identify duplicate calls?
• Which provider errors are temporary?
• When should I refund or credit a failed task?
• How do I disable one workflow without taking down the entire application?

A runbook is a load-bearing wall. It does not attract attention when everything works, but it keeps one small incident from becoming a long night. My previous program work taught me that ownership must be visible before the incident, not negotiated during it.

I also made one product decision that felt uncomfortable: I removed a workflow that looked impressive but failed too often on messy inputs. It produced excellent results in a controlled demo. In real use, it encouraged users to trust an answer before the system had enough evidence.

That is an important launch lesson. Removing a weak path is not giving up on the product. It is reducing the blast radius while you learn what the product can reliably do.

The strongest signal from the first week was not a user compliment or a dashboard spike. It was the growing list of cases I could explain. Once I knew why a request failed, I could choose between a code fix, a data fix, a model change, or a product change.

What I'm telling you is simple: production maturity begins when failures become legible.

What to do during your next launch week

If you are launching an AI product, do not fill the first week with roadmap meetings. Reserve time for direct observation. Watch a real user complete a task. Read the logs beside the support messages. Compare the system state with the user's understanding.

Use this sequence:

• Create a daily list of failed, slow, expensive, and confusing requests.
• Reproduce the highest-impact case before changing the prompt.
• Label the failure as input, retrieval, generation, cost, state, or interface.
• Add the case to the eval loop.
• Ship the smallest fix that prevents the same class of failure.
• Rerun the full evaluation set and inspect cost and latency.
• Write down the new operating rule in the runbook.

Do not judge the launch only by signups. Track activation, successful job completion, repeat usage, p95 latency, failed job rate, average cost per completed job, and support contacts per active user. Those numbers tell you whether the product is becoming dependable or merely attracting attention.

Give yourself permission to make the product narrower. A reliable workflow for one specific job is more useful than a broad assistant that produces uncertain answers across ten jobs. The first week is where you discover that boundary.

And keep a change log. Record the date, the observed failure, the fix, and the result. Six weeks later, you will not remember which prompt change helped or which one only felt productive. Future you will be grateful, future you is often a poor historian.

The model is only one component of what you launched. The product is the path from user intent to a trustworthy result, including what happens when that path breaks.

Start tomorrow by pulling the last 20 production requests. Classify each one, calculate the cost per completed job, and turn the worst failure into a test. That is how shipping generative AI becomes an operating practice rather than a launch event.

FAQ

What should I monitor during the first week after launching an AI product?

Monitor successful job completion, p95 latency, provider errors, application errors, retry count, duplicate requests, cost per completed job, and user-visible failures. Track these by workflow rather than only across the whole application, because one failing path can disappear inside a healthy average.

What is the most common early failure in a production AI product?

The most common early failure is usually the boundary between system state and user expectation. A request may continue running after the browser times out, or a retry may create a duplicate paid call. Request identifiers, idempotency handling, clear status states, and structured logs address this class of failure.

Should I change the prompt when users report bad AI answers?

Not automatically. Classify the report first as an input, retrieval, generation, cost, state, or interface problem. A prompt change is useful when the model misunderstood a valid instruction, but it will not repair stale data, missing context, duplicate calls, or a confusing error state.

How do I measure AI product quality after launch?

Build a test set from real production failures and rerun it after every meaningful change. Measure task completion, evidence support, format compliance, refusal behavior, and whether users can recover when the system cannot complete the task.

When should I remove a feature after launch?

Remove or narrow a feature when it fails often enough to mislead users, consume disproportionate cost, or create support work that you cannot explain. A smaller workflow with known limits gives you better evidence than a broad feature that appears impressive but behaves inconsistently.