The real cost of an AI feature: instrument AI cost before it surprises you

At 10:47 p.m., Priya sat at her kitchen table with a laptop, a cold cup of tea, and a cloud invoice that had climbed from $180 to $2,940 in a single month. Marcus, the engineer who had bolted on the support summarizer, kept refreshing the dashboard because the feature was supposed to save time, not torch the budget.

The real AI cost is the entire path from user request to delivered result, not just the token price on the model page. You need LLM cost telemetry tied to user actions, retries, latency, storage, and business outcomes before you can understand AI unit economics.

That sounds obvious. It is not how most teams build.

Priya and Marcus are fictional, but the moment is real. A feature works in staging. Users like it. The team watches request volume and celebrates adoption. Then finance asks why the bill rose faster than revenue, and the product team discovers that "one request" was really six model calls, two retrieval operations, a moderation check, a retry, and a background job that nobody had wired back to the original request.

I've watched this pattern across large technical programs and the AI products I've shipped. The surprise rarely comes from one outrageous call. It comes from a quiet chain of reasonable decisions that nobody measured as one system.

AI cost is a product metric, not a vendor invoice

The default advice is to reduce token usage. That helps, but it's a narrow answer to a larger problem. A cheaper prompt can still produce terrible AI unit economics if the system retries too often, sends requests to the wrong model, or generates work that users never actually finish.

Think of the feature as an assembly line. The model is one station. The line also has input preparation, retrieval, safety checks, output validation, persistence, retries, and delivery. If you only measure the model station, you don't know what the product costs.

For every completed user task, measure at least these components:

• Input and output tokens for every model call.
• Model name, version, region, and request type.
• Retrieval, embedding, reranking, and vector database charges.
• Retry count, timeout count, and fallback model usage.
• Compute, queue, database, object storage, and observability costs.
• Revenue, subscription tier, conversion, or another business result tied to the task.

OpenAI, Anthropic, and Google publish model pricing in token-based units, but the invoice usually doesn't tell you which customer workflow created the spend. Your application has to provide that connection.

That means a request ID is not enough. You need a trace that follows the request across services and model calls. A trace can show that a "draft email" action used Claude Sonnet for planning, GPT-4o for generation, an embedding model for account context, and a second generation attempt after a JSON parsing failure.

The user sees one draft. Your system sees a small supply chain.

What should count as one AI unit?

Define a unit around a user outcome, not an API call. For a support product, one unit might be "one resolved customer issue." For a writing tool, it might be "one accepted draft." For an internal research assistant, it could be "one completed brief delivered to a team member."

Then calculate the fully loaded cost of that unit:

• AI unit cost = total feature cost divided by completed units.
• Model cost per unit = all model charges divided by completed units.
• Waste rate = failed, abandoned, or duplicate work divided by all initiated work.
• Contribution margin = revenue per unit minus fully loaded cost per unit.

You don't need perfect accounting on day one. You need a definition that stays stable long enough to reveal movement.

At Zentoko, my multi-model AI marketing platform, the useful question was never "What did this call cost?" It was "What did it cost to produce an approved piece of work that someone could actually use?" Those are very different numbers.

Build LLM cost telemetry around the request path

LLM cost telemetry is the accounting layer your AI feature doesn't get from the provider automatically. It should let you answer a plain question: what happened to this request, and what did each part cost?

Start with a common event record. You can store it in Postgres, pipe it through OpenTelemetry, or use a vendor like Langfuse or Helicone, but the fields need to belong to your product rather than only to the provider.

A useful model-call event includes:

• `request_id` and `trace_id`.
• `user_id` or an anonymized account ID.
• `feature_name` and `workflow_step`.
• `model_name` and provider.
• Input tokens, output tokens, cached tokens, and estimated cost.
• Start time, end time, status, timeout, and error type.
• Retry number and fallback status.
• Prompt or template version.
• Input and output safety result.
• User outcome, such as accepted, edited, abandoned, or failed.

Do not log raw prompts by default. They may contain customer data, private documents, or secrets. Log hashes, template versions, token counts, and selected metadata unless you have a clear reason and retention policy for the content itself.

The event record should be created at the application boundary, where you know what the user was trying to accomplish. Provider dashboards are useful for reconciliation. They are not your product ledger.

And instrument the paths that don't look like model calls. A failed JSON parse may trigger a retry. A timeout may trigger a fallback. A retrieval miss may force a larger context window on the next attempt. A user who clicks refresh three times may create three separate billable workflows.

Here is the minimum dashboard I'd want before launch:

• Cost per completed unit by feature and customer tier.
• Cost per successful request, including failed attempts.
• Median and p95 cost, not only the average.
• Retry rate and fallback rate by error type.
• Token volume by prompt version and model.
• Abandonment rate after a model response.
• Gross margin by customer or usage tier.

The average can lie politely. A handful of accounts with long documents may produce a p95 cost that makes your low-volume pilot impossible to price later. (I've seen this exact thing happen, and it's a much worse discovery at Series A than at seed.)

Find the hidden costs before they become a pricing problem

The mistake I keep watching teams make is treating model spend as the whole bill. Model spend is visible, familiar, and easy to discuss. The hidden costs sit in the seams between services.

Consider a support assistant that has to answer a refund question on a Saturday night with no human in the loop. One user message might touch this path:

1. The application classifies the request.

2. A retrieval service searches Postgres with pgvector.

3. The system reranks the results.

4. A safety model checks the input.

5. A generation model writes the answer.

6. A validator rejects malformed output.

7. The application retries with a larger prompt.

8. The answer is stored and sent to an analytics queue.

The customer sees one answer. Your cost ledger should see eight steps.

Common sources of surprise include:

• Context growth from stale conversation history.
• Duplicate retrieval caused by separate services each building their own context.
• Background summarization after every turn instead of at a defined threshold.
• Retry storms during provider latency spikes.
• High-cost models used for classification or simple extraction.
• Image, audio, or document processing added to the workflow later.
• Logging and tracing that retain large payloads for too long.
• Human review triggered by low confidence or safety rules.
• Free users who discover the most expensive workflow.

Whisper, for example, can make voice input practical, but transcription is another billable step before the language model sees a single word. A document workflow may also use OCR, page rendering, embeddings, retrieval, and storage. The feature name stays the same while the cost path quietly expands.

Once you frame it this way, pricing becomes less mysterious. You can ask whether the expensive step improves completion, reduces support load, or lifts retention. If it does none of those things, it isn't a product investment. It's an unattended process.

How do retries change AI cost?

Retries multiply cost and often erode trust at the same time. A request that fails after consuming 70 percent of its normal tokens may still be charged, and a fallback request can double the total before the user sees an answer.

Track retries by cause, not only by count. A malformed tool call points to an output contract problem. A timeout points to latency or provider capacity. A content refusal points to routing or prompt design. Those require different fixes.

Set a retry budget for each workflow. A low-value autocomplete might get one short retry, while a paid report can justify a slower fallback. Don't give every path the same automatic recovery behavior, that's how a small error becomes a large invoice.

Use idempotency tokens so a browser refresh doesn't silently launch a second paid job. Add circuit breakers for provider failures. Put hard limits on input size and conversation history. These controls aren't hostile to users. They protect the service from behaving unpredictably when conditions shift.

Turn the measurements into operating decisions

Instrumentation only has value when someone acts on it. A dashboard nobody checks is a very expensive decoration.

Set a weekly review for the first version of the feature. Include product, engineering, finance, and someone who actually understands customer behavior. Keep the review tied to decisions, not theater.

Ask questions like:

• Which workflow has the highest cost per completed outcome?
• Which customer tier has negative contribution margin?
• What percentage of spend comes from retries and abandoned work?
• Which prompt version increased tokens without improving acceptance?
• Which model step can move to a smaller or faster model?
• Is a longer context producing better outcomes, or only longer bills?

Create a Cost Telemetry contract for every new AI feature. The contract should name the unit, list every model and service involved, define the success event, and state the budget limit. Make it part of the launch checklist, alongside access control and failure handling.

Then build alerts around behavior rather than only dollars. A daily spend alert arrives after the damage. A notification that retry rate climbed from 3 percent to 12 percent gives you a window to intervene while the cause is still traceable.

Useful alert thresholds include:

• Cost per completed unit rises by 20 percent week over week.
• p95 cost exceeds twice the median.
• Retry rate crosses a workflow-specific limit.
• Output acceptance falls below its normal range.
• A new prompt version increases input tokens by 30 percent.
• One account creates an unusual share of total usage.

Don't rush to optimize every token. Start with the largest source of waste that actually affects the customer experience. A 40 percent reduction in failed retries is often worth more than a 5 percent reduction in prompt length.

I've shipped enough systems to know the first dashboard will be wrong in places. That's fine. The goal isn't accounting perfection, it's making the invisible path visible enough that a team can learn before the bill teaches them the hard way.

What should you do before launch?

Before you release an AI feature, run one realistic workflow end to end. Use a long document, a slow provider response, a malformed tool result, a repeated browser click, and an abandoned session. Then inspect every event created by that journey.

Write down the cost of the successful path and the cost of the failed path. Compare both with the price you expect to charge and the value the customer actually receives. If you can't explain the difference, the feature isn't ready for a broad audience.

A practical pre-launch check looks like this:

• Name the completed user outcome.
• Trace every paid operation to that outcome.
• Record tokens, latency, retries, and errors.
• Set limits for context, retries, and usage.
• Test p50, p95, and worst realistic cases.
• Assign an owner for the weekly review.
• Add an alert before opening access widely.

You're not behind if you haven't done this yet. Most teams discover cost instrumentation only after a surprise. The important move is to stop treating that surprise as a finance problem and start treating it as a product design problem.

FAQ

What is the real cost of an AI feature?

The real cost includes model calls, input and output tokens, retries, fallback models, retrieval, embeddings, storage, compute, observability, moderation, human review, and support. Divide those costs by a completed user outcome rather than by raw API calls to get a useful AI unit cost.

What is LLM cost telemetry?

LLM cost telemetry is request-level data that connects model usage to a feature, user, workflow step, outcome, and estimated charge. It should include tokens, model name, latency, errors, retries, fallback behavior, and prompt version so your team can explain both spend and quality.

How can a startup control AI cost without hurting quality?

Measure cost per successful outcome first. Then reduce failed retries, cap context growth, route simple steps to smaller models, cache stable work, and test prompt changes against an evaluation set before releasing them.

When should AI unit economics be measured?

Measure them before launch, even when traffic is small. Early numbers will be imperfect, but they expose the cost path and help you set sensible usage limits, pricing, and margin targets before customer behavior makes the system expensive to change.

If you're staring at an unexpected AI bill, start with one request and follow it through the entire assembly line. Name every step. Price every step. Connect the total to something a customer actually finished.

The model is only one station. Your product is the whole line, and you can make the line understandable before it starts running faster than your budget.