Per-call cost telemetry: how to stop AI bills from surprising you

At 8:17 on a Monday morning, Sarah opens the cloud billing console in a Chicago office and sees an AI invoice that is 4.2 times higher than last month. Nothing dramatic changed in the product. No viral launch. No obvious outage. Just more retries, longer prompts, and one background workflow firing every six minutes.

That is how AI cost surprises actually arrive. Not as one giant mistake, but as thousands of calls nobody can explain. Per-call cost telemetry stops the surprise by recording the model, token counts, latency, retries, feature, customer, and final cost for every single request. You don't need perfect forecasting on day one. You need a Unit Ledger that traces a bill back to a specific product behavior.

I've watched this pattern repeat while building multi-model AI products and shipping technical programs across large telecom, technology, and hospitality organizations. The model invoice is not the problem by itself. The problem is that most teams can see the total and cannot explain the total.

Start with the economics of one model call

The phrase "LLM cost" sounds like one number. It is not. It's a stack of charges created by a request moving through your application.

A single customer message may trigger a classifier, a retrieval query, a planning call, a tool call, and a final answer. If the final response looks simple, nobody sees the four hidden calls underneath it. Your cloud provider does, and your invoice does.

Start by defining the unit you intend to measure. For most AI features, that unit is a model call. For some products, it's a completed user task. Record both when the distinction matters.

A useful per-call record includes:

• request ID and parent workflow ID
• model provider, model name, region, and API version
• input tokens, cached input tokens, output tokens, and total tokens
• prompt version, tool name, retrieval count, and response status
• latency, retry count, timeout count, and fallback model
• tenant ID, feature name, environment, and estimated dollar cost

The token fields matter because input and output often carry different prices. OpenAI, Anthropic, and Google publish model pricing around token usage, but their APIs do not expose identical fields or naming conventions. A normalized event lets you compare them without pretending their accounting systems are interchangeable.

Suppose a support answer uses 7,800 input tokens and 620 output tokens. A later prompt revision raises the input to 22,000 tokens while the answer stays the same length. The product may feel unchanged, but the economics have shifted hard. Without token telemetry, that change looks like a mysterious monthly increase.

Here's the part most people miss: the conventional advice is to track monthly spend. That helps finance and gives product teams almost no diagnostic information. A monthly total tells you that something happened. A per-call record tells you which workflow, model, prompt, or customer caused it.

So the deeper lesson is simple: treat each model call like a unit moving through an assembly line. You cannot improve yield if every defective unit gets mixed into one end-of-month bucket.

Build a cost event that survives production

Logging the model response is not enough. You need a cost event that stays useful after the request passes through queues, retries, tools, and fallback paths.

I call the schema a Cost Telemetry Event. It should be small enough to emit on every request and rich enough to answer an uncomfortable question from finance or customer success.

The event should answer:

• What did the application ask the model to do?
• Which model actually handled the request?
• How many tokens did that model process?
• What did the request cost at the time it ran?
• Which user, account, feature, and workflow created the charge?
• Was the response successful, retried, cached, or abandoned?

A practical event might look like this:

```json

{

"request_id": "req_8f21",

"workflow_id": "wf_1049",

"provider": "anthropic",

"model": "claude-3-5-sonnet",

"feature": "support_reply",

"input_tokens": 7800,

"output_tokens": 620,

"retry_count": 1,

"latency_ms": 2840,

"estimated_cost_usd": 0.0714,

"tenant_id": "hotel_group_17",

"status": "success"

}

```

Do not calculate cost only inside a dashboard query. Calculate it at ingestion and keep the pricing version used for the estimate. Providers change rates, introduce cached-input pricing, and offer batch discounts. If you overwrite yesterday's event with today's rate, your historical report becomes a fiction with decimal places.

OpenTelemetry is a reasonable transport for this data because it already handles traces, spans, attributes, and correlation IDs. The specific tool matters less than the relationship between a parent workflow and its child model calls. When a request to generate a hotel cancellation reply creates five spans, you need to see all five under the same workflow ID.

This is where many teams make a quiet assembly-line error: they attach cost to the user request, then lose the cost of internal calls. A retrieval step may use embeddings. A safety check may invoke another model. A retry may repeat the full context. Put cost on every call, then roll it up to the user request.

And record failures. A timed-out request can still consume tokens. A response your application discards may still appear on the bill.

Allocate shared cost before it becomes an argument

Some AI cost belongs clearly to one customer. A chat response does. Other cost is shared across tenants: embedding a document collection, refreshing a search index, running evaluations, or generating a cache used by hundreds of accounts.

If you ignore shared cost, your customer margins look better than they are. If you dump all shared cost onto the first customer who triggered a job, your account reporting becomes unfair.

Create allocation rules before the sales team asks for gross margin by account. The rules don't need to be perfect. They need to be stable, visible, and revisable.

Common allocation methods include:

• direct assignment for interactive requests tied to one tenant
• proportional allocation by document count for shared embeddings
• proportional allocation by retrieval volume for a shared index
• platform allocation by active users, task volume, or reserved capacity
• internal allocation to product development for evaluations and experiments

Consider a concrete case. A hotel group in Phoenix uploads 40,000 policy documents, but the embedding job creates an index used by 18 hotel groups. Charging the full embedding bill to Phoenix is wrong. Charging nothing to any account is also wrong. Store the source account, index ID, document count, and retrieval volume, then apply a monthly allocation rule.

The same issue appears with evaluations. When I run a prompt comparison across 2,000 test cases, that cost supports a product feature but not a specific end user. Tag it as product development or research. Do not hide it inside customer support cost just because the request happens to use the same model.

Your cost model should separate at least these buckets:

• interactive user requests
• background automation
• retrieval and embedding infrastructure
• evaluations and quality checks
• failed calls, retries, and provider fallbacks

This is not accounting theater. It changes product decisions. A feature with a 78 percent gross margin may become a 42 percent feature once shared embedding and evaluation work are assigned honestly.

I absorbed this kind of allocation discipline on large technical programs. Projects fail when ownership is vague, and AI products add a volatile input: every user action can spin up a variable number of paid operations. Give each operation a home.

Use telemetry to control the call path

A dashboard that reports yesterday's cost is not cost control. It's a rearview mirror.

The useful question is what your application should do when a request crosses a cost or latency boundary. That requires policies attached to the call path, not just charts attached to a warehouse.

Set thresholds around behavior you can actually change:

• stop a workflow after a defined retry count
• cap retrieved documents before assembling the prompt
• route low-risk classification to a smaller model
• use a shorter context window for repeat requests
• require approval before a background job exceeds its budget
• fall back to a cheaper model when the premium model times out

A customer support bot in Denver may answer a refund question with one model call. A poorly bounded agent may search five collections, call a planner, retry twice, and ask a reviewer to rewrite the response. The user sees one answer. Your bill sees a small committee.

The mistake I keep watching teams make is optimizing the model before they instrument the path. They debate whether a larger model is worth the price while retry storms quietly multiply the request count. Fix the call graph first.

Set alerts around rates, not only totals. A total-cost alert fires too late if traffic rises gradually. Better signals from the assembly line include:

• cost per successful task
• model calls per successful task
• retry rate by provider and feature
• output tokens per completed response
• percentage of requests using fallback models
• p95 cost by tenant and workflow

A cost increase can be healthy if completed tasks increase faster. A lower monthly bill can actually be a warning sign if users are abandoning tasks before completion. (I've seen this exact confusion cause a team to celebrate a number that should've scared them.) Pair AI cost with activation, completion, and quality measures.

An adaptive learning system should not stop at total lesson cost. It should separate the model calls used to generate feedback, assess an answer, and recover from an unclear response. The same principle applies to a marketing workflow, a support assistant, or an internal research tool.

Instrument the control surface before you tune the prompt. Otherwise you are tuning a machine with the hood closed.

Find the hidden causes of an expensive bill

When a bill jumps, resist the urge to open the prompt editor first. Start with a difference report between the last normal period and the first abnormal period.

Break the change across several dimensions:

• model and provider
• feature and workflow ID
• tenant and environment
• input-token band
• output-token band
• retry and timeout status
• prompt version and tool usage

Imagine your March bill rose from $9,400 to $18,700. The top-line number is alarming but not actionable. A useful investigation might find that 61 percent of the increase came from a new document-summary workflow, 24 percent came from repeated timeouts against one provider, and the rest came from a prompt that appended the full conversation history on every turn.

That diagnosis creates separate fixes. Limit the document batch size. Add a timeout and fallback policy. Trim or summarize conversation history. Do not tell the model team to "reduce costs" as if cost were a single code variable.

Pricing data also needs care. Provider rates change. Cached input may be priced differently from uncached input. Batch processing may carry a discount but a slower completion window. Store a pricing table with effective dates, model identifiers, input rates, output rates, cached rates, and currency.

Your estimated per-call cost will not always match the provider invoice exactly. Rounding, taxes, credits, regional fees, and billing adjustments can create a gap, that is acceptable if you monitor the gap and keep the reason visible. A cost estimate is an operational instrument, not a legal invoice.

Here is a useful weekly review question: which workflow has the largest difference between expected and observed cost per successful task? That gap almost always points to retries, hidden tool calls, or a shift in user behavior.

Keep raw events long enough to investigate. Aggregates are cheaper to store, but they erase the path that created the expense. For an early product, 30 to 90 days of raw cost events is often more useful than a sophisticated forecast with no traceability.

Make token telemetry part of product ownership

Per-call cost telemetry fails when it belongs only to finance or platform engineering. The product owner for a feature should be able to answer what the feature costs, what makes it expensive, and what happens when the cost rises.

Put these fields into normal product reviews:

• median and p95 cost per successful task
• token growth after each prompt release
• retry rate after provider or code changes
• cost by customer segment
• quality score alongside cost
• percentage of spend from experiments and evaluations

Use a budget during development. A prototype that spends $200 per day on test traffic can quietly build a $6,000 monthly habit before launch. Give experiments an owner, a time limit, and an automatic stop condition.

I also recommend a cost review whenever you add a model call. The review can take five minutes. Ask what the call does, whether it runs once or in a loop, what context it receives, what happens on failure, and how you will measure cost per completed task.

That sounds almost too basic. Basic is good. Production AI tends to break in the places nobody assigned.

For teams hiring an applied-AI leader, this is a useful interview signal. Ask the candidate to design telemetry for a support agent that can call search, billing, and a refund system. A strong answer will include trace correlation, token counts, retries, tenant attribution, pricing versions, and a quality measure. A weak answer stops at "put the logs in a dashboard."

The practical next step is small: pick one high-volume workflow this week. Emit a normalized event for every model call. Build a report showing cost per successful task, not just total spend. Then inspect the ten most expensive workflows and change one control in the call path.

The model is only one component on the assembly line. The real bill is built by the system around it, and the system is where you still have choices.

FAQ

What is per-call cost telemetry?

Per-call cost telemetry is the practice of recording usage and estimated price for every AI model request. It connects token counts, model choice, retries, latency, customer, feature, and workflow so you can explain total AI cost from the bottom up.

How do you calculate LLM cost per request?

Multiply each token category by its provider price, then add any applicable cached-input, batch, or tool charges. Store the pricing version used for the calculation because model rates and billing categories can change.

Which token fields should an AI product track?

Track input tokens, cached input tokens when available, output tokens, total tokens, and token counts for every model call. Also record the model name, provider, prompt version, workflow ID, retry count, and final status.

How can a small startup control AI cost?

Start with one high-volume workflow and measure cost per successful task. Set retry limits, cap retrieved context, route simple work to lower-cost models, and alert on unusual cost rates before monthly spend becomes the only signal.

Should failed AI calls be included in cost reporting?

Yes. Failed and timed-out calls can consume tokens and create provider charges. Track them separately from successful work so the team can see how much spend comes from reliability problems rather than useful customer outcomes.