Measure before you optimize: the honest way to cut LLM spend

At some point every AI team gets the same instruction: cut the model spend. The common shortcut is to switch to a cheaper model and see what breaks. Occasionally that works. Without measurement it is a coin flip with a budget attached, and the coin flip usually costs more than the instrumentation would have.

Instrument the request, not the invoice

The invoice gives you a total. It cannot tell you why the total is what it is. The unit that can be optimized is the individual request path, decomposed:

• Prompt weight: how many tokens, from which components (system prompt, retrieved context, conversation history, tool results)?
• Retry and fan-out: how many model calls per user action, counting hidden retries and parallel calls?
• Stage latency: where does wall-clock time go (network, provider queue, model, your own post-processing)?
• Cache behavior: what fraction of tokens repeat across requests that could share them?

None of this needs elaborate tooling. It is structured logging at boundaries the system already has, aggregated per feature. What teams find is rarely what they expected: the model choice turns out to be fine, and the real cost lives in duplicated retrieved context or a retry loop nobody knew existed.

The levers, roughly in order of impact

Context engineering

The least glamorous and usually the largest win. Deduplicate retrieved content, trim history to what the task needs, move static instructions out of per-request context so prompt caching can hold them, and send passages instead of whole documents.

Hidden fan-out

One user action triggering five model calls is five times the cost. Batch where you can, merge prompts where quality allows, and make every retry a deliberate decision instead of a default.

Caching, both kinds

Deterministic caching for identical requests. Semantic caching where near-duplicates are genuinely interchangeable, which is a product decision to be validated by evaluation rather than an infrastructure default.

Model routing

Right-size the model per task: fast models for classification and extraction, stronger models for reasoning-heavy steps. A quality-gated router beats one expensive model for the whole pipeline, but the gating requires the evaluation suite from day one.

Interaction design

Streaming, progressive disclosure, and deferring expensive work until it is needed. Perceived latency is a product property as much as an infrastructure one.

The rule that makes optimization safe

Every lever above can degrade quality, and ungated optimization is how a cheaper product becomes a worse product. One rule prevents it:

No optimization ships without passing the same evaluation used for releases.

If that evaluation does not exist yet, build it first. It is smaller than it looks from the outside, and it is what makes every cost decision after it cheap.

A first pass, concretely

1. Attribute cost and latency per request stage over two weeks of real traffic.
2. Rank findings by absolute dollars and by user-visible latency.
3. Fix the top context-weight offenders; deduplication and trimming usually lead.
4. Add routing for one clearly separable task, quality-gated.
5. Re-measure and report before/after on the same evaluation set.

Run it that way and the before/after numbers survive scrutiny, which is what separates cost engineering from cost theater.