Skip to main content
The Hidden Cost of LLM APIs: A Practical Cost Optimization Guide
5 min read
May 3, 2026 (2mo ago)

The Hidden Cost of LLM APIs: A Practical Cost Optimization Guide

Your AI feature works. Your AI bill does not. A field-tested playbook for cutting LLM spend by 50–80% without sacrificing quality — caching, routing, batching, and the prompt patterns that quietly eat your budget.

AILLMFinOps

👋 Introduction

Every founder we have worked with has had the same week-three conversation. The AI feature shipped. Users love it. The first invoice arrives. Suddenly the CFO has questions, and "we'll figure it out later" is no longer a valid roadmap line item.

LLM costs are unusual. Unlike most cloud spend, they scale with engagement rather than infrastructure. The more your product gets used, the more you pay — directly. And unlike compute or storage, the levers to cut the bill are not infrastructure tricks. They are prompt design, model selection, and architectural choices most teams never revisit after launch.

This is the playbook we use when a customer comes to us six months in and says, "the AI works, but the unit economics don't."

First, Measure

You cannot optimize what you do not see. The single highest-leverage thing you can do this week is add per-request cost telemetry. Log, for every LLM call:

  • Model used
  • Input token count
  • Output token count
  • Cached input tokens (more on this below)
  • Latency
  • The feature / endpoint that triggered it
  • The user / tenant ID

That is enough data to find the 80/20. In our experience, one or two endpoints account for 70–90% of the bill. Without telemetry you will optimize the wrong things.

The Five Highest-Leverage Wins

In rough order of how much money they save, here is what actually moves the needle.

1. Prompt caching (often a 90% reduction on cache hits)

This is the single biggest free lunch in the LLM space, and most teams are not using it. Modern providers — Anthropic, OpenAI, and the major open-model hosts — let you mark portions of your prompt as cacheable. The next request that uses the same cached prefix pays a fraction of the input token price.

The economics are stark. Anthropic's prompt caching, for example, charges roughly 10% of normal input cost on cache hits. If your system prompt is 4,000 tokens and you handle a million requests, that is millions of tokens not paid at full price.

Cache anything stable: system prompts, few-shot examples, tool definitions, large reference documents. Put the volatile parts (the user's actual message) at the end of the prompt so the cacheable prefix stays consistent.

// Anthropic SDK example
const response = await anthropic.messages.create({
  model: "claude-sonnet-4-6",
  max_tokens: 1024,
  system: [
    { type: "text", text: stableSystemPrompt, cache_control: { type: "ephemeral" } },
    { type: "text", text: largeReferenceDoc, cache_control: { type: "ephemeral" } },
  ],
  messages: [{ role: "user", content: userQuery }],
});

If you do nothing else from this post, do this.

2. Model routing (often 40–60% reduction)

Not every request needs the most capable model. A good routing layer sends easy requests to a cheap, fast model and reserves the expensive one for genuinely hard work.

A two-tier setup that works well:

  • Tier 1 (cheap): A small model like Claude Haiku or an open model. Handles 70–80% of requests — classification, simple extraction, formatting, short rewrites.
  • Tier 2 (capable): A frontier model. Handles complex reasoning, long-context tasks, and anything Tier 1 returns low-confidence on.

The router can be as simple as a rule-based dispatcher (if input < 500 tokens && task == 'classify' → Haiku) or as fancy as a tiny fine-tuned classifier. Start simple. Most of the savings come from "obvious cases go to the cheap model."

3. Batch APIs for non-real-time workloads (50% reduction)

If the work does not need to happen synchronously — overnight summarization, weekly digest generation, bulk data enrichment — use the batch APIs. Anthropic and OpenAI both offer roughly 50% off for batched requests with a 24-hour SLA.

Things teams routinely move to batch:

  • Embedding pipelines for new content
  • Periodic re-summarization of long-lived documents
  • Trust & safety scans of user content
  • Analytics enrichment of yesterday's events

If a feature can tolerate "results by tomorrow morning," it should not be paying real-time prices.

4. Prompt diet (10–40% reduction)

Audit your prompts. We routinely find:

  • 800-token style guides that could be 200 tokens of crisp examples
  • Redundant instructions repeated three different ways
  • Few-shot examples that have not been re-evaluated since the model that needed them was retired
  • Tool definitions for tools the agent has not used in months

Every token in your prompt is paid for on every request. Treat the system prompt like production code — review it, prune it, version it.

A useful exercise: take a representative request, run it with the current prompt, then run it with a 50%-shorter prompt. If quality is unchanged, you have a free 50% saving on that endpoint.

5. Output token control (5–20% reduction)

Output tokens are typically 3–5x more expensive than input tokens. Yet most teams set max_tokens to "however high it goes" and let the model ramble.

Concrete moves:

  • Set max_tokens to a realistic ceiling per endpoint, not the model maximum
  • Ask for structured output (JSON) instead of prose — it is shorter and more useful
  • Use a "be terse" instruction in the system prompt for endpoints where verbosity is not the point
  • For chat UIs, enable streaming so users start reading sooner — and so you can cut off generation early if needed

Architectural Patterns Worth Knowing

Cache hierarchy

The cheapest LLM call is the one you do not make. Add a semantic cache (a vector-similarity lookup of recent queries → responses) in front of your LLM for any feature where similar inputs should produce similar outputs. FAQs, support deflection, and document summarization all benefit. Hit rates of 20–40% are typical for mature deployments.

Async pre-computation

If you can predict what users will ask, generate the answer ahead of time. New blog post → pre-generate the summary, the social posts, the related-content embeddings. The user-facing latency drops to zero and you pay batch-tier prices.

Self-hosted for high-volume narrow tasks

Once a single endpoint hits roughly 100M tokens/day on a narrow task, the math starts favoring a self-hosted, fine-tuned small model. The break-even point depends on your GPU costs, but it arrives faster than most teams expect. We covered the fine-tuning side in our LoRA / QLoRA post.

Common Anti-Patterns That Burn Money

  • Streaming for non-UI workloads. If no human is reading the output as it streams, just wait for the full response. Streaming adds nothing and complicates retries.
  • Re-uploading the same context every turn. Long multi-turn chats that re-include the entire prior history without caching pay for the same tokens repeatedly.
  • Using the biggest model "to be safe." Most production endpoints over-provision the model. Run an A/B with one tier down — you may find quality is identical and the bill is half.
  • Per-user prompts that should be shared. If 1,000 users get prompts that differ only in their name, you are missing a caching opportunity. Move the user-specific bits to the end of the prompt.
  • Retry storms. A model that occasionally returns malformed JSON can quietly 3x your bill if your retry logic is naïve. Add structured output mode and exponential backoff.

A 30-Day Cost Cutting Sprint

If you want a concrete plan, this is the one we run with customers:

  • Week 1: Add per-request telemetry. Find the top three endpoints by cost.
  • Week 2: Apply prompt caching to those three. Measure the hit rate.
  • Week 3: Set up model routing — at minimum, send obvious-easy requests to the cheap tier.
  • Week 4: Audit prompts, set sane max_tokens, move any non-real-time workloads to batch.

That sequence, in our experience, lands a 50–70% reduction without touching product behavior. The rest comes from architectural changes (semantic caching, self-hosted fallback) that need more lead time.

Conclusion

LLM cost optimization is not a one-time project. It is an ongoing discipline — like database query optimization or frontend bundle size. The teams that do it well bake cost telemetry into their dev loop and make it part of the PR review for any new AI feature.

The good news: the wins are large, the levers are well-understood, and most of them require no model expertise — just discipline.

Want a Cost Audit?

We have run this exact playbook for early-stage and Series-B teams alike. If your AI bill is climbing faster than your usage, book a call. We will tell you within an hour where the money is going and what to do about it.

Thanks for reading 😄.