Language model APIs are billed by tokens, the chunks of text going in and coming out. A prototype handling a few hundred requests costs almost nothing. The same feature serving every customer, every day, with long prompts and the largest model, can become one of the bigger lines on the cloud bill.
The good news is that most of that spend is avoidable, and the savings usually come from engineering rather than from compromises in quality.
Measure cost per task before optimising
Log the input and output tokens, the model and the feature for every call, and calculate cost per completed task, not just per request. One user action may trigger several model calls. Knowing which feature and which step dominates tells you where to start.
Route each request to the smallest model that can handle it
Providers offer families of models at very different prices. Classification, extraction and short rewrites are often handled well by small, fast models; complex reasoning may need a larger one. Route by task, and use your evaluation set to confirm the cheaper model still passes. Some requests do not need a model at all: a rule or a database lookup is cheaper and more reliable for anything deterministic.
Use prompt caching for repeated context
If many requests share a long prefix, such as a system prompt, a policy document or tool definitions, the major providers including OpenAI and Anthropic offer prompt caching, which bills repeated input at a reduced rate and also cuts latency. Structure prompts so the stable part comes first and the variable part comes last, and check your provider’s current documentation for how caching is triggered and priced.
Batch work that does not need an instant answer
Overnight summaries, document backfills and bulk classification rarely need a reply in seconds. OpenAI and Anthropic both offer batch APIs that process requests asynchronously, typically within 24 hours, at a substantial discount compared with standard calls. Move every non-interactive workload there.
Send less, generate less
- Retrieve fewer, better passages instead of pasting whole documents into the prompt.
- Trim conversation history: summarise older turns rather than resending them all.
- Ask for structured, concise output and set a sensible maximum output length.
- Cache complete answers for questions that repeat exactly, such as common help queries.
Keep quality in the loop
Every cost change is also a behaviour change. Run the evaluation set before and after, and watch user feedback for a couple of weeks. Savings that quietly degrade answers end up costing more in support and lost trust than they saved.
