Before your agent reads your request, it may have already processed 50,000+ tokens. That is not my number. Anthropic's advanced tool use post, citing its earlier code execution with MCP post, notes that tool results and definitions "can sometimes consume 50,000+ tokens before an agent reads a request."
The expensive part of many agent sessions is not the answer. It is the context the model has to process before it can start answering, and in a typical agent loop that context is sent again on every call: system prompt, tool definitions, conversation history, tool results. Each step is a new API request, and the model has no memory between requests except what is in the request context.
So my working thesis is simple: pay once, not every turn. Below are the four levers I use to get closer to that, with vendor numbers dated as of Oct 2026.
Most of what you pay for was already sent last turn
A model call bills input tokens and output tokens. In an agent loop, the input side grows with every turn because the history grows, and the fixed overhead (instructions plus tool definitions) rides along each time. Anthropic's example makes the overhead concrete: a five-server MCP setup with GitHub, Slack, Sentry, Grafana, and Splunk is "58 tools consuming approximately 55K tokens before the conversation even starts" (Advanced tool use).
"Mostly" needs a scope. Output can dominate some workloads: long generations, or heavy reasoning, where OpenAI notes that reasoning tokens "count toward output usage and are billed as output tokens" (OpenAI tokens help). If your agent writes long documents in a few calls, this post matters less to you. If it runs dozens of short tool steps against a big prefix, it matters a lot.
Caching the stable prefix is the cheapest lever you have
Both Anthropic and OpenAI can reuse the processed state of a prompt prefix instead of computing it again. The pricing is what makes it worth caring about.
On Anthropic, per the prompt caching docs as of Oct 2026: 5-minute cache writes cost 1.25 times the base input price, 1-hour cache writes cost 2 times, and cache reads cost 0.1 times. A few newer models read even cheaper (0.05x or 0.025x per the docs' footnotes). The default lifetime is 5 minutes, refreshed for free each time the entry is used, with a 1-hour option at the higher write price.
On OpenAI, the prompt caching guide describes caching as on by default for supported models, with cached input "discounted up to 95%." For GPT-5.6 and later, writes cost 1.25x the uncached input rate and reads cost 0.1x on most of those models. OpenAI does the arithmetic for you: one write and nine full reads across ten requests "cost 2.15× at that rate, compared with 10× without caching." The minimum cacheable prefix is 1,024 tokens on GPT-5.6 and later, and on those models entries stay eligible for at least 30 minutes after the latest write or reuse (they may be kept longer).
Caching also cuts time, not just money. Anthropic says you "will generally see improved time-to-first-token for long documents," and OpenAI lists reducing "the time spent processing input before the response starts" as a core benefit. Neither page gives a latency multiplier, so I won't invent one.
The tradeoff is the write premium. If a prefix is written and never read again, you paid 1.25x (or 2x for the 1-hour TTL on Anthropic) for nothing. Caching pays off when the same prefix is reused within its lifetime, which is the shape of most agent loops.
A cache that never hits is a tax
Caching only works on an exact prefix match. Anthropic states that "cache hits require 100% identical prompt segments," and the prefix order is tools, then system, then messages. Change something early and everything after it misses.
The failure modes I look for:
- Volatile content up top. A timestamp, request id, or per-user detail in the system prompt makes every request unique. Anthropic's docs walk through exactly this case: a timestamp in the breakpoint block means "you pay for a fresh cache write on every request and never get a read." OpenAI's guidance matches: put timestamps and user-specific content at the end, not the beginning.
- Tool churn. Adding, removing, or reordering tools between calls invalidates the cache. OpenAI suggests keeping the
toolslist fixed and usingtool_choice: "none"orallowed_toolsto restrict what is callable instead of editing the list. - Rewritten history. Append, don't edit. OpenAI's guide is explicit that "summarization, compaction, or context truncation can change the prefix and reset cache reuse."
- Setting flips. Changing thinking or effort settings, tool choice, or images can invalidate part of the cache on Anthropic, and model, tools, reasoning effort, and verbosity changes can break reuse on OpenAI.
None of these show up as errors. The request succeeds; you just pay full price.
Trimming beats caching when the tokens never needed to be there
Caching makes repeated tokens cheaper. It does not make them free, and it does not make the model better at using them. Some context should simply not be in the window.
Load tools on demand. Anthropic's tool search tool lets you mark tools with defer_loading: true so only a search tool and a few core tools load up front. In Anthropic's example from the advanced tool use post, total context consumption drops to "~8.7K tokens" versus roughly 77K with everything loaded, which they describe as "an 85% reduction in token usage." Deferred tools also don't break caching, because they are excluded from the initial prefix and appended later. The cost is an extra search step, so Anthropic suggests it when definitions exceed 10K tokens or you have 10 or more tools. OpenAI's guide describes the same pattern with tool search and defer_loading.
Move bulk work into code. Anthropic's code execution with MCP post describes presenting tools as files the agent can read on demand and processing data in a sandbox instead of piping it through the model. In their example, that "reduces the token usage from 150,000 tokens to 2,000 tokens," a saving of 98.7%. The tradeoff is real: you now run agent-written code, which needs sandboxing, resource limits, and monitoring.
Drop or summarize stale history. This one is in tension with caching. Compacting history changes the prefix and costs you a cache miss, yet a smaller window can still be cheaper overall; OpenAI says as much and suggests comparing total input cost before and after. There is a quality argument too. Chroma's Context Rot report evaluated 18 LLMs and found that performance can degrade as input length grows, even on simple tasks. Longer context is not neutral.
You can't cut what you can't see
Every lever above depends on measurement. The usage fields are already in the response:
- Anthropic reports
cache_creation_input_tokens,cache_read_input_tokens, andinput_tokens. If both cache fields are 0, nothing was cached, possibly because the prefix was under the model's minimum. - OpenAI reports
usage.input_tokens_details.cached_tokensandcache_write_tokens, and suggests tracking a token cache-hit rate: cached tokens divided by total input tokens. - OpenAI's prompt cache diagnostics go one step further. Set
prompt_cache_options.comparison_response_idto an earlier response and the API reports a reason for a miss, such astools_changedorinput_changed. It is available in the Responses API for GPT-5.6 and later and has no additional cost. - Before sending anything, count. OpenAI's help page points to tiktoken for plain text and an input-token counting API for full Responses inputs, and warns that a plain-text count misses tools, schemas, and message structure.
This is the same itch that led me to build Context Mate. It is a local dashboard that reads the traces Cursor, Claude Code, and Codex leave on disk and shows them in one place: Cursor's context snapshot buckets by serialized size in bytes, and Claude Code's token usage per turn. It is honest about its limits, and so should I be here. It does not give a named, file-by-file inventory for Cursor, it does not convert bytes to tokens, Codex support is best-effort, and it depends on undocumented storage that can break. For the levers in this post, the API usage fields are the source of truth. What Context Mate did give me was a clear view of where those traces live and how unevenly the agents expose them.
The order I would apply this in
Measure first: log cached versus uncached input tokens per call for a day. Then fix prefix stability, because a cache you already pay for but never hit is the easiest money to recover. Then turn on caching deliberately where it is not already on, and put the breakpoint at the end of the stable part. Only then start trimming: defer tools, push bulk data into code, and compact history when the window is long enough that a cache miss is cheaper than carrying it.
It is the same discipline as any other hot path: find the work you repeat, stop repeating it, and check the numbers.
- AI
- Developer Tools
- Agents
- Cost
