23,545 tokens per turn: measuring the cost of re-prefill in coding agents

What I measured. What I learned. What it means for KV cache design.

In 2026, engineering teams are waking up to LLM bills that rival their cloud infrastructure. A significant portion of that cost isn't going into new work — it's going into reprocessing the same context, turn after turn, conversation after conversation.

I wanted to measure exactly how much. So I configured Claude Code to point to my own inference runtime, with Qwen2.5-32B as the backend, and opened a session. I worked normally. I watched the logs.

The point wasn't to replace Claude with a smaller local model. It was to measure something much more specific: how many tokens a coding agent actually re-prefills on every request, and how much of that could be saved with a properly designed KV cache.

The problem

Every coding agent — Claude Code, Cursor, Zed, Continue, any of them — sends a massive and largely redundant context on every request. The system prompt: several thousand tokens describing its 28 or 29 tools. The conversation history since the session started. The project files it has read along the way. The output of shell commands it has run. The diffs it has produced. Between two consecutive requests, 90 to 99% of that context hasn't changed.

The big providers know this. Anthropic has prompt caching in its API. OpenAI does too. But that's a provider-side cache: opaque, short TTL, monolithic, locked inside the provider's infrastructure. It's an optimization they built because they see the same requests come back a thousand times — not a capability you can compose, manage, or preserve across sessions.

Prefix caching helps, but it doesn't solve the problem. It optimizes a contiguous prefix: everything identical from the start of the request is reused, everything that diverges is recomputed. The system prompt — the largest stable segment — is cached just fine. But a coding agent's context isn't a prefix in continuous growth. It's a composition of independent segments: the system prompt never changes, read files rarely change, tool call results never change after execution, the conversation history grows with every turn. Each of these segments has its own lifecycle. Prefix caching drowns them in a contiguous stream and can't address them separately. It doesn't compose segments — it sees one growing block.

That's precisely the limit KV-first addresses: not a larger or longer cache, but a different cache unit — the segment, not the prefix.

The standard that defines this artifact — KV-first — is published on GitHub under Apache 2.0 + CC-BY-4.0. My runtime implements it.

The numbers

My runtime is KV-first: it manages a session cache, composes context segments, and persists activations across requests. But it also has a passthrough mode — a mode where it relays requests to the local model without activating the cache. That's the mode I used for this test, and it's precisely what simulates what happens when you talk to a provider that isn't KV-first: every request is stateless, everything is re-prefilled. Claude Code connected, and I started working normally.

On the third request, I looked at the logs.

A Claude Code request isn't a single call to the model. It's a sequence of tool calls — read a file, run a command, read another file — and each tool call sends context back to the model. In passthrough mode, the runtime re-prefilled everything on every tool call. The Claude Code system prompt — 5,936 tokens, 29 tools described one by one — was re-prefilled every time. The conversation history and files already read accumulated and were re-prefilled every time. The 23,545 tokens is the sum of those successive re-prefills within a single user request.

The setup: Qwen2.5-32B in Q4 quantization (GGUF), via llama.cpp, on an M4 Max 128 GB. No CUDA, no datacenter — an Apple Silicon workstation with unified memory. At real-world throughput on this configuration — around 24 tokens per second in decode — generation was slow, but the real cost was accumulated prefill. The full request cycle, 3 or 4 tool calls each with their own re-prefill, plus response generation, took minutes for a single user interaction. Throughput isn't the point. What matters here is the volume: 23,545 tokens prefilled when the vast majority already existed in computed form from the previous turn.

Next request, same structure. The context had grown, so the cumulative total climbed. Request after that: even more, because an additional file had been read. The runtime was re-prefilling everything, on every tool call, on every turn, because passthrough mode wasn't committing anything to the session cache.

Analyzing the logs, the diagnosis is clear: passthrough mode was disabling session cache commit, so history was re-prefilled on every turn even though the infrastructure to cache it already existed. The projection of the potential gain with session caching active gives:

Turn 1 : prefill system (5 936 tokens) + user (200)     = 6 136 tokens
Turn 2 : restore session KV + delta prefill             ≈   500 tokens
Turn 3 : restore session KV + delta prefill             ≈   500 tokens
Turn 5 : restore session KV + delta + fichier caché     ≈   500 tokens

About 97% reduction on turns after the first, if the session cache were active and read files committed separately. Note: this table is a projection of what session caching would allow — not a measurement. It's consistent with the measured numbers (5,936 tokens for the system prompt, growing history), but the actual gain hasn't been validated by a test with full caching active. The projection covers the gain between turns. The test revealed additional waste within a single turn: each tool call re-prefills the context. The real gain, tool calls included, would be higher than these numbers.

On another test, where the KV cache on the system prompt was active, I measured the gain directly: a request hitting a system prompt cache hit (5,936 tokens) took 1,024 milliseconds to restore activations, against 5 to 9 seconds to recompute them cold. A 5 to 9× speedup. Over a session where the same cache was reused five times, cumulative time saved was roughly 190 seconds on a total session of around 500 seconds — 38% of total session time, with no optimization on the model side, just by avoiding the recomputation of what had already been computed.

The direct measurements — 23,545 cumulative tokens, 5-9× speedup, 38% time saved — come from logs of a real test, on a real coding agent, against a real codebase. The inter-turn reduction is a projection, not a measurement.

What the test actually showed

Between two requests from a coding agent, 90 to 99% of the context is identical, and nothing in a provider's API lets you preserve it.

The gain is real and massive. Order of magnitude 5 to 9× speedup on restoration, 30 to 40% time saved on sessions with cache active, potentially 90%+ reduction in re-prefilled tokens on turns after the first. These aren't marketing numbers — they're measurements made on my logs in real time, on a workload I hadn't prepared to be favorable.

The provider APIs — OpenAI, Anthropic — aren't KV-first. They expose a request/response model where every request is stateless from the KV cache's perspective. No shared session, no segment composition, no client-side control over what gets cached and for how long. Everything has to be rebuilt on the runtime side, from the request stream coming in.

Identifying that two consecutive requests share 95% of their context: runtime. Deciding what gets committed to the session cache and what doesn't: runtime. Managing TTL, orphan purging, invalidation on file modification, artifact composition: runtime. That's work that looks more like building a storage system than adding an optimization. During the test itself, we saw contentions on parallel tool-call serialization, KV orphans that weren't purged in time, a gating semaphore with only a single permit — the kind of problems you encounter in a storage system, not in a feature.

And there's a deeper constraint: competent on-premise models require real hardware. The test used Qwen2.5-32B in Q4 on an M4 Max 128 GB — not a lightweight setup, but not a datacenter either. The economics of KV cache persistence only make sense when the prefill cost is high enough to justify the infrastructure. For light, stateless interactions, the waste is negligible. For heavy coding agents on large codebases, it's tens of thousands of tokens per turn — and that's where the artifact matters.

Why KV transfer won't come from the provider

Providers know about KV caching. They use it extensively on their own infrastructure — page-based KV memory management, prefix caching, prefill/decode decomposition, KV transfer between cluster nodes. What they expose in their API isn't the beginning of a move toward client-side KV transfer. It's a limited subset of what they already do internally, extracted to the API to stay competitive. There are three reasons they won't go further.

The first is economic. Providers bill per token consumed. Re-prefill is direct revenue. Reducing re-prefill reduces revenue. The prefix caching they offer isn't a step toward the solution — it's a compromise: lower the cost just enough not to lose clients to other providers, without ever eliminating re-prefill, which would cannibalize their monetization model. For the provider, a re-prefilled context isn't waste. It's a billed product.

The second is architectural. The most valuable form of KV transfer is split inference: prefill on the server-side GPU infrastructure, decode on the client's local machine. It's the model that eliminates re-prefill while decentralizing computation. But it requires the same model on both sides — the KV generated by prefill must be compatible with local decode. Claude's, GPT's, Gemini's models are proprietary. They won't be opened. Without access to the weights, the client can't decode. Without local decode, split inference is impossible. The only KV caching a proprietary provider can offer is server-side prefix caching — opaque, non-composable, non-portable. That's exactly what they already offer.

The third is practical. Even setting aside split inference and economics, storing KV client-side has a cost. A KV cache for a context of several thousand tokens represents hundreds of megabytes to several gigabytes per session. It consumes disk, bandwidth, and requires lifecycle management — TTL, purging, invalidation. A developer on a MacBook might not want their disk filling up with KV blobs. The provider has no incentive to introduce this friction when the status quo — billed re-prefill — is simpler and more profitable.

All three converge: KV transfer will never come from the provider. Not because it's technically impossible — they already do it internally. Not just because it would be less profitable — they bill per token. Because the most useful form of KV transfer, split inference, is structurally incompatible with a proprietary model. The only KV caching layer a provider can offer is an opaque server-side cache. Everything else has to live outside the provider, in the runtime, with open models — Qwen, Llama, Mistral — where split inference is possible and control belongs to the user.

That combination — real gain, hard engineering, no provider shortcut — is exactly what makes the KV cache the missing artifact in the stack. Not an optimization you add. A layer you build, between the agent and the model, because nobody else will build it for the user.