From Token-First to KV-First
- Jul 6
- 7 min read
The token is not a choice. It's a hardware constraint — and it's the wrong level of abstraction for thinking about what a model understands.
Imagine opening a book and having to read it from the beginning every time. Not because you forgot — because someone rips out your bookmarks between each reading.
That is what a language model does. On every call, it rebuilds its understanding of the context from scratch. The state it produced — what it had "understood" — is thrown away between calls.
I measured the cost of this waste on my own work. About $3,000 in tokens to build a software demonstration over a few weeks. Eight different models, thousands of API calls. At the end, I did the math — not the total cost, but what had been useful. A significant share of those tokens had served one purpose: re-explaining to the model what it had already understood on the previous call.
The same codebase, re-parsed. The same technical context, re-injected. The same architecture, re-described. Every new session, every model switch, every timeout — back to zero.
It's not a bug. It's the architecture.
Why the token
When we talk about generative AI, everything is measured in tokens. Context window: 128K tokens. Cost: cents to dollars per million. Memory: proportional to token count. Speed: tokens per second.
The token is the atomic unit. The entire pipeline is built around it. But why?
A neural network doesn't read text. It computes on numbers — fixed-size vectors. So each word has to be transformed into a numerical vector before the model can process it. But words are unpredictable: some are common, some are rare, some don't exist yet. You can't assign a fixed vector to every possible word in advance.
The token solves this. Instead of working with whole words, text is split into shorter, more regular fragments — sub-units that recur often and cover any text. Each fragment gets a fixed-dimension vector. That is the elementary brick the model can process.
And it's the GPU that made this brick the center of the entire system.
A GPU is not a general-purpose processor — it's a machine specialized in one operation: matrix multiplication. Thousands of cores executing the same instruction in parallel on fixed-size data blocks. The software layer that orchestrates this is called CUDA — it's what turns raw silicon into compute capacity for AI. And CUDA has a very precise appetite: regular tensors, predictable size, optimal grid filling.
The token satisfies this appetite perfectly. The transformer's central operation — attention — is a dense multiplication of these vectors, massively parallelizable. The token feeds the GPU optimally. That's why it became the unit of measurement for the entire industry — windows, costs, memory, speed: everything is counted in tokens.
Not because it's the best way to represent meaning — because it's the best way to fill a GPU.
The problem no one names
The reconstruction process — the model walking through the provided context and building an internal state on every call — has a technical name: prefill. And it's expensive: cost is proportional to context length.
Prefill produces a state that represents what the model "understood" of the whole. But when the call ends, that state is destroyed. The model has no memory not because it's incapable of understanding — but because no one keeps what it understood.
I've approached this observation from other angles in earlier writing, without naming it. In Le cadavre exquis, I described a model that continues a sentence without knowing where it's going.
"It's not losing the thread, it's never having had one."
What was missing was the infrastructural cause. Here it is.
From sequence to state
The internal state the model builds during prefill has a technical name: the KV cache (Key-Value cache). It's the consolidated output of transformer processing — the output of the computation, not the input.
In a token-first system, the KV cache is an ephemeral byproduct. It's born during inference. It dies after generation.
What if, instead of throwing it away, we kept it?
That is the founding idea of what I call KV-First: elevate the KV cache to a first-class artifact. A named, versioned, verifiable, transportable, persistent object. The unit of work is no longer "what the model receives" (tokens) but "what the model understood" (an internal state).
This is not an optimization of Token-First.
It's a change of abstraction level. We move from sequence to state.
What it changes, concretely:
Persistence. A codebase encoded once stays available indefinitely. No more re-prefill.
Transport. The state of understanding can move between machines, between sessions, between nodes in a network. Imagine an interpreter who, at the end of a session, archives her notes instead of throwing them away — the next day, another interpreter picks up those notes and continues without having listened to the original speech. What she holds is not the words of the speech. It's her predecessor's state of understanding.
Composition. Multiple KV artifacts — base context, specific module, session state — can be assembled into a single inference context.
Amortization. The cost of prefill is paid once and amortized over N subsequent requests, which only need to decode.
Take a codebase or a document archive — a heavy, stable context consulted regularly. For 100,000 tokens consulted 10 times in a day, a KV-First system performs prefill once. The 9 subsequent requests pick up directly where the first one stopped. The ratio measured on a 3-node Apple Silicon cluster, with a Llama 3.2 3B model and cross-session reuse (KV artifact loaded from persistent storage vs cold prefill on the same context): 6.3× reduction in time-to-first-token. The measurement protocol details are in the repo.
The context window and what was invented to work around it
The token-first problem doesn't stop at re-computation. There's a second one, just as structural: the context window is limited.
A model can only process a certain number of tokens at a time — 4K, 32K, 128K depending on the architecture. Beyond that, the text doesn't fit. The model simply cannot reason over a corpus that exceeds its window.
The industry has produced several responses to this limit.
On the cloud side, prompt caching (Anthropic, OpenAI, Google) caches a prompt prefix between calls — but it's a provider-side cache, opaque, limited to a monolithic prefix, with a short TTL and no inter-session persistence.
On the serving infrastructure side, the landscape has moved fast. vLLM, SGLang, and LMCache now manage KV cache as a persistent, reusable resource — not just an internal buffer. LMCache runs as a standalone process, surviving engine crashes, with tiered storage from CPU memory to distributed backends. Mooncake Store — the architecture behind Kimi's production serving at Moonshot AI — provides a distributed KV cache pool with RDMA and zero-copy transfer, integrated across vLLM, SGLang, and TensorRT-LLM. The industry has recognized what this article argues: the KV cache is an artifact worth keeping.
What these systems don't provide is a standard. Each manages KV inside its own walls, with its own format, its own lifecycle. The KV cache persists, but it doesn't travel — there is no compatibility contract between runtimes, no portable artifact a different system could consume without reinventing the encoding. The infrastructure is being built. The standard is not.
But caching and persistence are not the only responses to the context window. A different strategy has become the most widespread: instead of trying to keep what the model understood, reduce what enters the window in the first place. That approach is RAG (Retrieval-Augmented Generation): select the most relevant fragments before submitting them to the model. An embedding index, a similarity search, and only the passages deemed useful are injected into the context. It's ingenious — but it poses a structural paradox: the relevance of a fragment is often only determinable after reasoning over the whole. Retrieval decides before; reasoning arrives after. And a fragment torn from its document loses its emergence context — the conceptual framework, the constraints established earlier, everything that gives a passage its meaning without being in the passage itself.
KV-First approaches the problem differently. If the corpus fits in a persistent KV artifact, it doesn't need to be sliced. The model has already reasoned over the whole during the initial prefill. The global understanding is preserved in the activations — it influences reasoning without occupying visible space in the window on subsequent calls.
This is not a better RAG. It's an approach that eliminates the need for slicing for stable corpora where global coherence matters more than local relevance — codebase, regulatory archive, medical record, legal corpus. For dynamic cases, RAG remains the right approach. The two are not on the same axis.
What it doesn't do
Honesty requires clear limits.
KV-First works today on existing transformers. Persistence and transport of KV artifacts are implemented, deployed, and measured. But the KV cache remains constrained by the transformer architecture: the attention mechanism requires materializing the full artifact in memory. A 100K-token context requires proportional VRAM, regardless of how much is actually useful for the current request.
KV-First solves the re-computation problem. Not the memory problem.
The KV cache, even persisted and transported, remains an opaque internal state. You can store it, transport it, compose it at the position grain. You can't edit it surgically at the meaning grain — there's no way to say "replace Paris with Lyon in the model's understanding" without re-parsing the context.
It's a transport substrate, not a semantic database.
A KV artifact is also bound to the model that produced it. Switching models invalidates the artifact — cross-model transport is an open problem, not an acquired capability.
The gains are measurable and significant on stable corpora. They don't apply mechanically to every use case. Conversational dialogue that changes every turn, real-time data streams — those are not the cases where KV-First changes the game. The natural use case is heavy context that changes rarely and is consulted often: codebase, documentation, regulatory corpus, knowledge base.
A paradigm shift, not a feature
Moving from token-first to KV-First is not adding a smarter cache. It's a change in what we consider the fundamental unit of the system.
The token remains essential — it's the interface between text and model, and it's the unit the GPU knows how to process. But the token is not the right level for thinking about persistence, transport, and composition of a model's understanding.
The KV cache is.
I formalized this idea in a technical framework — KV-First — released open-source in March 2026. It defines compatibility contracts for KV artifacts, conformance levels, and a test suite. It's a starting point, not a destination.
The model, on every call, understands.
What's missing is someone to keep what it understood.
Comments