# How KV caching is built up in transformer architectures

[Skip to content](#lm-inhoud)Network/[NL](/en/kv-caching-opbouw)EN[Hubhub.llmnet.nlCompare models on task, language, cost and license.](https://hub.llmnet.nl/en/)[Communitycommunity.llmnet.nlPrompt techniques, patterns and system prompts.](https://community.llmnet.nl/en/)[APIapi.llmnet.nlLLMs in production: rate limits, routing, structured output.](https://api.llmnet.nl/en/)[Consultancyconsultancy.llmnet.nlRolling out AI in an organization, pilot to production.](https://consultancy.llmnet.nl/en/)[Newsnieuws.llmnet.nlAI developments, explained for the Netherlands.](https://nieuws.llmnet.nl/en/)[Benchmarkbenchmark.llmnet.nlMeasure AI quality yourself, on your own tasks.](https://benchmark.llmnet.nl/en/)[Careersvacatures.llmnet.nlAI roles, salaries and career paths in the Netherlands.](https://vacatures.llmnet.nl/en/)[Learnleren.llmnet.nlAI concepts in plain language, beginner to builder.](https://leren.llmnet.nl/en/)[Guidegids.llmnet.nlRun AI privately on your own Mac, PC, NAS or home server.](https://gids.llmnet.nl/en/)[Directorydirectory.llmnet.nlMapping the AI ecosystem: tools, models, companies.](https://directory.llmnet.nl/en/)[Radarradar.llmnet.nlSignals from X, research and communities for indie developers.](https://radar.llmnet.nl/en/)[Appsapps.llmnet.nlReviews of AI apps and open-source repos, with tips for builders.](https://apps.llmnet.nl/en/)[llmnet.nl — main site](https://llmnet.nl/en/)[](https://x.com/intent/post?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fkv-caching-opbouw&text=How%20KV%20caching%20is%20built%20up%20in%20transformer%20architectures)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fkv-caching-opbouw)[](https://www.reddit.com/submit?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fkv-caching-opbouw&title=How%20KV%20caching%20is%20built%20up%20in%20transformer%20architectures)[](#)[](https://x.com/intent/post?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fkv-caching-opbouw&text=How%20KV%20caching%20is%20built%20up%20in%20transformer%20architectures)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fkv-caching-opbouw)[](https://www.reddit.com/submit?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fkv-caching-opbouw&title=How%20KV%20caching%20is%20built%20up%20in%20transformer%20architectures)[](#)

 
# How KV caching is built up in transformer architectures

 By Ivo Donker — compiled with AI support (Claude & Gemini) · Last updated: 6 August 2026

 
 
## Introduction

 Within modern language models, efficiency is one of the main pillars of practical usability. When a model generates text, it does so token by token. This autoregressive nature of language models means every newly generated word (or word part) depends on all preceding tokens. In a naive transformer implementation, the model would have to recompute the entire history at every step. That leads to an enormous amount of redundant computation.

 To counter this computational redundancy, systems use key-value (KV) caching. This technique stores the intermediate computations of earlier tokens in memory, so that when generating a new token the model only has to perform the computations for that specific token. While KV caching drastically reduces the compute required, it introduces a new challenge: a substantial claim on the graphics processor's video memory (VRAM).

 

 
 
## Why KV caching exists

 To understand why KV caching is necessary, we have to go back to the basics of how a transformer model processes information. The article on [what a transformer is](https://leren.llmnet.nl/en/wat-is-een-transformer) describes how self-attention layers are used to establish relationships between tokens. Within these self-attention layers, three vectors are generated for every token: the query (Q), the key (K) and the value (V).

 When computing attention, the query vector of the current token is compared against the key vectors of all preceding tokens (including itself). This produces attention scores that determine how much weight to assign to the corresponding value vectors. The resulting output is a weighted sum of these value vectors.

 During the autoregressive generation process (predicting tokens in sequence), the context from the past does not change. The representations of tokens already written or read stay constant. Without caching, at token number 100 the transformer would have to recompute the key and value vectors of tokens 1 through 99 in full. By storing the already computed K and V vectors in a cache (the KV cache), the system only has to compute the Q, K and V vectors for the very newest token at each step. The new Q vector then performs a dot product against all stored K vectors from the cache, after which the weighted sum with the stored V vectors is determined.

 

 
 
## The construction and structure of the cache

 The KV cache is not a flat list of vectors; it is a structured multidimensional tensor. The size and shape of this tensor relate directly to the model's specific architecture. The cache is built along the following dimensions:

 
 
- Layers: Every transformer model consists of multiple stacked attention layers. A separate KV cache has to be maintained for each individual layer.
 
- Attention heads: Within each layer, the attention computation is split across multiple parallel heads. The cache stores the K and V vectors per head.
 
- Tokens (sequence length): The cache grows as the sequence (the context and the generated tokens together) gets longer. This is the dynamic dimension of the cache.
 
- Head dimension (head dim): This is the vector size per attention head (usually the hidden dimension divided by the number of heads).

 The shape of the KV cache tensor can roughly be represented as:

 [lagen, 2 (Key en Value), batch_grootte, heads, tokens, head_dim]
 This cache lives directly in the accelerator's fastest memory (usually VRAM on a GPU or HBM on a TPU). Because the cache has to be extended for every new token, its memory footprint grows linearly with context length. In long conversations or when processing large documents, this linear growth can saturate memory quickly, even when the model parameters themselves fit comfortably.

 

 
 
## Prefill versus decode: two different phases

 During inference the model goes through two fundamentally different phases: the prefill phase and the decode phase. This split has a major influence on how the KV cache is filled and used. For a broader perspective on these steps, see the article on [inference explained](https://leren.llmnet.nl/en/inference-uitgelegd).

 
 
### The prefill phase

 In the prefill phase the model processes the initial input (the prompt). Because all tokens in the prompt are known immediately, this processing can happen in parallel. The GPU can perform all matrix multiplications for the prompt tokens at once. During this phase the KV cache is built for the first time: the K and V vectors for the entire prompt are computed and stored in memory. This phase is computationally heavy (compute-bound) and makes optimal use of the hardware's parallel compute cores.

 
### The decode phase

 Once the prefill phase is complete and the first new token has been generated, the decode phase starts. Here the model generates token after token in sequence. Because each token can only be computed after the preceding one is known, this phase is inherently sequential. The GPU can no longer compute in parallel across the time dimension.

 During every decode step the model fetches the full KV cache from memory, adds the newly computed K and V vectors of the current step to the cache, and performs the attention computation. Because relatively few computations are performed compared with the amount of data that has to be loaded from memory, the decode phase is bandwidth-bound. This explains why generation speed often stalls at larger contexts: moving the ever-growing KV cache from VRAM to the compute cores is the main bottleneck.

 

 
 
## Memory trade-offs and VRAM consumption

 A transformer's VRAM consumption during inference consists roughly of two parts: the static model weights and the dynamic KV cache. While the model weights stay constant, the KV cache grows with every step. This means peak memory usage is determined directly by batch size and context length.

 To give a sense of proportion we can look at the memory footprint. The amount of memory (in bytes) needed for the KV cache can be expressed mathematically as:

 
 Formula for KV cache size:
 Geheugen (bytes) = 2 × lagen × heads × head_dim × tokens × batch_grootte × bytes_per_element
 
 Here the factor 2 stands for the two components: the key and the value vector. The bytes_per_element depends on the precision the model runs in (2 bytes for 16-bit floating point FP16/BF16, for instance, or 1 byte for FP8/INT8 quantization).

 This formula shows that the relationship with context length and batch size is strictly linear. If context length doubles, the KV cache required doubles too. If batch size (the number of users served simultaneously) doubles, the cache likewise doubles. This means that in large-scale systems the KV cache can quickly become larger than the model parameters themselves. It forces developers to make choices between maximum context length and system throughput.

 

 
 
## Techniques for shrinking the KV cache

 Because of the enormous pressure the KV cache puts on video memory, various architectural adjustments have been developed to reduce cache size without costing model performance. In the basic attention mechanism described in the article on [attention explained](https://leren.llmnet.nl/en/attention-uitgelegd), every query head has its own key and value head. Modern variants depart from this, however.

 
 
 
 Technique | 
 Query vs KV head ratio | 
 Impact on KV cache | 
 

 
 
 
 Multi-head attention (MHA) | 
 1:1 (every Q head has its own K and V head) | 
 No reduction (maximum cache size) | 
 

 
 Multi-query attention (MQA) | 
 Many-to-1 (all Q heads share 1 K and V head) | 
 Drastic reduction (up to 90%+ smaller) | 
 

 
 Grouped-query attention (GQA) | 
 Many-to-few (Q heads grouped per K/V head) | 
 Balanced reduction (most used in modern LLMs) | 
 

 
 

 
### Multi-query and grouped-query attention

 In multi-query attention (MQA) all attention heads in a layer share a single key and value head. This shrinks the cache by a factor equal to the number of heads (often 32 or more). While this relieves memory bandwidth enormously, it can lead to a slight drop in model accuracy, because the model has less capacity to distinguish complex patterns.

 As a middle ground, grouped-query attention (GQA) was introduced. Here the query heads are divided into groups (8 groups, for instance), with each group sharing one key and value head. This offers an excellent compromise: it preserves virtually the full quality of MHA while shrinking the KV cache considerably.

 
### Sliding-window attention and PagedAttention

 Alongside adjustments in model architecture there are also dynamic techniques at software level. Sliding-window attention caps the maximum size of the cache by keeping only the K and V vectors of the most recent tokens (the last 4,096 tokens, for instance). Older tokens are removed from the cache, so the cache no longer grows linearly without bound.

 Another important innovation is PagedAttention (introduced by vLLM). In traditional systems the KV cache has to be stored in one contiguous block in VRAM. That leads to fragmentation and waste, because systems have to reserve memory in advance for the maximum possible sequence length. PagedAttention divides the KV cache into small, non-contiguous pages (comparable to virtual memory management in operating systems), which virtually eliminates fragmentation and allows considerably more batches to run concurrently.

 

 
 
## Cache-related artifacts and limits

 It is important not to confuse the KV cache at model level with response caching at API level. The article on [caching LLM responses](https://api.llmnet.nl/en/caching-llm-antwoorden) explains how complete model answers are stored in order to answer repeated questions immediately. The KV cache, by contrast, works under the model's hood during computation and contains raw activation vectors, not text.

 
 An important characteristic of the KV cache is that it is strictly context-dependent and sequential. Because the positional encoding of tokens affects how the key and value vectors are built, you cannot simply split or reuse a KV cache for an entirely different conversation. Any change to the prompt (even changing a single letter at the start) invalidates the entire subsequent KV cache. In such a case the cache has to be rebuilt from the point of change through a prefill step.

 In systems that use long, static documents as a basis (a manual about which multiple questions are asked, for instance), advanced techniques can be used. For more on reusing prompts, see the article on [context caching explained](https://hub.llmnet.nl/en/context-caching-uitgelegd). Here the KV cache of the static part (the document) is stored on disk or in a less fast memory segment, so it can be loaded quickly on a new question and shorten prefill time.

 

 
 
## Practical observation and monitoring

 In practice, the KV cache's behavior is directly visible when you run local inference servers (such as vLLM, llama.cpp or Ollama). When starting a model you often see a large part of VRAM claimed immediately for the model parameters. As soon as requests come in, VRAM consumption rises further. That is the dynamic allocation of the KV cache.

 Many inference engines display statistics in their logs such as kv_cache_usage or show the percentage of available cache blocks in use. When this percentage approaches 100%, the server has to queue requests or abort active sessions because there is no room left to generate new tokens. In cloud services you sometimes see terms such as "cached tokens" in API response metadata. That indicates server-side prompt caching, where the provider reuses the KV cache of common system prompts to cut cost and latency.

 

 
 
## Further reading

 
 
- [What is a transformer?](https://leren.llmnet.nl/en/wat-is-een-transformer)
 
- [The attention mechanism explained](https://leren.llmnet.nl/en/attention-uitgelegd)
 
- [The anatomy of LLM inference](https://leren.llmnet.nl/en/inference-uitgelegd)
 
- [Caching LLM responses at API level](https://api.llmnet.nl/en/caching-llm-antwoorden)
 
- [Context caching in practice](https://hub.llmnet.nl/en/context-caching-uitgelegd)

 

 llmnet.nl - Course & education platform