Interactive KV Cache Memory Calculator
Welcome to this interactive calculation tool within module 5 of the curriculum on llmnet.nl. This module focuses entirely on the theme "Under the hood — architecture & efficiency". As a builder, indie developer, or homelab administrator, running Large Language Models (LLMs) locally or on your own infrastructure brings you face to face with the physical limits of graphics memory (VRAM). The model weights form only one part of the total memory requirement; as soon as a model generates or processes text, the dynamic Key-Value (KV) cache claims a substantial and rapidly growing share of the memory.
This calculator fills the practical hands-on gap on the platform as a second interactive tool, alongside the existing visualizer. The purpose of this tool is to precisely calculate the theoretical memory size of this cache under various settings. The structure of the KV cache itself is not explained again from scratch on this page; that substantive background is described extensively on the page about the structure of KV caching. This specific page purely calculates based on the parameters and helps you accurately estimate in advance how much VRAM the cache will occupy.
What you need to know beforehand
To optimally interpret the underlying variables in the calculator, basic knowledge of the transformer architecture is required. Go through the following three documents before diving deeper into the calculations:
- The attention mechanism explained: To understand how the Key and Value vectors arise from the internal projection matrices and why they need to be stored.
- Structure of the KV cache: For a detailed view of how tokens are added to the cache step by step during the autoregressive generation phase.
- Context engineering explained: Why context length is the most dominant variable in memory consumption and how prompt structure affects this.
The KV Cache Memory Calculator
Enter the technical parameters below of the model you want to analyze. The values are processed directly in the calculation without any data being sent to a server.
Important note on the result: This calculation gives the absolute lower bound of the memory required for the Key and Value tensors alone. In practice, actual VRAM usage turns out higher due to additional factors such as runtime activations, memory fragmentation, overhead from the inference framework (such as vLLM or Ollama), and reserved buffers of the GPU driver.
Guide for entering the values
To get correct results from the calculator, you need the exact architecture parameters of the chosen model. This information is usually easy to find via the following sources:
- The
config.jsonfile: In Hugging Face's storage repository, you'll find the model's configuration file. Look for fields such asnum_hidden_layers(layers),hidden_size(hidden size),num_attention_heads(heads), andnum_key_value_heads(KV heads). - The Model Card: Many developers list the network dimensions directly in the technical specifications on the model's introduction page.
- Determining the GQA ratio: If the field
num_key_value_headsis not explicitly mentioned, the model is probably using classic Multi-Head Attention (MHA). In that case, enter the exact same number for KV heads as for the number of attention heads.
If you're unsure about specific parameters of a local quantization or a modified model, enter the values conservatively. Setting a higher precision (such as 16-bit) gives you a safety margin when preparing your hardware capacity.
What the result means
The result of the calculator shows how drastically memory requirements scale up as requests get longer or when multiple users are served simultaneously. Every token processed by the model — both in the initial processing phase (prompt/prefill) and in the subsequent generated steps (decoding) — requires storing two vectors (the Key and the Value) in every layer of the transformer. To find out how many tokens a specific Dutch text yields, you can use the tokenizer visualizer tool to analyze the exact token density of your input.
Because autoregressive generation happens step by step, all these stored vectors must remain in VRAM until the entire conversation or task is completed. This memory stays reserved for the full duration of the generation. A detailed explanation of how and when these cache memories are freed in the lifecycle of a request can be found in the article about the course of the inference phase. Directly holding onto this data explains why expanding a context window from, for example, 4,096 to 32,768 tokens not only requires more compute time, but also increases the memory footprint of the KV cache proportionally by a factor of eight.
When you work on local hardware with a fixed amount of VRAM (such as a graphics card with 16 GiB or 24 GiB), the KV cache is the variable factor that determines whether a request succeeds or the software crashes with an 'Out of Memory' (OOM) error. While the model weights take up a static amount of memory, the KV cache grows linearly with the context length and the batch size. Read more on the guide site about how to set and optimize context length on local hardware to tightly manage these limits within your own system configuration.
Why GQA shrinks the bill
Grouped-Query Attention (GQA) is an architectural adjustment in which multiple query heads share a common Key and Value head. Without zooming in on the exact mathematical matrix operations of the attention layers, the direct practical effect is that the dimension of the stored KV vectors decreases drastically. In the calculator, you see this reflected directly through the field KV-heads. If a model has 32 query heads but only 8 KV heads (a ratio of 4:1), four times less data is stored per token and per layer compared to traditional Multi-Head Attention. A full explanation of how this technique works can be found on the page about Grouped-Query Attention.
Precision as a second lever
In addition to adjusting the network architecture (as with GQA), the precision of the stored numbers is the second important lever for influencing the memory usage of the KV cache. By default, the Key and Value vectors are stored in FP16 or BF16 (16-bit float), which costs 2 bytes per number. Modern inference engines, however, offer the option to quantize the KV cache separately from the model weights to 8-bit (1 byte per number) or even 4-bit (0.5 byte per number).
Lowering the KV cache precision halves (at 8-bit) or quarters (at 4-bit) the total size of the cache in VRAM. This means that with the same hardware, you can handle a significantly larger context length or run a larger batch size. There is a subtle quality trade-off in return: heavily quantizing the KV cache can lead to a slight decrease in the accuracy of the attention scores at very long contexts. In practice, 8-bit KV quantization turns out to be virtually lossless for most applications, while 4-bit is mainly used in extremely memory-constrained systems.
When the estimate deviates
The calculator computes using the purely theoretical size of the tensors. In a production environment or local setup, there are various factors that make the actual memory usage higher than the calculated value. It is essential to take these factors into account during capacity planning:
- Activation Memory: During the forward pass, intermediate results of matrix multiplications must be temporarily stored in VRAM. This memory fluctuates continuously per generated token.
- Memory fragmentation: As requests with varying context lengths are processed interleaved, unused space can arise between memory blocks (external fragmentation). Modern techniques such as PagedAttention greatly reduce this problem, but never rule it out 100%.
- Framework Overhead: Inference servers such as vLLM, TensorRT-LLM, or Ollama reserve a fixed percentage of VRAM (for example 90%) as a memory pool at startup. This also includes internal data structures and CUDA contexts.
- Model weights: The model parameters themselves claim the base space. An 8B parameter model at 4-bit precision already takes up about 5.5 GiB of static VRAM before the first token is even processed.
Therefore, use the calculator explicitly as an arithmetic lower bound. To understand the total system load and the operational costs of long-running processes, we refer to the overview on the power consumption and hardware load of local AI. For a quick reference of all the technical terms used, you can go to the AI glossary.
Example calculation
To make the formulas behind the calculator clear, we'll walk through a calculation example step by step. Suppose we have a model with the following characteristics:
| Parameter | Value | Description |
|---|---|---|
| Layers (L) | 32 | Number of consecutive transformer layers |
| Hidden Size (d_model) | 4096 | Total internal dimension of the model |
| Attention heads (H) | 32 | Total number of query heads |
| KV heads (H_kv) | 8 | Number of Key/Value heads (GQA ratio 4:1) |
| Precision | 16-bit (FP16) | 2 bytes per element |
| Context length (N) | 8.192 | Number of tokens in the context |
| Batch size (B) | 1 | A single active conversation |
The exact mathematical structure of the formula follows three steps:
Step 1: Determine the effective KV dimension per layer
Because this involves Grouped-Query Attention with 8 KV heads out of 32 attention heads, the KV dimension is smaller than the total hidden size:
kv_dim = hidden_size × (kv_heads / heads) = 4096 × (8 / 32) = 1024
Step 2: Calculate the memory size per token per layer
Per token, we need to store both a Key vector and a Value vector (factor 2). At FP16 precision, each value costs 2 bytes:
bytes_per_token_per_laag = 2 × kv_dim × (precisie_bits / 8) = 2 × 1024 × 2 = 4096 bytes (4 KiB per layer)
Step 3: Multiply by the number of layers, the context length, and the batch size
Multiply this by the network's 32 layers, the context length of 8,192 tokens, and a batch size of 1:
totaal_bytes = 8192 × 32 × 4096 × 1 = 1.073.741.824 bytes
To convert to mebibytes (MB), we divide by 10242 (1,048,576). This gives exactly 1024 MB. To convert to gibibytes (GiB), we divide by 10243 (1,073,741,824). The final result is exactly 1.00 GiB.
For comparison: if this exact model were to use classic Multi-Head Attention (where KV heads equal 32), the memory size per layer would rise from 4 KiB to 16 KiB per token. The total cache memory for the same 8,192 tokens would in that case amount to 4.00 GiB. This shows that setting the correct GQA ratio in the calculator reduces the calculated memory load by exactly a factor of four.
Continue reading with
Now that you have insight into the memory calculation of the KV cache, you can deepen your knowledge further with the following articles on the knowledge network:
- Grouped-Query Attention explained in depth: How GQA is mathematically structured and what impact this has on the quality metrics of LLMs.
- Speculative decoding explained: Explore an alternative method to increase generation processing speed without physically shrinking the KV cache.
- Guide: Optimizing the context window on local hardware: Practical steps to get the maximum context out of your own GPU using RoPE scaling and quantization.
