# Grouped-query attention and memory usage explained [Skip to content](#lm-inhoud)Network/[NL](/en/grouped-query-attention-uitgelegd)EN[Hubhub.llmnet.nlCompare models on task, language, cost and license.](https://hub.llmnet.nl/en/)[Communitycommunity.llmnet.nlPrompt techniques, patterns and system prompts.](https://community.llmnet.nl/en/)[APIapi.llmnet.nlLLMs in production: rate limits, routing, structured output.](https://api.llmnet.nl/en/)[Consultancyconsultancy.llmnet.nlRolling out AI in an organization, pilot to production.](https://consultancy.llmnet.nl/en/)[Newsnieuws.llmnet.nlAI developments, explained for the Netherlands.](https://nieuws.llmnet.nl/en/)[Benchmarkbenchmark.llmnet.nlMeasure AI quality yourself, on your own tasks.](https://benchmark.llmnet.nl/en/)[Careersvacatures.llmnet.nlAI roles, salaries and career paths in the Netherlands.](https://vacatures.llmnet.nl/en/)[Learnleren.llmnet.nlAI concepts in plain language, beginner to builder.](https://leren.llmnet.nl/en/)[Guidegids.llmnet.nlRun AI privately on your own Mac, PC, NAS or home server.](https://gids.llmnet.nl/en/)[Directorydirectory.llmnet.nlMapping the AI ecosystem: tools, models, companies.](https://directory.llmnet.nl/en/)[Radarradar.llmnet.nlSignals from X, research and communities for indie developers.](https://radar.llmnet.nl/en/)[Appsapps.llmnet.nlReviews of AI apps and open-source repos, with tips for builders.](https://apps.llmnet.nl/en/)[llmnet.nl — main site](https://llmnet.nl/en/)[](https://x.com/intent/post?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fgrouped-query-attention-uitgelegd&text=Grouped-query%20attention%20and%20memory%20usage%20explained)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fgrouped-query-attention-uitgelegd)[](https://www.reddit.com/submit?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fgrouped-query-attention-uitgelegd&title=Grouped-query%20attention%20and%20memory%20usage%20explained)[](#)[](https://x.com/intent/post?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fgrouped-query-attention-uitgelegd&text=Grouped-query%20attention%20and%20memory%20usage%20explained)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fgrouped-query-attention-uitgelegd)[](https://www.reddit.com/submit?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fgrouped-query-attention-uitgelegd&title=Grouped-query%20attention%20and%20memory%20usage%20explained)[](#) # Grouped-query attention and memory usage By Ivo Donker — compiled with AI support (Claude & Gemini) · Last updated: 6 August 2026 When scaling large language models, developers and infrastructure architects increasingly run into the physical limits of hardware. Although the compute power of modern graphics processors keeps rising, bandwidth to memory and the physical capacity of that memory form a hard bottleneck. Within the architecture of modern transformer models, the attention mechanism is the driving force behind the ability to understand context, but it is at the same time the largest consumer of runtime memory. To address this challenge, several variants of the attention mechanism have been developed. One of the most important standards applied in modern models is grouped-query attention (GQA). In this article we analyze how the technique works, why it is a necessary intermediate step between earlier approaches, and how it directly affects the scalability and speed of systems working with large language models. ## A quick refresher: queries, keys, and values To understand why attention mechanisms require so much memory, we have to go back to the basics of the transformer architecture. In a standard attention layer, the input is projected into three different vectors per token: queries (Q), keys (K), and values (V). For a broader perspective, see also the article on [what a transformer is](https://leren.llmnet.nl/en/wat-is-een-transformer). The workings of these three vectors can be summarized as follows: - Query: The search request of the current token. This token 'asks' the rest of the context which information is relevant to its own meaning. - Key: The label or index of each token in the context. The query of the current token is compared with the keys of all preceding tokens to determine relevance. - Value: The actual substantive information each token carries. Once the relevance (the attention score) between a query and a key has been computed, that score is used to weight and combine the associated values. During the autoregressive generation process (where the model generates token by token), there is an important difference in lifespan between these vectors. The query is only needed for the token being processed at that specific moment. Once the attention computation for the current token is complete and the next token is generated, that specific query is no longer needed. The keys and values of all processed tokens, however, do have to be retained. For every new token to be generated, the model has to relate it once more to all preceding tokens. Recomputing these keys and values for every new token would lead to an unacceptable amount of duplicate work. The solution is to store these vectors in memory, a technique known as KV caching. For an in-depth look at this structure, you can consult the article on the [structure of KV caching](https://leren.llmnet.nl/en/kv-caching-opbouw) . Caching prevents us from having to recompute the keys and values of steps 1 through 99 at step 100. It solves a compute problem, but immediately introduces a substantial memory problem. ## The memory cost of the KV cache with long contexts The stored keys and values are the primary cause of high memory demands during a model's generation phase. Where the model parameters (the weights) remain statically loaded in memory and unchanged regardless of conversation length, the KV cache grows dynamically with every step in the interaction. That growth is directly tied to the length of the context. The scaling formula in words: The size of the KV cache per active user session grows linearly with the number of tokens in the context, the number of layers in the model, the hidden dimension of the network, and the precision at which the numbers are stored (such as 16-bit floating-point formats). If a model has dozens of layers, for example, and each layer has dozens of attention heads with their own dimension, then long documents or extended chat sessions lead to an enormous amount of data. In a traditional setup with multi-head attention (MHA), every individual attention head has its own set of keys and values. That means that in a model with 32 heads, 32 key vectors and 32 value vectors have to be stored in memory for every processed token, and again for every layer of the model. When the context length grows into the tens of thousands of tokens, the KV cache of a single active user can become larger than the model parameters themselves. That not only limits the maximum context a model can process, but also drastically reduces the number of users that can be served simultaneously on the same hardware. For an extensive analysis of the challenges around context limits, see the article on the [explanation of the context window](https://hub.llmnet.nl/en/context-window-uitleg). ## The evolution: from multi-head to multi-query attention To understand how grouped-query attention solves this problem, we have to look at the two extremes that preceded it: multi-head attention (MHA) and multi-query attention (MQA). ### Multi-head attention (MHA) In the traditional transformer architecture, as introduced in the original documentation on attention, multi-head attention is used. Here every query head has a corresponding, unique key head and value head. If the model has 32 query heads, it also has 32 key heads and 32 value heads. This allows each head to focus on completely different aspects of the input. One head might learn syntactic relationships (such as linking a verb to a subject), while another focuses on semantic connections across longer distances. Although this delivers high expressiveness and accuracy, it also results in the maximum load on the KV cache. ### multi-query attention (MQA) In response to the memory problems of MHA, multi-query attention (MQA) was proposed. MQA takes the other extreme to save memory. In this setup the model still keeps multiple query heads (32, for example), but all of these query heads share a single key head and a single value head per layer. That means that regardless of the number of query heads, only one key and one value vector per token has to be stored in memory. The saving is enormous: the size of the KV cache is reduced by a factor equal to the number of heads. In a model with 32 heads, the KV cache shrinks by 96.875%. This makes it possible to work with very large batches and to extend context length considerably. The downside, however, is a loss of quality. Because all query heads are forced to look for information in exactly the same compressed key and value space, the model loses part of its ability to form complex relationships. This often shows up as reduced performance on tasks requiring deep logic, code generation, or precise text extraction. ## Grouped-query attention (GQA) as the optimal middle ground Grouped-query attention (GQA) is designed to combine the advantages of both extremes. It serves as an adjustable middle ground. Instead of giving each query head its own key/value head (MHA), or having all query heads share a single key/value head (MQA), GQA divides the query heads into groups. Within each group, the query heads share a common key head and value head. The ratio between the number of query heads and the number of key-value heads determines the degree of compression and can be tuned precisely by the model's architects. Suppose, for example, that we have a model with 32 query heads. We can divide these into 8 groups. Each group then contains 4 query heads. For every group of 4 query heads, one key head and one value head are projected and stored in the KV cache. In total this results in 8 key heads and 8 value heads for the entire layer, instead of the 32 that MHA would require. Attention type | Query heads (Q) | Key/value heads (K/V) | Relative KV cache size | Quality retention | Multi-head (MHA) | 32 | 32 | 100% (reference) | Maximum | Grouped-query (GQA-8) | 32 | 8 | 25% | Very high | Grouped-query (GQA-4) | 32 | 4 | 12,5% | High / medium | Multi-query (MQA) | 32 | 1 | 3,125% | Reduced | By applying this grouping, a large part of the model's expressive power is retained. Different groups can still focus on different aspects of the input text, while the memory footprint of the KV cache falls drastically. The ratio between the number of query heads and key-value heads acts here as a direct physical control knob. ## The ratio and the arithmetic saving The saving GQA delivers can be traced directly to the chosen ratio. There is no abstract or variable performance gain; the reduction in KV cache size can be determined mathematically and exactly from the model architecture. If we define the ratio as: Reductiefactor = Aantal Query-koppen / Aantal Key-Value-koppen then a configuration with 8 query heads per key-value head (a ratio of 8:1) means the KV cache is exactly eight times smaller than in a traditional multi-head attention architecture with the same number of query heads. This ratio is consistent across all layers of the model and is independent of the actual length of the context. Whether the context is 1,000 or 100,000 tokens long, the storage capacity required for the cache in this scenario is always exactly 12.5% (one eighth) of what MHA would have needed. ## Memory bandwidth and the generation phase An important aspect of GQA is that the gain translates mainly into higher throughput during the generation phase (generating tokens), and less during the prefill phase (processing the initial prompt). This has to do with the way processors such as GPUs perform computations. During the prefill phase, the model processes the entire input prompt at once. This is an operation that is heavily compute-bound . The processor performs large matrix multiplications in which the compute cores (the ALUs) are constantly at work. The time needed to load data from memory pales next to the time needed for the computations themselves. GQA offers only a marginal speed gain here. During the generation phase (decoding), the situation changes completely. The model generates token by token. For every new token, the processor has to load the full model parameters and the entire KV cache of all preceding tokens in order to perform a single computation. This process is extremely memory-bandwidth-bound. The GPU's compute cores are largely waiting for the required data to be moved from the slower High Bandwidth Memory (HBM) to the fast on-chip SRAM. By shrinking the KV cache by a factor of 4 or 8 thanks to GQA, considerably less data has to be moved across the bus at every step. That relieves memory bandwidth directly, which speeds up generation per token. For more on quantifying these processes and latency, see the article on [measuring speed](https://benchmark.llmnet.nl/en/snelheid-meten). ## Practical advantages at deployment In practice, developers and system administrators notice the advantages of GQA in several ways when hosting language models: - Higher batch size (throughput): Because the KV cache takes up far less memory per user, more data fits on a single graphics card. That means a server can handle many more simultaneous conversations before memory fills up. - Longer context lengths within reach: Models can process longer documents without out-of-memory (OOM) errors occurring. The physical limit of what fits in VRAM shifts considerably upward. - More efficient behavior at scale: In large-scale systems, the reduction in required memory translates directly into lower infrastructure costs, because fewer physical GPUs are needed to handle the same number of requests per second. ## The quality trade-off and evaluation Although GQA offers considerable efficiency advantages, it is important to keep a critical eye on the quality of model output. Sharing keys and values between different query heads inevitably means information is compressed. The model loses a small part of its fine-grained expressiveness. In practice, however, this quality trade-off turns out to be minimal at a well-chosen ratio (such as 4:1 or 8:1). Scientific evaluations show that models using GQA often deliver performance very close to their MHA equivalents, while the operational gain is substantial. Even so, the impact can vary per task. On complex reasoning tasks, mathematical computations, or highly specific programming tasks, the quality loss can be more noticeable than in creative writing or text summarization. Developers should therefore always evaluate the suitability of a GQA model against specific benchmarks representative of their final application. ## Interaction with other optimization techniques Grouped-query attention does not stand alone; it is part of a broader ecosystem of techniques for improving the efficiency of language models. The gain from GQA can be compounded with other methods for shrinking the KV cache, for example. A common combination is GQA with quantization. Where GQA reduces the number of vectors in the cache by grouping heads, quantization reduces the precision of the remaining vectors (by storing them as 8-bit or 4-bit numbers instead of 16-bit floating-point numbers, for example). That lowers the memory footprint per element. For a detailed explanation of how quantization techniques work on model parameters and activations, we refer you to the article on [quantization explained](https://gids.llmnet.nl/en/kwantisatie-uitgelegd). Together with advanced memory allocation techniques such as PagedAttention, GQA allows modern language models to run on relatively accessible hardware without sacrificing usability. ## Further reading - [The attention mechanism explained in detail](https://leren.llmnet.nl/en/attention-uitgelegd) - [The structure and workings of KV caching](https://leren.llmnet.nl/en/kv-caching-opbouw) - [What is a transformer architecture?](https://leren.llmnet.nl/en/wat-is-een-transformer) - [Quantization of LLMs explained](https://gids.llmnet.nl/en/kwantisatie-uitgelegd) - [Understanding the context window of language models](https://hub.llmnet.nl/en/context-window-uitleg) - [Measuring the speed and latency of LLMs](https://benchmark.llmnet.nl/en/snelheid-meten) llmnet.nl - learning and explanations about language models