Quantization explained: how models shrink with INT4 and FP8
Large language models contain tens to hundreds of billions of parameters. During the training phase, these weights are typically stored as 16-bit floating-point numbers (such as FP16 or BF16). For a model with 70 billion parameters, this means that roughly 140 gigabytes of video memory (VRAM) must be reserved just for the weights, not counting context processing. Quantization is the mathematical technique in which the numerical precision of these parameters is lowered to, for example, 8-bit or 4-bit representations. This significantly shrinks the memory footprint and allows computations to be performed faster on specialized hardware.
The mathematical core: linear projection and scale factors
At its core, quantization transforms a continuous set of high-precision numbers into a discrete set of lower-precision numbers. When we switch from a 16-bit floating-point representation to an 8-bit or 4-bit integer, we need to project a continuous range of values onto a limited number of integers. For an 8-bit integer (INT8), there are 256 possible levels (-128 to 127), while a 4-bit integer (INT4) can only take on 16 different values (-8 to 7 or 0 to 15).
The simplest form of this is uniform symmetric quantization. Here, we first determine the absolute maximum value of the weight matrix, denoted as alpha = max(|W|). The scale factor (scale factor) S is then calculated by dividing this maximum by the maximum representable integer value q_max (for example, 127 for signed INT8). Quantizing a weight w is done using the formula:
# Berekening van uniforme symmetrische kwantisatie
q = clip(round(w / S), -q_max, q_max)
w_gereconstrueerd = q * SBecause the original values are continuous and the converted values are discrete, a rounding error inevitably occurs: the quantization error. The goal of advanced quantization algorithms is to distribute this error across the matrices so that the final result of the matrix multiplications deviates as little as possible from the original mathematical state.
Numerical formats compared: INT8, INT4, and FP8
Not all quantization uses integers. Where INT4 and INT8 strictly use integers, FP8 retains a floating-point structure with an explicit sign bit, exponent bits, and mantissa bits (fraction). This allows FP8 to handle number ranges spanning a large dynamic range much better.
Within FP8, there are two important standards:
- E4M3 (1 sign bit, 4 exponent bits, 3 mantissa bits): Offers higher numerical precision at the cost of dynamic range. This format is extremely well suited for the weights and activations during the forward pass.
- E5M2 (1 sign bit, 5 exponent bits, 2 mantissa bits): Has the same exponent structure as FP16, giving it a much larger dynamic range. This format is mainly used when gradients need to be tracked or when activation values fluctuate strongly.
| Format | Bits per parameter | Memory reduction vs. FP16 | Dynamic range | Typical use |
|---|---|---|---|---|
| FP16 / BF16 | 16 bits (2 bytes) | 1x (reference) | Very high | Training and unquantized baseline |
| FP8 (E4M3) | 8 bits (1 byte) | 2x | Medium | Modern inference and FP8 mixed-precision training |
| INT8 | 8 bits (1 byte) | 2x | Low (uniform) | Standard enterprise inference and serving |
| INT4 (AWQ / GPTQ) | 4 bits (0.5 byte) | 4x | Very low (discrete steps) | Local inference and memory-critical systems |
Post-Training Quantization (PTQ) versus Quantization-Aware Training (QAT)
Quantization can be applied at two points in a model's lifecycle: after the fact on an already trained model, or directly during the training process.
Post-Training Quantization (PTQ) is the most widely used method due to its low computational cost. An existing FP16 model is analyzed using a small calibration set (for example, a representative collection of a few hundred texts). The algorithm measures how the activations move through the layers, determines the optimal scale factors per layer or per matrix block, and converts the weights to the target format. This process typically takes only a few minutes to a few hours.
Quantization-Aware Training (QAT) models the quantization error already during training or fine-tuning. Because the rounding function is non-differentiable (the derivative is zero everywhere except at jump points), QAT uses what is known as a Straight-Through Estimator (STE). During the forward pass, the weights are virtually quantized to simulate the disruption, but during the backward pass, the gradients are passed directly to the underlying FP32 weights. This allows the model to learn to compensate for the precision losses. At extremely low bit rates (such as 2-bit or aggressive 4-bit), QAT delivers significantly better performance than PTQ, but requires considerable computing power and a full training pipeline.
The problem of activation outliers
When language models grow beyond a certain size (often around 6 to 13 billion parameters), a specific phenomenon occurs: emergent activation outliers. In specific dimensions of the hidden layers, certain activation values suddenly spike to values dozens of times larger than the average. Although these outliers make up less than 0.1% of all activations, they are crucial for the network's syntactic coherence and reasoning ability.
In classic INT8 or INT4 quantization, these outliers force the scale factor S upward. This compresses all remaining 99.9% of normal values into just a handful of discrete levels, leading to a catastrophic loss of precision and the collapse of the perplexity score. Modern quantization architectures solve this using methods such as block-wise scaling (per-channel or per-group quantization with block sizes of, for example, 32 or 128 values) or by processing the outliers separately in FP16 while the rest goes to low precision (as applied in LLM.int8()).
Modern algorithms under the microscope: GPTQ and AWQ
To successfully bring models to 4-bit without significant quality loss, specialized algorithms have been developed that handle error compensation more intelligently.
GPTQ (Generalized Post-Training Quantization) is based on the classic Optimal Brain Surgeontheory. The algorithm quantizes weights column by column. As soon as a weight is rounded to a discrete 4-bit value, the resulting error difference is immediately compensated for by adjusting the not-yet-quantized weights in the same row. To do this, GPTQ uses the inverse Hessian matrix of the activations. This mathematical mechanism ensures that rounding errors actively cancel each other out as the matrix is processed further.
AWQ (Activation-aware Weight Quantization) takes a different approach. The algorithm observes that not all weights are equally important; weights corresponding to channels with large activations have the greatest influence on the final model output. Instead of treating all weights equally, AWQ identifies the top 1% most important weights based on activation magnitudes. By mathematically scaling up these specific channels before quantization, the relative rounding error on the most critical connections is minimized, without needing to keep mixed-precision matrices.
Impact on latency, memory bandwidth, and compute power
Lowering the precision has direct consequences for hardware efficiency when running a model. To understand how these hardware optimizations play out in everyday use, it's illuminating to read about what happens under the hood during AI inference.
In LLM inference, we need to distinguish between two separate phases:
- Prefill phase (prompt processing): The model processes the entire input prompt in parallel. This phase is compute-bound: the graphics processor (GPU) runs at full compute power to perform matrix multiplications. Formats such as FP8 and INT8 deliver a direct doubling of compute speed (TFLOPS or TOPS) on tensor cores here.
- Decode phase (token generation): The model generates token by token. Because only one vector needs to pass through all parameters per step, this phase is memory-bandwidth bound. The compute units must constantly wait for the weights to be moved from VRAM to the processor chips.
Because INT4 weights require four times less data than FP16, the memory bus only needs to transport a quarter of the data volume per generated token. This significantly increases generation speed (tokens per second) on memory-limited systems, even if the INT4 weights have to be converted back to FP16 on the fly during computation (weight-only quantization).
The limits of compression: measuring quality loss
Quantization is not a free operation. As the compression ratio increases, imperfections appear. To determine the quality of a quantized model, two indicators are primarily considered: perplexity and task-specific benchmarks.
Perplexity measures how surprised a language model is by an unseen reference text. Lower perplexity indicates better language understanding. When we compress a model from FP16 to FP8 or INT8, perplexity generally increases only marginally (often less than 0.05 to 0.1 points). With INT4, the damage remains very limited with algorithms such as AWQ and GPTQ. Once we drop to 3-bit or 2-bit, however, perplexity shows exponential deterioration: the model loses its coherence, starts mixing up grammatical structures, and more often misses the mark on logical inferences.
In addition, specific computing tasks are more sensitive to rounding errors than others. Creative writing and general summaries hold up excellently under 4-bit quantization. Complex tasks such as mathematical reasoning, formal code syntax, and long reasoning chains (chain-of-thought) more often show degradation under aggressive quantization, because subtle numerical relationships in the attention mechanisms are lost.
Practical implementation considerations
When designing a production system, the use case determines which format is optimal. Those who want to deploy models locally or on dedicated servers can read in the practical guide on local quantization which hardware configurations match different formats.
In modern infrastructures, we broadly see the following division of tasks:
- Datacenter serving at scale: FP8 has become the industry standard on modern architectures (such as Nvidia Ada Lovelace, Hopper, and Blackwell). It offers a perfect balance between hardware-accelerated compute, minimal quality loss, and halving the required memory without complex weight deconstruction.
- Local workstations and edge devices: INT4 formats (such as GGUF for CPU/Metal and AWQ/EXL2 for GPUs) dominate the local AI world. They allow developers to run advanced 70B models smoothly on consumer hardware with 32 to 48 GB of memory.
- Extreme compression: Techniques such as 2-bit and 3-bit sub-integer quantization (such as AQLM or BitNet 1.58b) require specific training from scratch or very intensive QAT cycles to remain practically usable.
