Quantization converts a model's weights and activations from 32-bit floating point to 8-bit integers, cutting memory by 4x and often making inference faster. It is the single most important trick for fitting real models into browsers, where download size and RAM are hard constraints. The surprising part is how little accuracy is usually lost — and understanding the math explains why.

What FP32 and INT8 Actually Are

A 32-bit float uses 1 sign bit, 8 exponent bits, and 23 mantissa bits. That gives it a dynamic range from roughly 1e-38 to 1e38 and about 7 significant decimal digits. An 8-bit integer has exactly 256 values, conventionally -128 to 127. Quantization is the problem of mapping the continuous range of real numbers a tensor actually takes onto those 256 buckets with minimum damage.

Weights in trained networks cluster tightly around zero and are roughly symmetric, and activations after a ReLU are non-negative. Those two observations drive the two standard schemes.

Symmetric Quantization

Symmetric quantization uses one scale factor and no zero point:

scale = max(|x|) / 127
q = round(x / scale)
x_approx = q * scale

Because the mapping is centered at zero, it suits tensors whose values are symmetric about zero — which is most weight matrices. In Python:

import numpy as np

def quantize_symmetric(x):
    scale = np.max(np.abs(x)) / 127.0
    q = np.clip(np.round(x / scale), -128, 127).astype(np.int8)
    return q, scale

As an example, a weight tensor in the range [-3.1, 2.7] gets scale = 3.1/127 ≈ 0.0244. The largest representable error is half a bucket, about 0.012, which is small relative to the values themselves. ReLU.chat auto-quantizes its policy network — an MLP with 25 inputs, two hidden layers of 128 and 64, and 6 action heads, about 13,079 parameters in total — to symmetric INT8 at load time, cutting its memory footprint by roughly 4x.

Asymmetric Quantization with a Zero Point

Symmetric quantization wastes half its range on negative values that never occur. Activations after a ReLU are non-negative, so an asymmetric scheme that shifts the mapping is more precise:

scale = (max - min) / 255
q = round(x / scale) + zero_point
x_approx = (q - zero_point) * scale

The zero point is the integer corresponding to the real value zero. With 255 buckets spanning the observed range instead of 127 spanning only half of it, the effective resolution roughly doubles for non-negative tensors. This is why production runtimes typically use symmetric quantization for weights and asymmetric for activations.

Calibration Matters

For weights, min/max is exact — you know every value. Activations are a different story: their ranges are only known after running the model. Calibration is the step where you feed a representative set of inputs through the model and record the observed min and max of every activation tensor. The choices:

  • Min/max calibration is simplest but lets a single outlier stretch the scale and waste buckets on the rest of the distribution.
  • Percentile calibration clips the extreme tail, trading a few outliers for better resolution on the bulk of values.
  • Entropy (KL-divergence) calibration chooses the clipping threshold that minimizes information loss between the original and quantized distributions, which is what TensorRT's calibration uses.

Per-tensor scaling uses one scale for an entire tensor; per-channel scaling gives each output channel its own scale, which usually preserves accuracy much better for weights at the cost of slightly more complex kernels. When you run a quantized model through ONNX Runtime, as ReLU.chat does with its ~22 MB quantized MiniLM model, the runtime applies these scales internally during inference.

How Quantized Inference Works

A quantized matrix multiply keeps the 8-bit weights and 8-bit inputs but accumulates in higher precision. Each product q_a * q_w is at most 127 * 127 = 16129, comfortably within a 16-bit or 32-bit accumulator, and the sum of hundreds of such products fits an INT32. The scale factors are applied once per output element rather than per product, so the integer core of the operation is a tight loop over INT8 loads and multiply-adds — exactly the pattern SIMD units are good at.

The memory savings are why this matters on-device: every weight goes from 4 bytes to 1 byte. That 4x reduction shrinks download size, page-cache pressure, and memory bandwidth, and on memory-bound kernels the speedup can be larger than the arithmetic would suggest. ONNX Runtime's INT8 kernels handle all of this transparently, which is why you can quantize a model offline and run it in the browser without writing the low-level math yourself.

Why 8 Bits Usually Survives — and When It Does Not

Quantization error behaves like small structured noise, and neural networks are robust to it for two reasons. First, individual weights are highly redundant — many near-duplicate values average out errors in the dot product. Second, the error is bounded by half a bucket per value, so the accumulated error in a sum of 384 products stays small relative to the signal. Quantization-aware training (QAT), which simulates rounding in the forward pass, can recover most of the remaining gap.

The failure modes are also known. A few extreme outlier weights can stretch the scale so far that most buckets go unused, and very small or very sensitive models have less redundancy to absorb error. The standard mitigations: clip outliers with percentile calibration, switch to per-channel scales, keep the most sensitive layers in higher precision, or train with QAT. The accuracy cost is task-dependent, so the honest engineering answer is to measure it on your own data. In practice, small MLPs like a policy network quantize almost losslessly, and sentence transformers like MiniLM tolerate INT8 well enough that the model still fits the browser's memory budget.

Key Takeaway

INT8 quantization maps a tensor's range onto 256 integer buckets — symmetric for weights, asymmetric with a zero point for activations — and calibration determines how well the mapping fits the data. Because networks tolerate small bounded errors, 8-bit models routinely keep accuracy while using a quarter of the memory, which is exactly what makes on-device inference practical.