Quantization (1 blogmarks)

In the Inference Engineering episode from the latent.space podcast, they referred to NVFP4 a bunch of times. That’s a quantization method. Quantization is the process of optimizing an LLM’s parameters so that it can fit better and perform better on broader sets of hardware. Quantization reduces the memory footprint of a model by finding opportunities to use lower-precision floating point values without sacrificing too much fidelity.

NVFP4 is a 4-bit floating-point format for NVIDIA Blackwell GPUs. It reduces memory bandwidth and storage for inference while keeping accuracy close to higher-precision formats.

Unlike uniform INT4 quantization, NVFP4 keeps floating-point semantics with a shared exponent and a compact mantissa, which improves dynamic range. Blackwell Tensor Cores support mixed-precision execution across FP16, FP8, and FP4, so models can use FP4 for weights and activations while accumulating in higher precision (typically FP16).