FP32, or single-precision floating-point, uses 32 bits to represent each number: 1 sign bit, 8 exponent bits, and 23 mantissa bits. It's the historical default for scientific computing and was the dominant precision for deep learning before the rise of mixed-precision training in 2017-2018.
In 2026, FP32 remains common in three areas: optimizer state (Adam keeps running averages in FP32 even when gradients are FP16/BF16), critical numerical operations like loss scaling, and scientific compute workloads like molecular dynamics where FP32 or FP64 accuracy is essential.
Most modern AI training uses mixed-precision schemes where forward and backward passes run in FP16 or BF16 but selected operations remain in FP32 for numerical stability. Pure FP32 training is rare for production AI workloads because it consumes 2x more VRAM and 2-3x more compute than mixed precision with negligible accuracy benefit.