The NVIDIA H100 SXM is the flagship Hopper architecture datacenter GPU, announced at GTC in March 2022 and shipping to customers beginning in late 2022. It represents a generational leap from the A100, introducing the Hopper architecture with significant innovations in AI training and inference capabilities.
The H100 is built on TSMC's custom 4N process node and contains 80 billion transistors in the GH100 GPU die. It features 80GB of HBM3 memory with 3.35 TB/s of bandwidth - the first GPU to use HBM3. The chip includes 16,896 CUDA cores, 528 4th-generation Tensor Cores, and 50MB of L2 cache, a 4x increase over A100.
The SXM5 form factor is designed for NVIDIA's HGX H100 baseboard, supporting 8-GPU configurations with 4th-generation NVLink providing 900 GB/s bidirectional bandwidth between GPUs - a 1.5x increase over A100's NVLink. The full NVLink mesh enables all 8 GPUs to communicate simultaneously with full bandwidth. TDP is 700W, requiring liquid cooling in most deployments.
Key architectural innovations include the Transformer Engine, which uses FP8 precision with automatic mixed-precision to double AI training throughput versus FP16. The H100 also introduced DPX instructions for dynamic programming algorithms, accelerating genomics and graph analytics workloads. It retains Multi-Instance GPU (MIG) capability, now supporting up to 7 isolated instances. The H100 SXM is the most widely deployed GPU for large-scale AI training as of 2024.