The NVIDIA Transformer Engine is a cutting-edge library that accelerates transformer model performance on NVIDIA GPUs during training and inference phases. The engine leverages the capabilities of 8-bit floating point (FP8) precision on the latest NVIDIA Hopper and Ada Lovelace architecture GPUs, significantly accelerating performance while reducing memory consumption. The Transformer Engine's FP8 capabilities on the NVIDIA H100 Tensor Core GPU result in a 60% boost in performance compared to traditional FP16 operations, and achieve 3x the speed of the A100 GPU when using BF16 precision. This allows for the use of larger models that were previously constrained by memory limitations, unlocking significant performance gains and memory savings.