Exploring HiFloat8: A Tapered Format Complementing the FP8 Ecosystem for Robust Model Training

The quantitation to FP8 formats has significantly accelerated LLM training. Standard FP8 formats suffer from frequent gradient overflows and heavy reliance on complex dynamic scaling (Delayed Scaling), which often lead to training instability or suboptimal convergence in billion-parameter models. This session introduces HiFloat8 (HiF8) — a tapered precision format that offers an alternative approach to managing dynamic range. This "natural" alignment with neural network weight/gradient distributions allows HiF8 to capture high-magnitude outliers without the aggressive scaling required by standard FP8.We explore how HiF8 can works in the training and inference procedure.

We will demonstrate the implementation of HiF8 within the PyTorch ecosystem using torch.npu. They allow developers to evaluate performance of Hif8 on current-generation GPUs.
Also, we will give an analysis of training stability and final loss parity, highlighting specific scenarios, where HiF8 provides relatively the same accuracy and 1.5-1.7 times GEMM performance than FP16. Finally, we will share insights from our ongoing collaboration on dedicated hardware support for HiF8.