BackgroundandRelatedWork

Background and Related Work

Quantization Techniques for Efficient LLM Inference and Training

The rapid increase in the size of large language models (LLMs) has necessitated the development of efficient quantization techniques to reduce memory and computational costs while preserving performance. Traditional uniform quantization methods are often insufficient for LLMs due to the wide dynamic range of weights and activations. Recent work has explored fine-grained quantization strategies such as vector-wise quantization and block-wise quantization. Vector-wise quantization partitions weight tensors into vectors and individually scales each vector, mitigating the loss in precision compared to scalar quantization. Block-wise quantization further extends this by operating on fixed-size blocks (e.g., 64 elements), enabling more hardware-friendly implementations.

In terms of scale handling, zero-point quantization and absolute maximum (absmax) quantization have been proposed as efficient alternatives for balancing dynamic range and quantization error. Zero-point quantization introduces an offset to minimize reconstruction error, whereas absmax scaling normalizes values based on the absolute maximum within a block or tensor. These techniques have become foundational in modern quantization libraries.

Data Types for 4-bit Quantization

Recent advances have introduced specialized low-precision formats tailored for LLM quantization. NormalFloat4 (NF4) is a 4-bit floating-point format designed for normally distributed weights, providing better approximation fidelity compared to standard linear quantization. FP4 represents another 4-bit floating-point format, offering a general-purpose 4-bit representation. Compared to NF4, FP4 is less optimized for the statistical properties of transformer weights but can be simpler to implement. These formats enable aggressive compression of LLMs without incurring severe performance degradation.

Quantization Libraries and Toolkits

Several high-performance libraries have been developed to support efficient inference and training with quantized models. BitsAndBytes is a widely used library offering CUDA-accelerated implementations for 8-bit optimizers, 8-bit inference (LLM.int8()), and 4-bit quantization (QLoRA). Notably, BitsAndBytes supports block-wise quantization, mixed-precision decomposition, and fine-grained outlier handling, enabling near-lossless inference with significant memory savings.

The QLoRA approach builds upon BitsAndBytes and introduces a 4-bit quantization scheme combined with low-rank adaptation layers (LoRA) for finetuning. QLoRA incorporates the NF4 data type and strategically preserves critical weights in higher precision, achieving state-of-the-art finetuning efficiency.

Unsloth extends these ideas further by supporting efficient 4-bit training through manually optimized kernels written in Triton. Unsloth creates a dynamic precision selection strategy: critical layers (e.g., attention output layers) are retained in 16-bit, while the rest of the model is quantized to 4-bit NF4 format. Parameters are stored using block-wise quantization (64-element blocks), and inference reconstructs tensors via the dequantize_block operation. This hybrid approach achieves substantial memory savings with minimal accuracy loss, enabling high-throughput training and inference on a broad range of NVIDIA GPUs.

Efficient Inference Frameworks

Efficient deployment of quantized models requires highly optimized inference frameworks. SGLang is a fast LLM/VLM serving system that integrates backend runtime and frontend language design to deliver controllable and scalable inference. The backend supports numerous optimizations, including radix attention for prefix caching, token attention (paged attention), speculative decoding, zero-overhead CPU scheduling, tensor parallelism, and structured output generation.

SGLang also incorporates quantization-aware features, supporting FP8, INT4, AWQ, and GPTQ quantization formats natively. Recent efforts have extended its capabilities to support mixed-precision 4-bit checkpoints generated by Unsloth, addressing challenges related to loading hybrid precision models and maintaining inference efficiency.