LeapQuant: Accelerating AI Inference by Compressing Linear Attention States to 8 Bits

Background

Recently, LLMs are increasingly adopting linear attention (such as Gated DeltaNet, Kimi Delta Attention, etc.) which compresses context into a fixed-size recurrent state instead of traditional attention. While this method significantly reduces the cost of processing long contexts, the process of repeatedly reading and updating states remains a major bottleneck in inference. Quantization (a technique representing numbers with fewer bits) is a natural way to reduce this cost, but accumulated rounding errors and extreme value rows and columns within states can severely degrade model quality.

Method

LeapQuant reduces error accumulation through window-wise quantization, which quantizes states just once per window consisting of multiple tokens. Within the window, outputs are calculated using a low-bit fixed state together with updates temporarily stored in high precision. In addition, some of the largest extreme values in the state are stored separately as high-precision Compensator Tokens, and the remaining residuals are smoothed before quantization to reduce errors generated in each quantization step.

Main results

Experiments on Qwen, Kimi, and GLM family models report significant reductions in memory and computational costs. Accuracy remained comparable to the FP32 baseline, showing an average kernel-level speedup of 2.05 to 3.70 times and an end-to-end inference speedup of 1.47 times on NVIDIA B200, RTX PRO 6000, and RTX 5090 GPUs.

Limitations & open points

The abstract alone does not reveal exact figures for accuracy degradation, specific benchmarks used for evaluation, or differences by model size. Furthermore, since experiments are limited to the presented model families and GPUs, the effects on other hardware or models cannot be known from the abstract.

Glossary

• LLM — A large-scale AI model trained on massive text data to understand and generate text.
• Attention — A computational method that determines how much focus the model should place on different parts of the input sentence.
• Linear Attention — An attention method designed so that computational complexity grows almost linearly even as the context gets longer.
• Recurrent State — A storage space that holds a summary of the context read so far in a fixed size and updates it whenever a new token arrives.
• Quantization — A technique that reduces memory usage and computational cost by representing numbers with fewer bits.
• 8-bit Quantization — A method of representing a single number with 8 bits (256 levels), using less memory than the original.
• FP32 — A 32-bit floating-point format, typically serving as the original precision baseline for accuracy.
• Kernel — A unit of computation code executed on the GPU, where kernel-level speed refers to performance at this unit.
• End-to-end inference — The measured speed of the entire process from receiving input to generating the final answer.
• Training-free — A method that can be applied directly to existing models without retraining them.