Paper
Research improving block diffusion language model training by incorporating future context
d-OPD: Future-Aware On-Policy Distillation for Block Diffusion Language Models
- What
- The authors propose d-OPD, a distillation training method that converts conventional autoregressive language models (which generate one token at a time) into block diffusion language models that generate multiple tokens in parallel.
- Key result
- Experiments with Qwen3 models (0.6B to 8B) showed that d-OPD improved the average score across six benchmarks by up to 4.0 points compared to the existing OPDLM method, while reducing training time by 1.35 to 1.58 times.
- Why it matters
- The abstract suggests that generating multiple tokens simultaneously in blocks has the potential to increase text generation speed, and this study demonstrates that the efficiency of the training process for creating such models can be improved.AI summary
