SoL-Refiner Upscales Low-Resolution Videos to 4K Quality in a Single Step

Background

High-resolution videos have a large number of tokens spanning space and time, causing generation costs to grow rapidly. Therefore, generating videos at a lower resolution first and then improving the quality is used as a realistic alternative, but existing multi-step quality improvement methods themselves become processing time bottlenecks.

Method

The research team used a three-stage training procedure: continued training on high-resolution data, post-adjustment via reinforcement learning (a training method that refines the model based on rewards), and finally a distillation step that compresses multi-step tasks into a single step. They also built a benchmark called Refiner-Bench, created from the outputs of various video generators, to evaluate performance by feeding the same inputs for comparison.

Main results

At 2K resolution, the single-step SoL-Refiner outperformed all comparative external refiners in the average scores of VBench and UniPercept. At 3840x2176 resolution, it also scored higher on both metrics than the three-step LTX-2.3 refiner, and when applying all acceleration techniques under 2K latency settings, it showed an 8.91-fold speedup compared to the same baseline.

Limitations & open points

The abstract does not mention the computational resources or costs required for training, nor the performance in actual service environments. Because the benchmark and speed measurements were performed under conditions configured by the authors and a 2K latency setting, results in other environments cannot be verified.

Glossary

• token — A data unit split into smaller pieces so that a model can process videos or text. Videos have a large number of tokens along spatial and temporal directions, increasing the computational load.
• refiner — A post-processing model that refines already generated videos to make their image quality clearer and more natural.
• denoising step — A single computational process that gradually transforms a noise-mixed state into a clean video. A higher number of steps usually takes more time.
• distillation — A training technique that teaches a model to achieve results close to the same level using fewer steps, instead of going through multiple steps.
• reinforcement learning (RL) — A training method that assigns scores (rewards) to a model's outputs and adjusts the model in the direction of increasing those scores.
• latency — The time it takes from inputting data to receiving the output. A smaller value means faster processing.