Meta and researchers from the University of California, Berkeley have developed StreamDiT, a revolutionary AI model that can create 512p resolution videos in real-time at 16 frames per second, requiring only a single high-end GPU. Unlike previous methods that needed to fully generate a video clip before playback, StreamDiT enables real-time video stream generation frame by frame.
The StreamDiT model has 4 billion parameters and demonstrates impressive versatility. It can instantly generate videos up to one minute long, respond to interactive prompts, and even edit existing videos in real-time. In an impressive demonstration, StreamDiT successfully replaced a pig in a video with a cat in real-time while keeping the background unchanged.
Custom Architecture for Exceptional Speed
The core of the system lies in its custom architecture designed for speed. StreamDiT uses a mobile buffer technique, allowing it to process multiple frames simultaneously, processing the next frame while outputting the previous one. New frames start off noisy but gradually improve until they are ready to be displayed. According to the research paper, the system can generate two frames in about half a second, which after processing results in eight final images.
StreamDiT divides its buffer into fixed reference frames and short blocks. During the denoising process, image similarity gradually decreases, forming the final video frames.
Multi-functional Training and Acceleration Techniques
To enhance the model's generality, StreamDiT's training process covered various video creation methods, using 3,000 high-quality videos and a large dataset containing 2.6 million videos. The training was conducted on 128 Nvidia H100 GPUs, and researchers found that using block sizes ranging from 1 to 16 frames yielded the best results.
To achieve real-time performance, the team introduced a key acceleration technique, reducing the required computational steps from 128 to just 8, while minimizing the impact on image quality. StreamDiT's architecture is also optimized for efficiency, with information exchanged only between local regions rather than between every image element.
Performance Exceeding Existing Methods
In direct comparison tests, StreamDiT outperformed existing methods such as ReuseDiffuse and FIFO diffusion when handling videos with a lot of motion. Other models tend to create static scenes, while StreamDiT can generate more dynamic and natural motion.
Human evaluators assessed StreamDiT's performance in terms of action smoothness, animation completeness, frame-to-frame consistency, and overall quality. When tested on an 8-second, 512p video, StreamDiT ranked first in all categories.
Potential of Larger Models and Current Limitations
The research team also tried a larger 30 billion parameter model, which provided higher video quality, although its speed was still insufficient for real-time use. This suggests that StreamDiT's approach can be scaled to larger systems, indicating the potential for future high-quality real-time video generation.
Despite significant progress, StreamDiT still has some limitations. For example, it has limited "memory" of the first half of the video, and visible transitions may occasionally appear between different parts. Researchers stated that they are actively researching solutions to overcome these challenges.
Notably, other companies are also exploring the field of real-time AI video generation. For example, Odyssey recently launched a self-regressive world model that can adjust videos frame by frame based on user input, providing a more convenient interactive experience.
The emergence of StreamDiT marks an important milestone in AI video generation technology, signaling a broad future for real-time interactive video content creation.