NVIDIA has recently launched Cosmos DiffusionRenderer, a new video diffusion framework designed to achieve high-quality image and video re-lighting and de-lighting. This technology is a major update to NVIDIA's original DiffusionRenderer method, achieving higher quality rendering through an improved data curation process.
To use Cosmos DiffusionRenderer, users need to meet some basic installation requirements, including Python 3.10 and an NVIDIA GPU with at least 16GB of VRAM. It is recommended to use a graphics card with at least 24GB of VRAM. In addition, users need to install the NVIDIA driver and CUDA 12.0 or higher, and ensure there is at least 70GB of free disk space.
Users can set up by creating a conda environment named "cosmos-predict1" and installing the relevant dependency packages. After installation, users need to download the model weights, which can be obtained from Hugging Face. After downloading, users can start inference, using DiffusionRenderer to perform de-lighting and re-lighting on images.
In image inference, users can use the trained inverse rendering model to extract G-buffer information such as base color, normal, and depth from the input image. Through command-line operations, users can easily process images located in a specific folder and save the results to a specified output folder.
After completing the inverse rendering, users can use the forward renderer to re-light the image. At this point, users can choose custom environment lighting maps for rendering, producing different re-lighting effects.
If users want to process videos, they can first extract the frames from the video and then perform inverse rendering and re-lighting on each frame sequentially. The entire process supports the selection of multiple environment light sources and can generate corresponding re-lighting videos.
Cosmos DiffusionRenderer not only provides users with great flexibility and creativity but also significantly improves the rendering quality. The release of this technology marks another major advancement in video rendering technology and is expected to play an important role in various visual effects creation in the future.
Project: https://github.com/nv-tlabs/cosmos1-diffusion-renderer
Key Points:
🌟 This technology is a major upgrade to NVIDIA's original DiffusionRenderer, offering higher quality image and video rendering.
💻 Users need to install Python 3.10 and an NVIDIA GPU with at least 16GB of VRAM, and create a related conda environment.
🎥 Supports de-lighting and re-lighting of images and videos, and can render using various environment lighting maps.