Recently, a research team from the University of Hong Kong, Harbin Institute of Technology, and Zhejiang University jointly released an innovative technology called "OmniPart." This technology aims to address the shortcomings of current mainstream 3D generation models in terms of detail construction, making the generated 3D models not only visually realistic but also with clear component structures.
Existing 3D generation models such as DreamFusion and TRELLIS can generate seemingly complete 3D shapes, but lack internal part structures, making it difficult to meet the needs of detailed modifications. For example, when adjusting the arm of a 3D character or the gears of a mechanical device, using these models becomes very challenging because they cannot provide precise control at the part level. To solve this problem, the introduction of OmniPart is aimed at achieving independence between parts while forming a reasonable whole.
OmniPart adopts an innovative two-stage generation framework. The first stage is structural planning, which uses a autoregressive model and flexible 2D part masks to plan the spatial layout of 3D parts. The autoregressive model generates the bounding boxes of each part step by step based on existing sequence information. The generation of bounding boxes is both flexible and efficient. To ensure that the generated bounding boxes cover the corresponding object parts, OmniPart also designs a novel part coverage loss function, thereby improving the generation effect.
In the second stage, OmniPart efficiently generates high-quality 3D representations for all parts based on the part layout generated in the first stage, using the pre-trained overall 3D generator TRELLIS. This stage particularly focuses on semantic awareness of each part and consistency of the overall structure, ensuring that the generated parts are semantically independent and structurally closely connected. In addition, OmniPart introduces a voxel dropout mechanism, further enhancing the accuracy and efficiency of the model.
This technology marks an important advancement in the field of 3D modeling and is expected to be widely applied in multiple creative fields such as game development and animation production.
Project: https://omnipart.github.io/
Key Points:
🌟 The OmniPart technology achieves the independence of 3D model parts, making creative design more flexible.
🔍 A two-stage generation framework using autoregressive models and part masks improves the accuracy of 3D modeling.
🚀 Innovative mechanisms such as voxel dropout enhance the model's performance in complex scenarios.
