# Direct3D-S2 **Repository Path**: mirrors_trending/Direct3D-S2 ## Basic Information - **Project Name**: Direct3D-S2 - **Description**: Direct3D‑S2: Gigascale 3D Generation Made Easy with Spatial Sparse Attention - **Primary Language**: Unknown - **License**: MIT - **Default Branch**: main - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 0 - **Forks**: 0 - **Created**: 2025-06-16 - **Last Updated**: 2026-01-31 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README # Direct3D‑S2: Gigascale 3D Generation Made Easy with Spatial Sparse Attention (NeurIPS 2025)
Teaser image of Direct3D-S2
--- ## ✨ News - June 3, 2025: We are preparing the v1.2 release, featuring enhanced character generation. Stay tuned! - May 30, 2025: 🀯 We have released both v1.0 and v1.1. The new model offers even greater speed compared to FlashAttention-2, with **12.2Γ—** faster forward pass and **19.7Γ—** faster backward pass, resulting in nearly **2Γ—** inference speedup over v1.0. - May 30, 2025: πŸ”¨ Release inference code and model. - May 26, 2025: 🎁 Release live demo on πŸ€— [Hugging Face](https://huggingface.co/spaces/wushuang98/Direct3D-S2-v1.0-demo). - May 26, 2025: πŸš€ Release paper and project page. ## πŸ“ Abstract Generating high-resolution 3D shapes using volumetric representations such as Signed Distance Functions (SDFs) presents substantial computational and memory challenges. We introduce Direct3D‑S2, a scalable 3D generation framework based on sparse volumes that achieves superior output quality with dramatically reduced training costs. Our key innovation is the Spatial Sparse Attention (SSA) mechanism, which greatly enhances the efficiency of Diffusion Transformer (DiT) computations on sparse volumetric data. SSA allows the model to effectively process large token sets within sparse volumes, substantially reducing computational overhead and achieving a 3.9× speedup in the forward pass and a 9.6× speedup in the backward pass. Our framework also includes a variational autoencoder (VAE) that maintains a consistent sparse volumetric format across input, latent, and output stages. Compared to previous methods with heterogeneous representations in 3D VAE, this unified design significantly improves training efficiency and stability. Our model is trained on public available datasets, and experiments demonstrate that Direct3D‑S2 not only surpasses state-of-the-art methods in generation quality and efficiency, but also enables training at 10243 resolution with just 8 GPUs, a task typically requiring at least 32 GPUs for volumetric representations at 2563 resolution, thus making gigascale 3D generation both practical and accessible. ## 🌟 Highlight - **Gigascale 3D Generation**: Direct3D-S2 enables training at 10243 resolution with only 8 GPUs. - **Spatial Sparse Attention (SSA)**: A novel attention mechanism designed for sparse volumetric data, enabling efficient processing of large token sets. - **Unified Sparse VAE**: A variational autoencoder that maintains a consistent sparse volumetric format across input, latent, and output stages, improving training efficiency and stability. ## πŸš€ Getting Started ### Installation ### βœ… Tested Environment > πŸ’‘ *If you're setting up on Windows, check out [issue #11](https://github.com/DreamTechAI/Direct3D-S2/issues/11) and [issue #12](https://github.com/DreamTechAI/Direct3D-S2/issues/12). Big thanks to the contributors who helped get Direct3D-S2 working on Windows!* - **System**: Ubuntu 22.04 - **CUDA Toolkit**: [CUDA 12.1](https://developer.nvidia.com/cuda-12-1-0-download-archive) - **PyTorch**: Install `torch` and `torchvision` first. Make sure the PyTorch CUDA version matches your installed CUDA Toolkit. ```bash pip install torch==2.5.1 torchvision==0.20.1 --index-url https://download.pytorch.org/whl/cu121 ``` - **Torchsparse**: follow the [offical guide](https://github.com/mit-han-lab/torchsparse) or: ```bash git clone https://github.com/mit-han-lab/torchsparse cd torchsparse && python -m pip install . ``` - Install dependencies: ```bash git clone https://github.com/DreamTechAI/Direct3D-S2.git cd Direct3D-S2 pip install -r requirements.txt pip install -e . ``` ### Usage > Note: Generating at 512 resolution requires at least 10GB of VRAM, and 1024 resolution needs around 24GB. We don’t recommend generating models at 512 resolution, as it’s just an intermediate step of the 1024 model and the quality is noticeably lower. ```python from direct3d_s2.pipeline import Direct3DS2Pipeline pipeline = Direct3DS2Pipeline.from_pretrained( 'wushuang98/Direct3D-S2', subfolder="direct3d-s2-v-1-1" ) pipeline.to("cuda:0") mesh = pipeline( 'assets/test/13.png', sdf_resolution=1024, # 512 or 1024 remove_interior=True, remesh=False, # Switch to True if you need to reduce the number of triangles. )["mesh"] mesh.export('output.obj') ``` ### Web Demo We provide a Gradio web demo for Direct3D-S2, which allows you to generate 3D meshes from images interactively. ```bash python app.py ``` ## πŸ€— Acknowledgements Thanks to the following repos for their great work, which helps us a lot in the development of Direct3D-S2: - [Trellis](https://github.com/microsoft/TRELLIS) - [SparseFlex](https://github.com/VAST-AI-Research/TripoSF) - [native-sparse-attention-triton](https://github.com/XunhaoLai/native-sparse-attention-triton) - [diffusers](https://github.com/huggingface/diffusers) ## πŸ“„ License Direct3D-S2 is released under the MIT License. See [LICENSE](LICENSE) for details. ## πŸ“– Citation If you find our work useful, please consider citing our paper: ```bibtex @article{wu2025direct3ds2gigascale3dgeneration, title={Direct3D-S2: Gigascale 3D Generation Made Easy with Spatial Sparse Attention}, author={Shuang Wu and Youtian Lin and Feihu Zhang and Yifei Zeng and Yikang Yang and Yajie Bao and Jiachen Qian and Siyu Zhu and Philip Torr and Xun Cao and Yao Yao}, journal={arXiv preprint arXiv:2505.17412}, year={2025} } ```