# ndg-fitting
**Repository Path**: mirrors_intel/ndg-fitting
## Basic Information
- **Project Name**: ndg-fitting
- **Description**: Official implementation of the SIGGRAPH 2024 paper "N-Dimensional Gaussians for Fitting of High Dimensional Functions"
- **Primary Language**: Unknown
- **License**: MIT
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No
## Statistics
- **Stars**: 0
- **Forks**: 0
- **Created**: 2024-07-10
- **Last Updated**: 2026-05-02
## Categories & Tags
**Categories**: Uncategorized
**Tags**: None
## README
# N-Dimensional Gaussians for Fitting of High Dimensional Functions
Stavros Diolatzis, Tobias Zirr, Alexandr Kuznetsov, Georgios Kopanas, Anton Kaplanyan
### [Project Website](https://www.sdiolatz.info/ndg-fitting/)
Abstract: In the wake of many new ML-inspired approaches for reconstructing and representing high-quality 3D content, recent hybrid and explicitly learned representations exhibit promising performance and quality characteristics. However, their scaling to higher dimensions is challenging, e.g. when accounting for dynamic content with respect to additional parameters such as material properties, illumination, or time. In this paper, we tackle these challenges for an explicit representations based on Gaussian mixture models. With our solutions, we arrive at efficient fitting of compact N-dimensional Gaussian mixtures and enable efficient evaluation at render time: For fast fitting and evaluation, we introduce a high-dimensional culling scheme that efficiently bounds N-D Gaussians, inspired by Locality Sensitive Hashing. For adaptive refinement yet compact representation, we introduce a loss-adaptive density control scheme that incrementally guides the use of additional capacity towards missing details. With these tools we can for the first time represent complex appearance that depends on many input dimensions beyond position or viewing angle within a compact, explicit representation optimized in minutes and rendered in milliseconds.
## Code Notes
This codebase was developed and tested on Windows, we would like to support more platforms but for now we haven't tested this on them.
### Motivation
This code base is meant to be used to replace the usage of neural networks (mainly MLPs) with N-Dimensional Gaussian mixtures to improve training from hours to minutes.
Ideal scenarios are cases where the target application has many input dimensions with strong inter-dependencies. We demonstrate two scenarios:
* Global Illumination with 10D+: MLPs have been used to turn G-Buffer renderings into global illumination images in previous work but usually took many hours. Our mixture can be trained in minutes.
* Radiance Fields with a focus on view-dependcy 6D: In this application we are inspired by 3D Gaussian Splatting but instead of considering only the world xyz we consider it together with the view direction xyz allowing the Gaussians to represent complex view dependent effects. In this case we don't evaluate the whole mixture but first project to 3D and then use splatting to render the final image.
## Cloning the repository
Use `recursive` when cloning the repo:
```
git clone --recursive https://github.com/intel/ndg-fitting.git
```
## Code Overview
The codebase includes application specific scripts and code that is common between applications. Here is a list of the main scripts and what they represent:
* `generators\gaussian_generator`: The implementation of the N-Dimensional Gaussian Generator which has the implementation of the optimizer controller main-sub Gaussian densification.
* `synthetic\`: Scripts related to the 10D+ Global Illumination application
* `splatting\`: Scripts related to the 6D Radiance Fields application
## 6D Radiance Fields `(./splatting)`
In this application we use 6D dimensional Gaussians parameterized by world position and view direction. For a given camera all Gaussians are projected to 3D and are then splatted using the rasterizer from https://github.com/graphdeco-inria/gaussian-splatting.
### Training
First install all the requirements in `requirements.txt` including the `ext\diff-gaussian-rasterization` and the `ext\simple-knn` modules using pip.
To train first you should have a compatible scene. We currently support COLMAP and Blender type scenes (for information to turn videos into these formats follow information at https://github.com/graphdeco-inria/gaussian-splatting). Once you have your scene you can train offline with:
```
./splatting/splatting_train.py --dataset_path --tensorboard
```
This will start training and it will save in the .\models\ path a .pth and .ply file every 1000 iterations. It will also track progress using tensorboard like test PSNR, number of Gaussians etc.
### Training with Preview
Alternative you can run a training with an online preview of the test images using ImGUI. This is slightly slower but allows you to inpsect training, observe how the refinement works etc.
```
./splatting/splatting_train.py --dataset_path
```
### Pretrained Models
The CD and Tools models shown in the paper are pending approval and should be up by late August - early September. Alternatively you can train on theses scenes by downloading them from here: https://nex-mpi.github.io/
### Note on 360 scenes
Currently our method and specifically the refinement does not support 360 scenes. In cases where the initialization of Gaussians is very approximate the refinement cannot overcome the local minima. We are aiming to adapt our refinement for that scenario in the future. For now you either need a good initialization or mostly front facing scenes.
## Global Illumination `(./synthetic)`
In this application we take a **custom** Mitsuba 3 scene with optional variability, render buffers and evaluate the Gaussian Mixture on the surfaces of the scene.
First you need to compile the custom Mistuba 3 version in `.ext/mistuba3` directory. For instructions on how to do this follow https://mitsuba.readthedocs.io/en/stable/src/developer_guide/compiling.html.
## Training
Convert a scene into one compatible with our method. This comes down to changing the `.xml` to include rendering the necessary buffers and to define the camera to be be variable for a given range:
```
```
Make sure to adjust min_bounds and range bounds for the range of possible camera placement.
Once you have this xml and the scene ready you can create a dataset to train on in the following way:
```
./synthetic/data_generation/create_dataset.py --scene_path
```
Once you have the generated dataset you can train on the synethetic scene. To do this run:
```
./synthetic/synthetic_train.py -- scene_path --dataset_path
```
## Preview Trained Model
To render using the trained model use the preview script:
```
./synthetic/synthetic_preview.py --scene_path --model
```
## Adapting the method your own application
We believe that our N-Dimensional Gaussian Generator is a good fit for various applications. To use it in your applications here are a few tips:
* Use the GaussianGenerator as a model instead of an MLP
* Start with the learning rates used in either application
* Initialize the generator with a few Gaussians (you can samle your input space) and give a starting covariance bias (i.e the Gaussains are starting isotropic, how much should they span in each dimension)
* Usually it's good to normalzie your inputs (as you do for MLPs)
* Follow our applications to frequently prune, seed and grow the Gaussians
* If you want to evaluate the whole N-Dimensional mixture use its forward function
* Alternatively if you want to handle the Gaussians in some other way (like we project to 3D) use the `get_gs_total` function
We aim to improve this apect of the method, so your feedback is valuable.
## License Note
The license of the `diff-gaussian-rasterization` module is defined in `https://github.com/graphdeco-inria/gaussian-splatting/blob/54c035f7834b564019656c3e3fcc3646292f727d/LICENSE.md#gaussian-splatting-license`