Comment by dagmx
2 hours ago
I know this comes up a lot on HN because its not primarily a graphics community but:
1. Gaussian Splats are very expensive to render. They capture a lot of detail which makes them seem cheaper than an equivalent raster render of that quality, but they wouldn't meet real time AAA game performance requirements
2. Gaussian Splats don't have a concrete surface. Want to cast shadows or do physics? It's doable but very tricky. Want to relight them? Also tricky. What is the exact surface point that you want to affect or sample for any particular operation? Deformations also become very difficult to do well.
3. Gaussian Splats are not sharp. You can get sharper with different kernel types or higher density of points, but your costs go up as well.
4. Gaussian splats are awful for any kind of path tracing. You can do it but you go back to the issues above. So mixing and matching traditional content with splats becomes a performance bottleneck.
I don't think you'll see a AAA game use splats for more than something like cinematics in the near term.
I'm working on the vision component of a drone racing stack. Could I use GS to render my living room as a digital playground to train my vision models in?
I know nothing about the technology but the alternative is creating a 3d model of my living room which is also outside my skill-set.
Splats would work and it’s what a lot of automation folks use. But the risk is that your splats aren’t tight to the surface and that can cause false positives.
For training you can do a hybrid geometry plus splats workflow. Have geometry that you can constantly raycast against and have as an input to your vision training or to get accurate depth buffers.
The workflow for splats and photogrammetry are very similar.
> Could I use GS to render my living room as a digital playground to train my vision models in?
Yes, its what the autonomous car people are doing.
However you might want to do photogrammetry first (https://github.com/alicevision/meshroom opensource) as that produces a mesh which you can use to detect collisions easier. The downside is that transparent objects render really badly. but it is a lot faster to render
Ehhh if it's just for looking and you don't have anything lidar just go for splats they're way better behaved, mostly because they don't need to understand a concept of "surface" they just understand "splat with spherical harmonics of view-dependant color".