Comment by sovok
14 hours ago
Also interesting: https://github.com/apple/ml-sharp
Apples model to generate Gaussian splats from a single image. Takes about 30 seconds on an M1 Pro.
It falls apart once you move too much, but for a little side-wiggling or a second-eye view for VR, it's great. And looks a lot better than the old approach of depth map + vertex shaders that I use in https://github.com/combatwombat/tiefling. But ml-sharp has 2.6 GB weights, a bit too big to run in the browser.
Is this thing the basis of what Apple is doing in the Photos app in the most recent OS revisions?
It seems to work a lot quicker than 30s now on iDevices and Macs.
Any idea what kind of vram is needed to run this on Nvidia?
It runs natively on the Photos app in Vision Pro, which shares 16GB of ram with GPU and CPU.
The Python process peaks at about 15 GB, so maybe a 24 GB GPU works. But others seem to have problems wih Nvidia: https://github.com/apple/ml-sharp/issues/84
The smallest Nvidia gpu I've run Sharp on is an Nvidia T4 which I think has 16 GB on the chip