Comment by Scene_Cast2
1 day ago
I wonder if there's research into fitting gaussian splats that are dependent on focus distance? Basically as a way of modeling bokeh - you'd feed the raw, unstacked shots and get a sharp-everywhere model back.
1 day ago
I wonder if there's research into fitting gaussian splats that are dependent on focus distance? Basically as a way of modeling bokeh - you'd feed the raw, unstacked shots and get a sharp-everywhere model back.
Multiple groups working on this:
https://dof-gs.github.io/
https://dof-gaussian.github.io/
Thanks for the links, that is great to know. I'm not quite sold if it's the better approach. You'd need to do SfM (tracking) on the out of focus images, which with macro subject can be really blurry, I don't know how well that works.. and a lot more of images too. You'd have group them somehow or preprocess.. then you're back to focus stacking first :-)
The linked paper describes a pipeline that starts with “point cloud from SfM” so they’re assuming away this problem at the moment.
Is it possible to handle SfM out of band? For example, by precisely measuring the location and orientation of the camera?
The paper’s pipeline includes a stage that identifies the in-focus area of an image. Perhaps you could use that to partition the input images. Exclusively use the in-focus areas for SfM, perhaps supplemented by out of band POV information, then leverage the whole image for training the splat.
Overall this seems like a slow journey to building end-to-end model pipelines. We’ve seen that in a few other domains, such as translation. It’s interesting to see when specialized algorithms are appropriate and when a unified neural pipeline works better. I think the main determinant is how much benefit there is to sharing information between stages.
1 reply →