Hacker News Reader

Comment by Retro_Dev | Hacker News Reader

Every time I hear about Gaussian splatting, I think about triangle splatting[0] - GPUs were built to render triangles, so I wonder what games would be like if we supported generating and rendering them as much as we do Gaussian splats.

[0] https://trianglesplatting.github.io/

View on HN

The Dreams game engine (only for PS4) does this and is able to achieve scenes that no other game engine can. Here's a technical talk about how they did it: https://www.youtube.com/watch?v=u9KNtnCZDMI

Actually sorry not sure if they use triangle splatting, but gaussian splats at least.

The Sony Dreams beta in 2018, was truly revolutionary and even today the world hasn't caught up with a lot of its breakthroughs. I'd been deeply involved in the design of high-end, pro digital media authoring tools for two decades and when I saw that beta it blew my fucking mind.

Media Molecule (LittleBigPlanet) spent over 8 years creating Dreams, which would never have happened without Sony's funding, but the tragedy was forcing what could have been a brilliant high-end authoring tool into a gamepad / lean-back interface. Even now, there are still innovative UX approaches in Dreams that could be adapted into high-end 2D, 3D, compositing and animation tools like Blender.

I wish they would have broken down the walls of the garden and made it a first-class engine for shipping “real” games on the platform. Even from a business perspective, it seems like a huge competitive advantage—no way would people be able to port any of it to other platforms.

Pretty sure they use SDFs, not triangle or gaussian splatting

If I recall their talk correctly, models are unions of SDFs, which get sampled into a point cloud, and each point gets splatted as another SDF primitive called a "fleck".

Ah thanks for the correction. The splat terminology they also use confused me.

I think SDFs is the data underneath, but baked into SDF voxel tiles at runtime and not always evaluating SDF shapes. The SDF tiles still get splatted.

At one point it used triangles for the large splats, and still compute shaders for the small ones due to quad overdraw. I'm not sure if the final version stuck with that or went all compute, but quad complexity is also why nanite does compute shaders for small triangles and traditional rasterization for the larger ones.

In my experience it isn't really about quad overdraw/helper lanes, but rather about primitive assembly. On my RTX 4090, vertex shaders max out at about 20M tris/ms, while mesh shaders, which bypass primitive assembly, can reach 40M tris/ms. Maybe quad draw might be the next bottleneck after primitive assembly.

But it's a bit difficult to find the exact reason since profilers won't really tell you a concrete bottleneck when drawing vast amounts of tiny tris.

Nanite does support mesh shaders and does use that for the larger triangle rasterization when available. But quad overdraw is brutal with triangles near pixel size, making 4K as expensive to render as 8K, and they've said it's the reason they fall back to compute shaders even when mesh shaders are available for small patches. That might have been part of the reason on Dreams though since there weren't any mesh shaders on PS4.

4K has 8 million pixels so should be able to do pixel per triangle with a separate depth pass and still fall within that 20M even with some overdraw from not all patches able to be culled.

edit: opps thought you said per frame, per ms would give 330M/660M triangles per frame, doesn't seem like a big bottleneck; PS4 was more like 26M per frame though but rendered at 900p-1080p. I think it's translucency was stochastic so lots of stuff could still get culled, but wouldn't be able to cull as aggressively (like nanite with masked materials).

They've made a follow up to that called MeshSplatting. https://meshsplatting.github.io/

I think that Gaussian splatting is generally better, as the Gaussians are smoother and easier to optimize, if you can use them.

Things like triangle splatting are useful if you have to integrate into an engine that only support triangles. But with modern graphics pipelines, you can implement Gaussians quite efficiently, including newer techniques like Gaussian point splatting that can scale up seamlessly to enormous scenes:

* Video: https://www.youtube.com/watch?v=2AWv_nIfp-U

* Paper: https://jorisar.nl/gaussian_point_splatting/gaussian_point_s...

* Repo: https://github.com/JorisAR/gaussian-point-splatting

Already done in 2023: https://mobile-nerf.github.io/

runs on a phone: https://storage.googleapis.com/jax3d-public/projects/mobilen...

Every time I see a Gaussian splatting post, someone brings up triangle splatting. So I think it's an inevitable pivot.

Yeah, there were other splats before Gaussian splats, so lots of cool possibilities depending on the style you're after. However, hard triangles have hard edges, and then you might be anti-aliasing those edges in some way. Gaussian splats can just be a triangle or two where the pixel shader feathers towards the edges.

Has anyone doing this but with shared vertices

Isn't nanite in unreal engine exactly that?

To be clear, I'm not proficient in it's use, just watched a few videos about it's theory when it was initially announced

No, nanite is just having different levels of detail for objects, and which LOD gets selected is done automatically by the engine. So users can use ultra-high-def models (including 3d scans of real-world objects), and the engine automatically generates and uses lower polygon versions for when the object is far away or whatever.

Strange, when I looked into it that seemed to be just a desirable side effect with the rendering being done via the triangles - which was also supposedly the reason why you could directly use scanned models, despite them having millions of triangles in their model

But again, I've really only super superficial understanding of this, I just understood it like that back then

I learned about gaussian splattering mere months before nanite was introduced and their technical talks and articles always sounded exactly like what I just learned about the splattering, that's why I got that impression

Honestly there should be many more types of splatting imo...

Beta splatting seems pretty promising. It addresses the fuzziness aspect of gaussians among other things.

https://rongliu-leo.github.io/universal-beta-splatting/

https://linus-franke.com/hibs/

There are various papers on variants of Gaussian splatting (keyword "differential rendering") which replace the Gaussians with some other explicit geometric representation that can also be rendered in real time. A triangle soup is just one example. I recall one which uses some sort of tetrahedral foam instead.

Reply on news.ycombinator.com