Nvidia hints at neural rendering replacing rasterization and ray tracing

Create two renderers -- the real-time one and a super-high quality offline one. The offline render always renders at 8K resolution with full ray tracing, all possible effects, etc... basically like how movie special effects are made: minutes of computation time per frame.

The real-time render doesn't try too hard to output the full-quality result, and it might not even output anything at all that visually matches the desired output.

Instead, the real-time render outputs a lot of metadata for the AI to consume. So it might output a depth map, a motion vector map, a surface normal map, an un-shaded but textured rendered scene, and a material ID map.

The AI then takes those various inputs and produces the final shaded scene with all sorts of complex "special effects" applied such as realistic skin, hair, or whatever. To train it, the inputs are the offline-rendered 8K scene and the real-time rendered metadata. This is an ideal scenario for AI training, because the "labeling" is always pixel-perfect ground-truth data. (Compare this with, say, LIDAR depth maps or motion vectors from video, which are always full of large errors.)

The material ID is typically an integer, sort of like a "token id" in an LLM. In a similar way, it can be expanded into a learned "embedding" by the AI, which can then combine the embedding with the metadata to produce a very high quality output. So rare materials like skin become learned as various "skintone embeddings", modulated by the texture mapping input metadata. Because arbitrary training data can be synthesised at will, it's also easy to train the AI by generating many scenes that occur rarely in the actual game. E.g.: each character can have many points of view generated in various poses, in various lighting conditions, etc...

Simplified variants of this is already used in popular games via NVIDIA's DLSS upscaling and motion-interpolation technology.