I'm not talking about real time though. Pre-recorded light field videos would already be high enough resolution. It might not be perfect but like the CGI nowadays close enough where there's moments you can't tell
Perhaps, but realistic scenes are bounded by the number of lights and triangles more than resolution. There are heuristics and approximations to get around this, but I think we will only be able to fake a subset of the human visual experience. Lots of smooth surfaces and simple lighting setups: think car shows and apartment renders, not nature.