The problem with such an approach would be that NERFs require a set of input images with their exact poses and exact poses are only available if the underlying geometry is static. However if you use SD to generate new views it's only an approximation and you wouldn't be able to get the exact poses.

Not all hope is lost though. I'm pretty sure in a few years (perhaps sooner) we'll be able to generate entire 3D scenes directly without going through 2D images as an intermediate step.