Check out 360 degree Panorama formats, especially "Cubic" [1]. A similar approach was used back in 1992, in CAVE [2], using 6 virtual cameras.
[1] https://wiki.panotools.org/Panorama_formats#Cubic [2] https://en.wikipedia.org/wiki/Cave_automatic_virtual_environ...