Check out the Tom Scott videos - the presenters get a screwed up perspective, but sight lines work correctly.

Also, anything aligned with the video panels (most notably the floor) is correct from all angles.

Video walls can change perspective. They're heavily in use for shows like the Mandalorian among others.

They'd essentially just show what's being composited in behind them, and react to the camera tracking.

This is how the Mandalorian works

Take a look at how they filled the Mandalorian. Quite a lot of it was filmed inside a large video wall / cube. One of the big advantages for them was that it resulted in far more realistic lighting.

https://www.youtube.com/watch?v=gUnxzVOs3rk