For me, the biggest issue with “virtual windows“ would be the lack of parallax. It would be very jarring to not see the view change as you move relative to the window.
They could implement head tracking to solve this, but that breaks down as soon as more than one person need to look through the same window.