I'm not so sure. Human vision is only sharp in a coin-sized area at any time. If you fix your eyes on a single word in your comment you can't actually read the entire comment, for example.
In other words, you don't need 4k in your entire FOV you just need to ensure that most of the pixels are spent in the middle of the viewing area.
I believe that it would be possible to have the screens in the headset generate a very distorted image, where the edges are compressed to a small area of the actual screen and therefore low-res, while the lenses stretch this image to fill the viewing FOV. Kind of like anamorphic movie lenses, or even how wideangle lenses distort the edges more than the center.
I have no idea if Vision Pro does this but it seems theoretically possible at least.