I have a Quest 3, and I suspect a lot of the people who are thinking AVP and not Quest 3 haven't liked Meta's pitch. A lot of the metaverse stuff is silly. It was a poor pitch, and it wasn't remotely ready.
Beyond that, the Meta Quest 3 doesn't have the best screens, so things like watching video aren't actually very good. The passthrough is comically bad, so any AR stuff is really a no go.
The only things the Meta Quest 3 does well is video games and video game fitness experiences. The reason to consider the Quest 3 is almost 100%, "do you want to play VR video games without breaking the bank?" That's it.
The Vision Pro is not making that pitch at all, and doesn't even support motion controllers.
I don't think there is a lot of cross shopping between the two, and I don't think people looking at the Vision Pro are just Apple enthusiasts. They simply aren't that interested in video games, but are interested in the other experiences Apple is touting.