The problem is it's often the devs themselves who request the siloing. I used to assume otherwise, as I think the way you do. And then I had a lot of my devs rebelling because I was keeping them from their "real work" by having them deal with users.
I used to be this developer.
The more the development team complains, the more valuable the system is. If there is never any friction between the developers and customers, it's likely that everyone is on the same page and the meeting is largely redundant.
I don't have a lot of patience or sympathy for this attitude even though it's practically universal. There might be a few legitimate cases where someone basically has a handicap where they are clinically incapable, but then that's a handicap like a broken leg. A defect. Maybe it's ok to make allowances for them in the name of equality but it's absolutely making allowances, and should not be a norm and should not be something just everyone gets to claim for their mere comfort and convenience. IMO anyway if I were king and all that.
If I'm graded on getting tickets done or getting some feature out, then meeting with users may be beneficial and the right thing to do but incentives turn it into swimming upstream...
In an idealized world, maybe, but enterprise software isn't paid for by its actual users, but by their company's executives/management.
If I put some devs in front of a customer I'd be very much afraid of them doing what they ask rather than what they want, for instance, or overindexing on one particular customer's problem.
If the devs understand and can empathise with the users and there aren't a lot of them, sure, maybe but on most projects it's more effective to have the distilled and filtered customer feedback from a PM who is good at being a PM.