I'm the exact opposite. Short meetings with a small group (preferably just one other person, as discussed in the OP) at a scheduled time with an agenda are a much more efficient use of my time than ambient conversation. I've worked on teams and with people who feel like you do and it drives me pretty nuts. I'm constantly getting pulled into low value conversations that are a distraction from what I'm doing. I can't determine ahead of time whether the conversation is more or less valuable than whatever else I'm doing because there is no agenda for "hey you got a sec?".

What I don't know is: am I just wrong about this, or are you, or is it just pure subjective preference? If it's just subjective, what's the best way to structure teams with respect to this? Is it all or nothing, or can you build a team with both kinds of people in such a way that they respect each other's preference?

Edit to add: To make a different point, I'm a big fan of standing closed-door 1:1s with your manager, because the conversations you can only have with your manager are the most awkward ones and asking for a one-off closed door meeting when it's uncommon makes it way worse. For instance, if you're having strife with a co-worker that you need to bring up with your manager, the last thing you want to do is walk over to their desk and say, "can we talk privately?". Waiting for your next standing meeting is a much better option.