It can bog down depending on your communication patterns. Context switching isn't too bad, it's 'green threads' so there's not the OS overhead of switch threads when you switch between Erlang processes; it's not zero overhead of course. Messaging passing latency is mostly driven by memory latency and lock acquisition; if you're in a big system, message queue locks tend to be uncontested as there's enough things going on that it's unusual for two processes to message the same third process at around the same time. But you can't really get away from that --- regardless of your language, if you've got multiple threads of execution working on the same data, you've got contention and that'll increase latency; even lockless algorithms have contention on the atomic values, and you have memory communication between processors.
All that said, BEAM isn't chosen because of its high performance, it's chosen because it enables a simple programming model for distributed systems while being fast enough. It's gotten faster over time too; of course, so has everything else.