This is speculation, but I believe this is a vestige of when voice chat applications widely used by PC gamers like TeamSpeak and Ventrilo did actually rely on independent servers, usually paid for and administered by individuals (or guilds/clans).
Back in the bad old days if you wanted to voice chat you had to know someone who could invite you to join a "server" (or set one up yourself), and I think the verbiage just kind of stuck.