Everyone relies on those servers. That's how WordPress ships. There is no way to make it use other servers and WordPress is an otherwise very flexible platform; it benefits from centralizing all of that as illustrated by the way it's designed. The whole ecosystem works around it.

So should everyone have to pay then? Everyone who uses the software uses the servers. If not what's the threshold? And remember that Matt has insisted WordPress.org ("those servers") belong to him personally, not to WordPress or to Automattic.

If you're going to monetize access to plugins and themes produced from volunteer work on your open source code... can they monetize too? Does everyone get a cut?