If Facebook is already getting custom Intel and AMD motherboards, ARM motherboards should be no problem. (Note that a major undercurrent of Facebook's announcement is that "custom" is actually cheaper than "commodity" at "Web scale".) ARM server chips (Armada XP and Calxeda) don't have GPUs.
The next-generation high end ARM core, the Cortex A15, will be the first one that's intended to be suitable for server use: they've added virtualization support, PAE for up to 1TB of RAM, and cache coherency to their bus to support SMP. The server-oriented implementations will be quad-core 2.5Ghz chips, of which you would be able to put at least 4 on a motherboard like this. When they start hitting the market next year, they'll probably all but kill the market for Intel Atom-based servers.