I think that would be true for testing, but not for actual development. When I'm working I want to marginalize anything that might slow me down. You did mention the build machine should be fast - but don't most people build locally first before pushing out?
I guess it also depends on what you're building. I mostly work on stuff that isn't used by a consumer. It's going to run as a service on a machine or network of machines far more powerful than my PC. I imagine most people developing for consumers though are building websites or mobile apps. Mobile apps should be tested on a variety of devices, and it would take a pretty intense website to start bogging down most people's machines I imagine.