Yeah, this. We helped my boss' brother out with his wordpress ecommerce site.
His machine could easily handle the traffic thrown at it, but the page load time was slow. 2 seconds at best, with all machines involved being almost 100% idle. With various common tweaks such as caching, we got it down to about 800ms. We eventually replaced it with our own solution and got it to 50ms.
Scale never entered the picture because from our testing, we had the machinery in place to handle enough traffic to be wildly profitable. The user experience at 2 second page load times was vastly different from the user experience at 50ms though.