I think you're largely referring to sites that work based on their content, and not their workflow. i.e. site you interact with to consume content.
Most of those sites are "free". People are willing to wait for things that they want to consume and are available for free, up to some limit.
If we're talking about websites that offer some sort of utility -- i.e. it does something for you and you need to interact with it often. Then its responsiveness is likely going to be a much bigger factor.
If gmail loads your inboxes and emails in hundreds of milliseconds but yahoo took several seconds, and a person needs to respond to lots of emails during the day, I'm sure the person would develop a preference for the former if all other things are equal.