What if the page you're looking for just happens to contain a lot of content / images / etc?
Adding speed to Google's ranking algorithms is only useful for searches where there are several equally good search results (in which case the fastest would be the one you would want). But in the event that you're actually searching for a lot of information, having fast (but less informative) sites propagate to the top would be detrimental.