Yahoo became the goto portal because the quality of their links was high. Google overtook them because metadata was the only practical way to keep up with the pace at which the web was expanding.
However, after more than a decade of SEO, it's limitations are evident as the long tail gets increasingly harder to reach with search engines based on the Google model, e.g. "Purchase empiricist philosophers on eBay."
My suspicion is that Google is abandoning neutral search for personalized search in part because of the problems SEO represents when it comes to neutral searching and personal tracking provides an algorithmic way of establishing the quality of results for ranking purposes. One which is easier than attempting to develop a neutral curation algorithm.
That's pretty much precisely how PageRank (in its original form) changed web search. At the time, at least, it worked because we were really bad at automatically analyzing content, and the connectivity signal provided better results.