Comment by downandout | Hacker News Reader

Comment by downandout | Hacker News Reader

Parent Full thread

The key statement in this article, that renders this report largely useless, is of course contained at the very bottom of it:

"The authors conceded that there were differences between their click surveys and actual search results, and that they didn’t have access to internal Google data to verify their results."

In other words, their conclusions are meaningless because they simply don't have access to to the data to determine how Google would rank search results in the fictional scenario that they were attempting to study. They essentially did A/b testing, showing some users fictional, Yelp-heavy search results injected into a fake Google results page, and compared those CTR's to those of an actual Google results page. While the fake pages achieved a higher CTR, that would probably be the case for many custom-crafted results pages curated by humans.

This was a "study" that was paid for by Yelp, in their quest to force Google to feature their results more prominently. I guess when you build your entire business on quasi-extrotion, heavy handed tactics come naturally. But this one falls completely flat for me.

Maybe I'm reading it differently than you, but I don't see that statement as being particularly meaningful. All that says is that they didn't have access to Google's actual internal data on clickthrough rates. Which is not surprising.

It turns out what they did was use a site called UsabilityHub (which is intended for testing website designs) to show the study participants a screenshot of search results. One screenshot showed Google's results, while another shows search results that were purportedly sorted purely by relevance. I'm assuming users simply reported whether they would have clicked on any results on the page, but the article doesn't go into that level of detail. I also assume the users weren't told whether they were looking at the genuine results or not.

Given that, there's nothing at all surprising about that statement you highlighted. In fact, unless Google were willing to actually run the study themselves, randomly showing either their current results or results sorted purely by "relevance", and gathering the real clickthrough rates, then what the article describes here seems like the only real reasonable way to do this.

One screenshot showed Google's results, while another shows search results that were purportedly sorted purely by relevance.

Therein lies the problem. They don't know how Google would order the results in the hypothetical scenario where results didn't include those that make Google money. This goes beyond not having clickthrough data; what they were showing to their test users was simply a not-very-good, human interpretation of what they presume the Google algorithm might return in a nonexistent situation. Unsurprisingly, these fictional search results had worse CTR's than were achieved by displaying presumably custom-crafted Yelp results.

The term relevance is unique to each search algorithm, and these researchers did not have the data to determine what the relevance would otherwise be for Google. This report essentially says "if we ran Google, we could get better CTR's by showing Yelp results". That does not prove consumer harm.

> They don't know how Google would order the results in the hypothetical scenario where results didn't include those that make Google money.

They kind of do. Google's very public about most of how they sort results, and the article said they used Google's "algorithm" to rank them. I know Google used to even expose the underlying PageRank number for various results, but I don't know if they still do so today (if so, then it becomes even easier to sort them the way you think Google would).

> what they were showing to their test users was simply a not-very-good, human interpretation of what they presume the Google algorithm might return in a nonexistent situation

You're making a very bad assumption here. From the article:

> One set of users saw a page reflecting results currently displayed by Google

Based on that it sure sounds like they actually performed various searches on Google, screenshotted the results, then fabricated a second set of results that reflects what they believed Google would have shown if it sorted purely by "relevance" instead of prioritizing its own services. And these fabricated results had significantly higher CTRs than the genuine results from Google.

> The term relevance is unique to each search algorithm, and these researchers did not have the data to determine what the relevance would otherwise be for Google.

Sure they do. The way Google ranks search results is quite public and has been so from the beginning. What isn't public is all the various tweaks they perform to the results, such as removing what they believe to be spam or content farms, as well as the biasing they do in favor of their own services. But the core algorithm is public.

> This report essentially says "if we ran Google, we could get better CTR's by showing Yelp results". That does not prove consumer harm.

The report says that they believed that Google was suppressing results from "third-party review sites such as Yelp and TripAdvisor" (not just Yelp) that it would have otherwise deemed to be relevant, in order to prioritize Google's own services, and that this artificial adjustment of the search results is harming consumers because it's yielding results that are less useful to consumers (as measured by CTR). Which is a bit different than what you're saying.

And this actually makes sense to me. When I search on DuckDuckGo for places in my city, I often get yelp results near or at the top. So they're obviously deemed relevant by the unbiased ranking that DDG uses. And I often get Yelp results if I search on Google as well. So obviously Google thinks Yelp is relevant for many searches. So if I perform a search that Yelp has relevant results for, and Google isn't showing me those results but instead shows me e.g. a Google+ business listing, then it seems quite reasonable to claim that Google is artificially suppressing the Yelp results in that case, and to further claim that this is causing consumer harm because the Yelp results were more relevant than what Google chose to show instead.

I think it's more of a case that Yelp paid them to find something and they don't want to turn around and say 'Hey thanks for the money, we didn't find anything' so they stitched this accusation up.

Shouldn't it be a highly poignant here, that only Google has access to the data necessarily to police Google's conduct?

Reply on news.ycombinator.com