Comment by rjf72 | Hacker News Reader

Comment by rjf72 | Hacker News Reader

Parent Full thread

You know there's something weird. You could prove your little hypothesis here wrong in less time than it took you to write it. So why state something that's wrong?

I tested this by choosing to search for something completely random that would return unusual results and give a decent indicator of whether or not DDG was just copy-pasting results, so to speak. I searched for "extent space taco secure", no quotations. Bing seems to have exploded when I searched for this. The results are completely nonsensical and seem to have nothing to do with what I searched for. The first two hits are to Google, the first about webhp (apparently some virus?) and the other to their chrome page. Other links include things like links to fidelity.com, americanexpress.com, and the huffington post front page.

DDG's results are [mostly] reasonable and populated mostly by things like some recent event where the head of NASA mentioned space security. Though it does have a link to the SF bay area craigslist alongside a couple of other really weird ones. In any case, you can see for yourself. Suffice to say they're definitely not just replicating results, at least not from Bing.

You could also read their documentation: https://duckduckgo.com/api

> Because of the way we generate our search results, we unfortunately do not have the rights to fully syndicate our results, free or paid.

That page links to: https://help.duckduckgo.com/results/sources/

Where, after you get past a bunch of stuff about their "instant answers" gets to the root of it:

> We also of course have more traditional links in the search results, which we also source from a variety of partners, including Verizon Media (formerly Yahoo) and Bing.

So yes, maybe I was wrong saying results come only from Bing. But they definitely source their search results.

I'm not knocking DDG here, I use it as my daily search driver. If you were to try and build a search engine today with limited resources would you really try to start from scratch? The way DDG has approached the problem (by sourcing results from other search engines) seems like the only reasonable way to be even remotely competitive.

Yes, they source their results. This is not the extent of your claims. You were stating that they directly rip from other engines meaning that, in your words, "In order for search results to get better on DDG, they first need to get better on their partner search engines".

That is simply completely wrong, as my example showed extremely clearly. The one and only weakness of sourcing from third parties is that if they are not indexing some site, you also will not be indexing it. You were implying they are directly dependent on the ordering and quality of the other engines, which is obviously and provably false.

First of all, I never claimed that DDG results were exactly the same as Bing's. I said the top results were the same. Admittedly, I used less random terms than you did.

However, I did try your experiment, same terms. The top results from DDG were indeed found in the top results of both Yahoo and Bing. You are right though, not in the same exact order. Then I tried the same terms on Google. There is only one shared result on the front page (the Nasa chief one). I went through 4 more pages of Google, none of the results that both DDG, Yahoo, and Bing all seem to share were on any of them. This aligns more with the point I was trying to make:

DDG is going to have a hard time improving their results if the information they're getting is from inferior sources to begin with. It's like working on incomplete information. No matter how good their algorithm is that ranks and filters results from other search engines, if those other engines suck, there is no way their results can be that much better. If, for a given term, Yahoo returns site A, B and C, and Bing returns Site D, E and F, is there a way for DDG to determine that actually site G is the better result? The results can't appear out of thin air.

Also, you claim with emphasis, that the only weakness of sourcing from third parties is the indexing problem. That's absurd.

Obviously, a huge weakness on sourcing from other search engines is that DDG are bound to the terms of those partnerships. Or a complete severance of them. No partners, no search results.

One result of the terms in these partnerships is that DDG can't provide a search API. There was a time I thought perhaps I could write a developers search engine with DDG as the backend. Turns out you can't, and as a small search engine popular with HN types, I feel like that is a huge weakness.

Obviously yes it is possible for DDG to determine that result G is "better", and prioritize accordingly. "Better" of course is subjective, but this search is a pretty clear example that they're doing something right. They do source results from Bing yet their engine determined, quite accurately, that Bing's results are really quite bad.

But beyond that, consider the bias in your statement. You're claiming, by definition, that anything that doesn't match Google results is "inferior". I found Google's results here to be much better than Bing (not a high bar to pass) but much worse than DDG. Here are the results I get for Google, though I think it's also worth mentioning the hassle. For the privilege of being able to search I was required to go through a captcha. This is presumably because I prefer to use TOR when directly using Google. Then there's some giant "privacy reminder" at the top that requires me to agree to have all my data hoovered up and combined across services to generate a profile on me. Presumably a GDPR related thing since each time this happened it was on an EU exit node, though it used dark patterns to coerce consent which was supposed to be unlawful to my knowledge. Anyhow, after that nonsense I get:

- 4 monetizable links (Amazon 'space taco' lighter, 2x space taco restaurant facebook pages, some soundcloud 'space taco' band)

- 4 nonsensical links of google linking to its own books.google service

- 1 link to FT.com where Neumann raised $700m through share sales.

I changed my IP a couple of times and got similar results - the monetizable ones were always the same (order varied - though always at the top), so I expect this is probably at least similar to what you got. Of course it's silly to debate what results should be returned for an intentionally nonsensical query, but nonetheless I think this inadvertently ends up emphasizing what each engine's priorities are. DuckDuckGo tries to return whatever results are likely most appropriate for what you searched for. Google tries to return whatever it can make the most money from. And Bing... well Bing is "special."

--

I do fully agree with you that being tied to the partnerships is potentially exploitable by the partner. And as DDG continues to grow this could become an issue if, for instance, Microsoft decides they have more to gain by undermining DDG than they do by continuing to partner and profit alongside them. My "only" was in reference to the quality of the search, which is what we were discussing.

Reply on news.ycombinator.com