Hacker News Reader

Comment by mojuba | Hacker News Reader

Yandex doesn't index the entire Internet though, but only some selected part of it, mostly pages under .ru and all ex-USSR TLDs, plus domains submitted to them manually. Not very impressive I must say. news.ycombinator.com, for one, isn't in their index.

Edit: in addition to ex-USSR TLDs they also index web sites that map to IPs physically located in ex-USSR, regardless of the domain.

View on HN

AFAIK it's their policy to index only Russian language content and they do it pretty good. They can index domains in any zone as long as their content is in Russian.

Anyway IMHO Google gives better search results in Russian. I guess they have more users here for the same reasons Yahoo does better in Japan. It's all about culture.

The problem with Yandex is that it does index English language content, however not all users (by far not all!) are aware that it's limited geographically and by TLDs. If I look up, say, "Succinctness is Power", which should bring up PG's essay on his web site, Yandex instead retrieves a bunch of pages that mention the article and also Russian translations rather than the original. So the user is left thinking that the original is not even on the web (unless she carefully examines the links on web pages that pop up in the result set).

They were clever enough to index en.wikipedia.org, although it is usually not at the top for some very obvious requests (e.g. try "Turing Machine"), which is an indication of a rather poor ranking system.

My impression is, Yandex does not compete with Google in any way. Despite its slogan ("Everything can be found" or something like that) it's just a limited web index that doesn't even state explicitly about its limitations. And that's strange at least.

Edit: NB: they have a minimalist front-end at ya.ru

Yandex is local - limited to CIS (ex-USSR zone) in terms of people the search results (and the search index) are optimized for.

Yandex is mainstream, it's not a geeky site, and most Russians don't speak English well enough to prefer originals to translations. So most of its users just don't arrive at the limits of Yandex's index ever.

Of course it's nice for me to be able to go to google.com and search in Russian or in English or in any other language (and get results in the language I search), but it's just not what the majority of Russians needs or wants.

Their policy on the search index was 'only russian-language content' (their bot analyzes page text rather than looks at the TLDs or IPs), then a couple of years ago they started including more and more sites in english, starting from encyclopedias etc. This is possibly also because their indexing technology was deep into morphology of Russian language from the very start in late 1990-s.

PS their slogan is a bit more dramatic - 'Everything WILL be found' :)

Thanks for clarifications, but to my understanding they won't index Russian content if it's not under exUSSR TLD AND it's not physically hosted in the exUSSR. It took me 3 years (!) to convince Yandex (through submissions, mails and even phone calls) to include my Russian-language blog, only because it was under .com and hosted in Texas, despite the fact that my blog had a lot of references from the .RU space.

And no, they confirmed, it wasn't because my web site was marked as spam. They explained to me that "some" servers have to be included manually because they are beyond their "default" space.

What puzzles me is that Yandex is definitely not the entire Internet, not even the entire Russian-speaking internet, and yet it claims to be finding "Everything" without mentioning of the limitations in place. My point is, this is misleading for the users, if not just plain cheating.

Soemthing is definitely wrong with their policies with regard to indexing the Web.

It's definitely everything for most "normal" users (like my sister or Russian-only speaking friends). Mentioning of Russian-only limitations or providing English-language results would be confusing for such users (most of them don't even know what Google is).

Not long ago I submitted .com English/Russian/German site to them and they refused to index anything but Russian part, but Russian part was index successfully (although the sever is in Russia).

I used Yandex for many years for Russian-language searches and also I read all PG essays and I've never thought about using Yandex for English-language searches. If I search Google for "Succinctness is Power" with "Russian language only" option I will get similar results. 99% Yandex users don't expect it to search anything but Russian. And each search results page have links to Google, Yahoo and MSN at the bottom where you can search for the same keywords globally.

Yandex explicitly state its limitations at http://company.yandex.com/

Being No.1 Russian-language search engine they do compete with Google in local market (few years ago Google wasn't even an option for Russian-language search).

And its slogan is Russian-only ("найдётся всё") - only for Russian speakers.

Reply on news.ycombinator.com