Hacker News Reader

> If you are going to rely on domain-level reputation then your search engine is going to be way behind the search engines that can judge content more specifically, which is all of the other search engines.

Then why do Google and DuckDuckGo return 90% garbage for most queries?

"All of the other search engines" have completely failed to keep pages from the results that are not only low-quality, but outright spam.

They definitely do not return "90% garbage for most queries". This 8s an unsubstantiated claim I see often i HN and honestly not backed by any real data. e.g. You can check your search history and see it yourself.

I just tried searching for "python str" on Google. I expected the top result to be a link to the official Python docs for the `str` type, then ideally some relevant StackOverflow questions highlighting common Python issues with strings, bytes, Unicode etc.

Instead, the top result was W3Schools. Then came the Python docs, then 5 pages somewhere between blogspam and poor-quality tutorials. Then a ReadTheDocs page dating to 2015. And that was it. No more official Python resources, no StackOverflow. In the middle of the results some worthless "Google Q&A" dropdowns that lead to more garbage quality content.

So for this query, using my definition of "garbage", the "garbage percentage" is somewhere between 80% and 90+%, depending on how many Q&A dropdowns you waste your time opening.

For me, https://docs.python.org/3/library/stdtypes.html is the top result.

The fact that the ranking of results for queries that have nothing to do with location-based services depends on where you are located (and, possibly, on whether or not you are logged in) is one of the worst things about Google. And the fact that you can't seem to disable that behavior is even worse.

I just tried searching for “python str” on searchhut and the top result is Postgres docs, then Wikipedia article for empty strings and then Drew’s blog. Official Python docs isn’t in the index at all.

For me the second hit is: https://docs.python.org/3/howto/clinic.html

So at least some official Python docs are indexed.

Afaik Python does not have a str type (I think you meant string?).

You could instead search for "python string" to find more information about python strings.

Even then the very first result for Python str is actually relevant for me (Python documentation about built in types.

> Afaik Python does not have a str type

It does have it:

  $ python3 -c 'print(type(""))'
  <class 'str'>

> Then why do Google and DuckDuckGo return 90% garbage for most queries?

If you can give me a list of 10 normal-ish queries where 9 out of the first 10 results on Google or DDG are "garbage", then I'll concede your point.

I think you are creating an impossible standard for search engines, then using it to deem the current ones as failures. While at the same time ignoring that this new search engine is, as present, unusable with no realistic argument for why it might eventually be better.

See my reply on the sibling comment for an illustrative example.

> why one would believe a search engine with a hand picked set of domains would be expected to outcompete a search engine that can crawl the entire web and determines reputation by itself.

Because SEO manipulation is a well developed field, ensuring that the search engines trying to determine reputation automatically will (and does) end up with bad results.

Indeed. Whatever "smart" algorithm you use to rank results, you can be certain that half the web will turn into adversarial examples once your engine becomes popular enough.

> Notice! This product is experimental and incomplete. User beware!

Seems like your expectations are misplaced. Being at top of HN is not an indicator of quality, just interest.

If you were to tell me curated domains is just a bootstrapping method and as the search engine evolves it will change, fine

This makes me think of a possible approach. Curate a giant set of domains that almost exclusively host high quality content. Crawl said domains. Use all of the crawled data as a training set to create a model with which to ascertain the quality of random Web pages from other domains. Then spider everything and run it against the model.