Hacker News Reader

Comment by brundolf | Hacker News Reader

Maybe this is naive, but I don't understand what Google's impenetrable moat is around search.

It's a hard technical problem, but not one with a concrete barrier to entry. And there are lots of really smart people solving really hard technical problems out in the open. On top of that, much of the recent progress in AI is open-source. Surely that could help?

I guess indexing at that scale takes a lot of hardware. This is the most plausible barrier that I can see, but it still doesn't seem insurmountable.

What if you rethought what a search engine is? Does it really have to cover all of the text on every page on the internet? Could it be more focused?

This just seems like a solvable problem once there are enough smart, motivated people. Which there appear to be.

View on HN

Indexing isn't the hard part. Ranking is. For any given query, there are probably millions of web pages that match. How do you pick the best ones to show the user? How do you do that even when most people are really bad at expressing what they want?

Even if all websites were acting in good faith, this would be a really hard problem. Now add in the fact that there's an $80 billion industry devoted to gaming search results, and it suddenly becomes much harder.

Would it be impossible to build a search engine as good as Google? No. But you could very well spend billions of dollars to match or even slightly surpass Google's performance and still lose. Why? Because unless it's clear and obvious that you're superior to Google, people are still going to use Google because that's what they're used to.

That's why the smart players in the space like DDG aren't competing head-to-head in search, and are instead focusing on areas, like privacy, where Google can't compete. As for the others, I suggest you try Googling "cuil" some time.

Then Google’s ranking algo must be pretty bad because all I have is a first page full of bullshit Advertorial articles when I’m looking for serious stuff.

(Maybe that’s what the average users are looking for, though.)

The main barrier to enter the search market is not tech, it's money: https://0x65.dev/blog/2019-12-22/google-competition-is-just-...

And of course there are lots of other issues. For example the Web is very hostile when you are not Google Bot (i.e. there are a lot of big sites that will forbid you from crawling their content, unless you are Google, or Bing).

The money mentioned in that article is about being the default search engine, not being a functional one. Basically the question I'm asking is why DDG has to outsource to Bing.

> there are a lot of big sites that will forbid you from crawling their content, unless you are Google, or Bing

If you're talking about robots.txt, that's just a suggestion. It doesn't hold any actual preventative power.

IP blocking and rate limiting for clients outside the blessed range operated by G

IIRC that thought process was where DDG started some years back now. It was developed in the open wrt HN, by a then HN regular.

My recollection is they started by doing their own crawling.

(I want to say the username was Weinberg?? But honestly can't recall if that's right.)

TBH it's probably also the thought that Google started on, as principally we had manual indexes back then.

Again IIRC, DDG got along way with private indexing and then realised it didn't really gain them much and they instead wanted to focus on being a Google competitor. They did that by focusing on instant answers as a distributing feature and by buying results to get closer to the coverage they needed.

Lucky for them Google made their search a lot worse so DDG's quality caught up.

Gabriel Weinberg, aka ei0Bauqu https://news.ycombinator.com/user?id=epi0Bauqu

Their crawler is whitelisted by all the websites.

Google search engine uses 2 million computers. This is a pretty concrete barrier to entry.

Reply on news.ycombinator.com