Hacker News Reader

Comment by thematrixturtle | Hacker News Reader

I suspect this is largely due to DDG using Bing under the hood, which has led to similar weirdness in the past, eg all major porn sites disappearing from the results in Singapore (while Google still showed them, mind you!).

Still super disappointing though, and yet another reason why trying to build a better search engine on top of someone else's tech is a non-starter.

View on HN

Having worked on Google's indexing system, I can't imagine how much it would cost to write all of the crawling and indexing code from scratch, and then run it on over the visible web. You need to bootstrap somehow form an existing index if you want to get anywhere in any reasonable time.

If there's a market for alternative search engines out there, AWS is already serving so much of the web that Amazon should really provide crawling and basic indexing (for a fee, of course) so it's done once for everyone, with Amazon Lambda processing to allow search engines to customize their indexing. I'm not sure if it would make them much money, but it makes more sense to start up a search engine on AWS using Amazon's crawl and basic indexing vs. using a search engine competitor's crawl and indexing.

That's an insanely good idea. Since behind the scenes, Amazon would know who owns each website that it's crawling, an AWS search database would inherently be better-curated and more trustworthy than anything an external crawler could put together. Having your website on AWS would be crucial for SEO.

Awful for competition and whatnot.

Google and Bing can already crawl AWS sites just fine. As long as Amazon doesn't provide any first-party search engine, just providing search engine infrastructure, I think it would be a net win for competition.

As it stands, if you want to bootstrap your own search engine, you need to base your search engine on either Google or Bing's index, or perform the herculean task of making your own crawler+indexer+search engine. If Amazon can commoditize the back end, and let AWS-hosted search engines provide differentiators late in the indexing pipeline and on the search/serving side, I think we'd see more niche search engines spring up.

You know what? Not that crazy of an idea.

Amazon is a weirdly plausible competitor to Google. They have a surprisingly robust ad business, and obviously AWS.

https://www.cnbc.com/2022/02/03/amazon-has-a-31-billion-a-ye...

Amazon ran A9.com as a search engine from 2004 to 2008. https://en.wikipedia.org/wiki/A9.com#A9.com_search_portal

Yes, A9 was one of the competitors for some of my time at Google, but I'm talking about Amazon basically creating some search engine infrastructure and a framework for building search engines on top of. Selling shovels to the gold miners, in a sense.

The crawling and initial indexing stages need a lot of scaling and performance tuning work, which is a core competency for the AWS engineers. Furthermore, essentially zero people say "I want to build a startup around my new crawl scheduling idea, but it's too time consuming and expensive to write the rest of the search engine." Many more people say "I have a new idea for ranking" or "I have a new idea for content extraction, but it's too expensive and time-consuming to write the rest of a search engine".

Google and Bing already expose "custom search engines" which are basically just filters on top of their search engines. I'm talking about something similar, but much deeper customization.

Also, I was at Goldman when Google Search Appliance was EoL'd. GS had plenty of internal content that needed to be indexed, absolutely needed to be kept private, and wasn't well indexed by any of the replacements GS evaluated. For companies that keep lots of documents in AWS storage, it would be good to open up the space for Google Search Appliance replacements. Amazon could write their own, but given that Google abandoned the space, the profit margins aren't great for a huge company. However, creating a search ecosystem within the AWS ecosystem makes the AWS ecosystem more attractive. Maybe forgoing the search profits and inviting in search providers would be a net win in selling more AWS compute and storage.

Unfortunately I suspect this would be selling gold shovels to dirt miners and the economics of it are not compelling.

I don't see how it's a non-starter. DDG is still a better search engine than bing, if for no other reason than bangs. If a censored search engine is the non-starter then every well-known search engine is a non-starter.

Maybe that's the real problem. There have been some recent articles on HN about how "search is broken" and maybe this article falls under that. Because what really gets me is that there is plenty of legit content on "pirate sites" and blocking them completely cuts off that content.

Reply on news.ycombinator.com