Ask HN: Why isn't there a Google competitor emerging? | Hacker News Reader

Ask HN: Why isn't there a Google competitor emerging?

204 pointshubraumhugo319 comments

Despite all the posts about the declining quality of Google, we haven't seen any serious competitors arise over the last years. DDG is at about 2% market share, which is great but still very low.

Apart from Google's monopoly and the big technical challenges, what would a competitor need to defy Google?

View on HN

Ultimately I think it’s because the internet is just too big! Google were only able to do it because they had the right algorithm early in the age of the internet. They were then able to grow with it to achieve the scale required. Starting from scratch now on a general internet search engine would be close to impossible without 10s, if not 100s, billions of investment. And you would need that to build the index before even beginning to be competitive. No one is making bets that big on search, especially when the online advertising industry (which is the only way to fund it currently) is in danger of massive regulation.

I think there is massive opportunity for domain specific search engines though, imagine a search engine specifically designed for software engineers and developers, or one for academic research (not just papers but all online scientific content, news and discussion), or one targeting the arts. I think it’s these verticals that could be incredible.

You then potentially move towards a building “meta” search engines (if your are older than about 35 you will remember these) that work out what you are searching for and uses a domain specific engine.

Edit:

Just to add to this, people who say that “decentralised” search engines are the only way to compete with Google are not completely wrong, it’s just that it’s not about protocols and distributed indexes. It’s about a community of smaller search engines working within specific domains and collaborating (commercially) on meta search engines, prompting people to search on each others engines if it would be better for that search.

We almost need an “Open Search Co-Op” which smaller search engines can join to share technology and refers users to each other.

I think your post is key. I would be interested in anyone posting what they think is an economically viable path to challenging Google. Pretty much the one company that has done it at scale (Microsoft) isn't exactly eating Google's lunch (though I think there are a few areas that Bing is better).

It would just take at astronomical amount of capital to challenge Google at generic Internet search, and the few organizations with that capital are pretty clear they don't think it's a good bet. For example, Amazon is quite content just being the place you want to go to first for the things you want to buy, as that is obviously the most lucrative area.

Even in niche areas, I think it's still an enormous task to challenge Google. Despite the fact that 90% of my searches for "WTF does this stacktrace mean?" end up on StackOverflow, I invariably start my search on Google because every now and then it gives me useful non-StackOverflow tidbits, and I know I can always drill in more with specific tags on StackOverflow later.

> Even in niche areas, I think it's still an enormous task to challenge Google. Despite the fact that 90% of my searches for "WTF does this stacktrace mean?" end up on StackOverflow, I invariably start my search on Google because every now and then it gives me useful non-StackOverflow tidbits, and I know I can always drill in more with specific tags on StackOverflow later.

It's telling, that a Google search scoped to a particular web site (xyz site:website.com) almost always produces better results than that web site's actual built-in search engine. If a web site that knows in detail its own content, its own users, and its own domain, still can't make their built-in search better, how does anyone else have a chance?

I don't think the fact that the web site owner hasn't implemented better search than Google for their site is evidence that they can't do it, simply that they haven't.

This entirely. Big companies grow complacent. Which to me, in time, equates to Google's timely downfall.

No empire lasts forever; when was the last time you went to Sears?

I guess the motivation is low as I frequently have to do this it seems.

Google knows about external inbound links, so has more data on which to rank your pages than you do.

Assuming someone has ever used that link you should have that data as well and with potentially even more relevance than google because you can see which pages are most often visited through external links.

I think the most important data that is difficult to obtain is a history of which links people click on for each search query term. I think that's really the special sauce that Google has that others do not. There is a catch-22 where you need a lot of users to get the data but you can't get the users without the data. Initially Google pulled ahead with inbound link counting, but anyone can clone that. It's the click data that's missing.

hah, that one is easy. People click the one at the top the most, then nr 2, after that nr 3, etc you see the pattern?

Google was epic when it started but quickly ended up indexing it self in a near closed loop. Everyone else was not important, it was all about the picture in front of them that was really just a mirror. Everyone stopped making home pages, forums and weblogs because it just doesn't make sense to do it.

Shit, people looking to buy something talking among their own before doing the purchase? KILL IT WITH FIRE!

my take is that the big bottleneck in search is not the index or the search but in understanding what it is the user is searching for in the first place.

that becomes a big can of worms very quickly, and starts to put into perspective some of googles other most successful verticals: NLP, Translation, Maps ("Near me"). Of course a lot of the verticals are also a generic means to get value out of all the server resources and experts required to operate the engine and then additional related branches to those verticals (email -> drive -> docs)

>It would just take at astronomical amount of capital to challenge Google at generic Internet search, [...] For example, Amazon is quite content just being the place you want to go to first for the things you want to buy, as that is obviously the most lucrative area.

I understand your intent with that domain-specific example but surprisingly, that reasonable intuition sometimes doesn't work...

E.g. for Amazon product search, I often deliberately use Google's engine to find the Amazon page because Amazon's own search can't do it. See example screenshots in my previous comment about that: https://news.ycombinator.com/item?id=27539199

Surely the only viable path to challenging Google is to own the web browser. Most users will use the default search engine.

I don’t know about “only” path but not sure why this is downvoted as it’s a valid point of view.

I would have thought this back in 2005, but eventually IE did fall. If users eventually dropped IE, why can't they also learn about Kagi or DDG or bing?

> I would be interested in anyone posting what they think is an economically viable path to challenging Google.

You only search whitelisted sites, which removes a lot of the SEO and blogspam from the results, and makes the index is wildly smaller. Search is free, but users can also pay a cheap subscription for the ability to flag sites as spammy, submit sites they think are missing, and submit queries the site has trouble with. Someone manually goes through the top N most-repeated submissions each day (early on this is all of them) and determines whether to tweak the whitelist.

Because your initial audience has to be people who are very enthusiastic about better search results, you should identify what they're interested in and focus your efforts on initial whitelists to things they care about. That might be technical questions (MDN StackOverflow, every random framework's docs site), or maybe whatever it is journalists would want to have access to (I'm not qualified to answer).

I don't think VC-encouraged hypergrowth is possible with the above strategy, but you could probably get to ramen profitable in a reasonable time frame, with slow growth ever after.

> I don't think VC-encouraged hypergrowth is possible with the above strategy, but you could probably get to ramen profitable in a reasonable time frame, with slow growth ever after.

The fate of runnaroo seems to provide evidence to the contrary.

"though I think there are a few areas that Bing is better"

Curious what you think these are. I prefer Bing's UI to Google's but aside from that Bing is (IMO) a dumpster fire that quite literally makes me feel negligent (towards myself) when I use it - am I missing something?

Porn, in particular, but surprisingly I find that video search in general is better in Bing. Surprising because most of those video searches end up on YouTube, yet I seem to get better/higher quality results when I search on Bing than I do when on Google or YouTube.

Would there be a way to crowdsource the web crawling effort? Crawling@Home? I think that even with this you'd need a legal department to deal with takedown requests and illegal material.

Could not agree more. Not only that, would be willing to pay, for a domain specific, high quality, ad free search engine. In the same range of prices of a Netflix or Amazon Prime subscription.

You do have a point. However, the internet today is very different from its early days. Its content is much more centralized.

In the past, Google search results will point to some good websites you never heard of and are very low traffic. But nowadays, the first couple pages are all popular websites with huge traffic even when they are low quality on the content. You can only find low traffic but good content websites in comments from social sites like reddit and HN.

So it seems to me, to build a usable search engine, you only need to index most popular websites.

There is an opportunity for niche applications. I would like a search engine that only indexes discussion forums. These are hard to surface on the SEO driven major search providers.

Kagi is working on this with its "lens" feature. I think it works pretty dang well. Most results I get when using the discussions lens come from Reddit, but you can block or reduce the ranking of sites, making this less of an issue, if it is even an issue for you. I suggest trying it out and joining their discord to share feedback (they're responsive).

So basically google is the MLS listings of the web now. Sounds like a job for regulators to step in for both the problems.

Agree we need a coop of search engines, but I'll add that probably an org like mozilla is best suited for it, that way it can stay non-profit and recycle any profits back into R&D. personally I believe most of these solid business plan companies can be replaced by a simpler open-source version as IMO a lot of their energy is spend in securing revenue lines & profits.

I suspect Mozilla would penalize sites that track you, which is most sites, which would cause people to not use it and go back to Google when it wasn't showing them what they expected.

Mozilla is probably in the best position to do it, I'm just not sure they actually would do it in a way that appealed to consumers.

Another way is to add search for walled gardens: The most common by far are corporate documents and datastores.

It’s also a harder problem to solve than Google’s algorithm.

After two decades Microsoft seems to have brought in support, but they left the hard part to others.

Isn’t Algolia pretty much eating this market?

They seem to be very dominant on the problem of “here’s my dataset, please provide me with an api / interface to search it”

Maybe?

When searching for something on HN, I get much better results using Google and site: rather than using the embedded algolia search.

I would have to think that this would be true of a corporate data set as well.

Their api might be what sets them apart, though. I’m not sure… not in that space.

Maybe a search engine for niches, where the mid-sized contributors to those niches may even pay something to support the search engine? And new people contributing sites to those niches could pay some small fee to be listed (maybe under certain conditions or after 1 year free) and would be "elected" into the search results by a random selection of others in the niche? Where the interests of the group are aligned because they all want more people to use the search engine to find them, so they are incentivized to grow the amount of quality content available.

`We almost need an “Open Search Co-Op” which smaller search engines can join to share technology and refers users to each other.`

Can you elaborate?

Something like a cooperative where companies become the franchisee owner for a specific search vertical. They could get access to technology, software and knowledge, but the costs of infrastructure would be the responsibility of the member. As well as having their own “front page”, website and branding there would then be a “meta” search engine that intelligently searches the franchisees search engines based on the query.

If someone searches for something on one engine and it doesn’t have the results the “meta” search engine could be checked and direct the user to another engine that would better fit their query.

To clarify, if I search for "Bob Marley" what would happen?

I think are right but not because of the size of the internet. Although the web has grown, the cost of computing and bandwidth has decreased and there’s a lot of open tools, technologies, and open datasets like YaCy and Common Crawl. The data you would want to index is a tiny fraction of the total internet.

The challenge comes from the massive investments in turning that data into a good search engine and cutting through spam. That is the investment I think Google has scaled over time. Secondly, mind share; the moat Google has is immense, although I think DuckDuckGo and Bing has shown given enough time it is possible to get mindshare slowly; though if anything this demonstrates the challenge.

> Ultimately I think it’s because the internet is just too big!

I would disagree actually. I think the majority of searches people do and the solutions they would be satisfied with require only indexing major sites. Create a better UX attached to PageRank of the top 10-100k+ Alexa ranked sites and you'd have a competitor that could focus on other improvements.

At the bottom you could search Google or DDG if you want more niche results.

I.e., optimizing UX of the p90 searches would be enough for people to move and the p90 of searches are happy with a pretty narrow range of sites. (You could insert value prop of: no ads, no tracking, and crowd-sourced refinement -- i.e., realtime upvote/downvoting -- of the top results, previews, caching, more knowledge graph stuff on the right side, deeper metadata processing of top sites (i.e., avoiding the need to 'site:*'), or some other revolutionary UX like a one sentence hypothesis solution of exactly what the person is asking for again upvoted/downvoted for refinement -- Google does this but there is a lot of room for improvement). It's fun to think about. There is an order of magnitude in both precision and accuracy that can be improved for huge swaths of common questions people ask on Google.

The internet is big, but information has shrunk. Content has been slurped into walled gardens.

> Ultimately I think it’s because the internet is just too big!

This doesn't address why MSFT can't compete with Google. If you solve that problem, then you really get to the heart of the solution.

> No one is making bets that big on search, especially when the online advertising industry (which is the only way to fund it currently) is in danger of massive regulation.

Isn't this exactly the reason why one should do it, as Google is the essence of the online advertising industry and this very regulation could threaten its existence?

>I think there is massive opportunity for domain specific search engines though

Challenge with niche search engine is discoverability. Imagine you had 10 of the greatest vertical search engines available today - how do you remember which one to use and how? The beauty of a general web search is that it is an easy to remember entry point to all of what is out there.

> You then potentially move towards a building “meta” search engines

Ever wondered why is this called meta search engine, and an email software that connects to multiple email servers is called email client and not "meta email". "Search engine client" sounds like a more appropriate name.

The internet is too different now to what Google (or any SE) was designed to index.

Too much of the up-to-date high value information is posted onto semi-public channels. Like facebook groups, instagram posts, telegram channels, etc.

At the same time, too much of the publicly available websites are becoming clickbait and mindless marketing drivel.

Just try to research anything slightly obscure that has to be up-to-date information (from the last few years).

Like "how to get from Nairobi to Kisumu by bus". Google maps doesn't have this (not surprisingly), but what about the SERPs? Nothing there. 90% of the results are useless bus ticket sale websites that have no info, just SEO. (You can't buy this ticket online anyway).

Is Kenya just such an off-the-map destination that no backpacker has ever written about taking this route? No!

The information is out there. You'll find it inside the related facebook groups for travelers in east Africa. And if not? That's where you'll ask. But the answer? It'll never reach a SERP in the future.

Even more sad, is that if you keep looking on Google, you may find buried results from years ago. From travel forums and such. What's sad is that the info will be 10 years old, since no one uses public forums anymore. (Unless it's on reddit, pretty much the last public forum with any reach).

Somewhat related is how there is a whole generation of people that think they are entitled to finding information on a search engine, or a web indexed accessible source, and if its not there then it must not exist!

I see this in discussions a lot where many times the answer is that the data doesn't exist, but thats not satisfactory to the person making the challenging.

Anecdotally, I'm in a couple+ tech (ish) groups on FB. More and more the questions posted are obviously by passing internet search (e.g.,Google) and going right to the group. Simple shit that people simply can't be bothered to search for, and they don't even bother with the FB group search. Imho, this behaviour wastes too much of the groups' collective bandwidth.

So not only has FB hoarded the infomation, but it has no incentive to fix the group experience (e.g., prompt an input for group search during the asking bit) as it keeps engagement up.

I mean, this is the state of the world currently. I doubt I'd be able to do my job without some sort of search engine. I do talk to people in my company who worked in tech before the internet and they mention going to bookstores and reading books and how weird that seems now even to they who lived through it.

You know, if that were the only roadblock, I wish someone would take the top 10,000 Google queries and just manually curate them. Hire different interest groups -- outdoors, travel, cuisine, etc. -- and manually find the best hits buried deep in some subreddit. No fancy algorithms, just old-fashioned librarian-style research, but constantly updated with the latest findings and queries.

Would happily pay for that... even if it only has 5% of the coverage of Google, that's fine because Google is like 95% noise anyway.

+1 I see this repeated over and over in tech where an algo is great but why not manually craft the top value queries?

Like with smart speakers, why not manually add a bunch of interactions? You can do it yourself in settings by adding a phrase and desired response, why not have the team adding a bunch they would want?

The top 10.000 queries at Google are probably 99% brands, pop culture or current news things, one word or name only, that you would not care for at all. Have a look at the top 20: https://www.statista.com/statistics/265825/number-of-searche...

I think there is an easy exclusion to be made for these types of 'too lazy to type a url' type searches (or 'weather' which is similar in getting a specific piece of data).

I remember one of the top queries we worked at one company with was the number of the US form to fill your taxes

>, I wish someone would take the top 10,000 Google queries and just manually curate them. Hire different interest groups -- outdoors, travel, cuisine, etc. -- and manually find the best hits buried deep in some subreddit. No fancy algorithms, just old-fashioned librarian-style research, but constantly updated with the latest findings and queries.

It's not exactly your specifications but 2007 Mahalo attempted something like that and they shut down after a few years:

https://en.wikipedia.org/wiki/Mahalo.com

The website is no longer there so you have to use Wayback Machine or image search to see what the landing page for Mahalo search UI looked like : https://www.google.com/search?q=%22mahalo.com%22+search&tbm=...

You just invented pre-Google Yahoo!

I know. I remember and loved it then.

I also remember altavista and the like and how bad they were. When google came on the scene it was a total game changer.

Google was definitely better than the algorithmic crawlers. But Yahoo often still had gems for a while, a few years maybe, before they gave up. It's too bad.

Isn't that more or less what Yahoo was supposed to be (an index of curated links)?

Yes. There was a period of several years when Google's results were better than Yahoo, but once the human search engines bankrupt, the algorithmic SEO just kept getting worse and worse =/ Too bad...

The problem is, you probably won’t pay for it.

Google’s revenue per user in the US underlines that point, taking a credit card out for an annual subscription for a $100 probably isn’t going to score a good conversion rate for anything beyond a very small userbase.

The predicted economic/behavioural constraints are inhibiting innovation. You’re forced to play Google’s game.

I'd gladly pay for a good search engine, but I'm probably in the minority.

That would be awesome!

This. Whenever I search anything informative, I append “Reddit” to the phrase to get the top Reddit threads about the topic. Everything else in the SERPS is usually SEO trash

Try https://chrome.google.com/webstore/detail/hypersearch/feojag... then.

Is there a way to provide a date range when specifying a site? I frequently append reddit to my google searches as well, but the results often include outdated posts. I don't want restaurant/service/whatever reviews from 7+ years ago.

Press "Tools", then change "Any time" to "Custom range" and set the range you want.

I've tried that but the filter seems inconsistent. I can specify a date range and get some focused results, but if I remove the date filter and search the same terms I often receive additional results that weren't previously displayed but are within the range I used.

after:YYYY-MM-DD & before:YYYY-MM-DD work on Breeze, Google, & some others, e.g.,

- recent March Madness results; thread of examples, https://twitter.com/DotDotJames/status/1506826439961858052

- hot-off-the-press startup funding, https://twitter.com/DotDotJames/status/1504901028520640514

- new free trip contest announcements w/ query breakdown, https://twitter.com/DotDotJames/status/1506443371828686848

after: & before: cover many use cases, and can be used in any combo

before X date, before:2022-01-01 range of X to Y dates, after:2022-01-01 before:2022-02-01 multiple ranges, easiest to split up queries since X date, after:2022-03-14

& per OP question, can be used with site, e.g.,

after:2020-12-31 before:2022-01-01 site:news.ycombinator.com intitle:search

all posts w/ search in title during 2021 on HN

https://breezethat.com/?q=after%3A2020-12-31+before%3A2022-0...

after:2020-12-31 before:2022-01-01 site:reddit.com intitle:russia intitle:ukraine

all posts on Reddit during 2021 w/ Russia & Ukraine in title

https://breezethat.com/?q=after%3A2020-12-31+before%3A2022-0...

(all posts with search in title during 2021 on HN)

there's also some more advanced techniques, post on that forthcoming

> It'll never reach a SERP in the future.

Why not ? A scrapper may not be able to find it, but we all have a Facebook account. You dont need an automated scrapper to create a search engine. Info can be manually curated or scrapped through web extensions

> but we all have a Facebook account.

Nah.

> but we all have a Facebook account

God no.

I don't have a fb account. I am very happy about that :)

Either Google engineers reacted to your hour-old post, or you picked an example where Google actually does well.

The top-of-page "featured snippet":

> Kenya Airways and Jambojet Limited fly from Nairobi to Kisumu every 4 hours. Alternatively, The Guardian Coach operates a bus from Nairobi to Kisumu every 3 hours. Tickets cost $6 - $11 and the journey takes 6h 30m. Two other operators also service this route.

Then the rest of the page links to route-planning websites, a Tripadvisor discussion about bus from Nairobi, and other actually relevant results.

Google disappoints me too, but TBH I can't objectively gauge how good it is. My own opinion is just based on spotty personal experiences that I don't keep track of.

You are actually right :) Stupid me. But other similar searches have failed me in the past.

Your Nairobi question results in the top answer today on [https://millionshort.com/] It has some info, not sure how useful.

Yandex gives good results on the first page[1] with tripadvisor and other community sites. Google I think got majorly messed up when they started doing "anti-misinformation" patches that down ranked forums and other community sites where people can freely post wrongthink. Yandex doesn't do any of that.

[1]https://yandex.com/search/touch/?text=how+to+get+from+Nairob...

>, what would a competitor need to defy Google?

One thing a competitor needs is a new and innovative technical algorithm.

Back in 1998, Google's PageRank was an innovative algorithm that calculated relevance based on counting back links instead of parsing the word counts in embedded HTML text like other search engines. This created a noticeable improvement in quality of results.

Nobody seems to have The Next Big Idea for a better search engine yet. Somebody did a Shown HN of a new search engine based on whitelisted curated domains such as reddit discussions. But there are many technical problems with that (e.g. Goodhart's Law & Hawthorne Effect creates bad feedback loop of gaming the reddit discussions which then poisons the search engine.)

Another technical idea of crowdsourced decentralized search index creates a very slow query engine which is a hard sell when web surfers are used to Google results appearing in less than 1 second.

DDG's idea of "privacy" is interesting, but being (mostly) based on Microsoft Bing's search engine doesn't actually create a quantum leap in better search results.

What's the next breakthrough idea that extracts the good signal from all the noise of a trillion web pages?

There’s a potential technical improvement with Vector Search. Specially for the type of queries that are more “human” (“how do I…”, “where can I…”.

At least for me, I find myself adding “site:Reddit.com” all the time I need a good answer and not just a SEO-tricked ad page. Vector Search could be a technical solution for this. Although it’s fairly new and there is still a lot of research to do.

Yeah Google does this already though

I like the idea of "crowd sourcing". When the SEO-tricked ad pages come up and I either scroll past them or, if I do click on them but come right back to the search results and continue scrolling/clicking ... rank downward those brief or passed-over links.

Seems like with enough actual humans I should never have to see another geeksforgeeks page come up in my search results.

Heh, Geeks for geeks wasn't that bad when I frequented it a few years back.

> counting back links

Do the algorithms weight links based on visibility or relevance? A link at the top of a sorted content site HN should probably be worth more than one at the bottom, and one buried in invisible metadata somewhere should be worth 0.

PageRank is a neat algorithm that recursively weights incoming links based on their own incoming links. So a link from HN has more weight than from a small blog. And how does it know that HN should have more weight? Well, because there are many important sites pointing back to HN. And how does it know that those sites are important? And so on...

But AFAIK Google no longer uses PageRank, or at least not the same version as we know it.

> What's the next breakthrough idea that extracts the good signal from all the noise of a trillion web pages?

To me this is going to be manual ranking, but the biggest issue with manual ranking is not that you have to do it manually, but how do you know if you can trust users who rank pages. The Wikipedia system is a good start but not enough, and the second issue is how do you prevent its business model to kill its quality

You let the rank sources be selectable. Allow me to connect/select individuals or groups to include/exclude.

It seems the problem of "search" is mostly well solved. What isn't well solved is the problem of "SEO gaming". Perhaps the advancement in search isn't by having an AI that chooses a better page, but rather an AI that is really good at identifying a good result from a gamed result.

However as for Google, I feel like even merely surfacing a range of relevant filter options can help their search engine, since it's what informed users are already doing by structuring a query when the first one produces crappy results. E.g. These results mostly contain links to Pinterest.com, would you like to repeat the search with these omitted? (or even just being able to collapse these site-by-site.)

Additionally I think modern search could be broken down into a range of sub-tools that exist beyond google or even online. One such example is searching our web history, perhaps browsers could produce a privacy-preserving way of storing information about each webpage and its images that the user can then search later (by either terms or images.)

Read an article which mentioned that Google degraded the search service because good results no longer made financial sense, perverse incentive is that it benefits them better for a customer to pay money for ads than provide a good search (meaning that competition for keywords is now the algorithm). The next innovation in search then necessarily must be to find a different funding model for the engine besides ads, otherwise the same perverse incentive will arise.

This hits the nail on the head. I feel like Google results have gotten worse not because technical problems need to be solved, but because the financial incentives means that I'm now scrolling through tons of ads, little "tidbits" that have just gotten more annoying over time, AMP carousels, and god forbid I'm searching for anything that happened over 20 years ago that has any slight resemblance to a current event - all I get are pages and pages about the current event, and Google has "memory-holed" the past.

IMO, all startups have a sweet-spot where they've figured out what the market needs, and they can be profitable, but they haven't yet sucked that bone dry to where they're forced to extend more tentacles to grow their revenue. For Google, for me, that was about the 2007-2012 time frame.

Any other competitor would have the same financial pressures eventually. Long term, it's impossible to "not be evil".

That's ridiculous. Google's search dies if people stop using it. They stop using it if the results are poor. There are no perverse incentives here unless you believe Google is hoping to cash out tomorrow and close its doors.

It's not ridiculous at all.

> They stop using it if the results are poor.

These are not laws of nature. There are so many reasons why most people would still use $X even if the performance of $X was underwhelming.

The quality would have to degrade dramatically for people to stop using Google em masse for that reason. It can still degrade a bit to increase revenue.

DDG also lost the plot when they started censoring results based on geopolitical opinion.

Evidence?

https://www.nytimes.com/2022/03/11/technology/duckduckgo-rus...

In practice, it used to say that 2+2=5 and they blocked that. How is that wrong?

> What's the next breakthrough idea that extracts the good signal from all the noise of a trillion web pages?

Here's another recent Show HN that demoed a new search engine:

Show HN: Goopt – Search Engine for a Procedural Simulation of the Web with GPT-3

https://news.ycombinator.com/item?id=30443738

I know this was offered as a bit of joke. But it prompted a really thought-provoking discussion and seemed to me to suggest a genuine paradigm shift in search or information organization. It is a novel way to extract a useful signal from all the noise. It brought to my mind "Dr. Know" from Spielberg's AI movie:

https://www.youtube.com/watch?v=x0QkgAuEPbk

Will the signal be a good one? It will need to get better and inevitably will. The thing that worries me: will it even matter?

The only viable alternative to PageRank I know of is social search: show me results that my friends (and friends of friends and so on) liked/bookmarked.

Delicious sort of tried it but never executed well on the search side.

In a way it's a personalized PageRank algo with users linking to pages they like and to other users they follow. This always made more sense to me than a global authority ranking.

When looking for a babysitter you want recommendations from friends you trust, not some global ranking of popular babysitters.

I've heard this a lot. I think there are two problems with this. First, you need massive adoption for the "social" part to be at all useful, hence only the few big social platforms can even try it. Second, even at maximum possible adoption, you'll probably run into the problem that actual human social networks have nowhere near enough branching breadth to help rank the behemoth that the web is today, i.e. for most queries and most possible hits, there's not enough signal coming from the social network.

The benefit of this approach is that it's an addition to PageRank, not a replacement. You don't need massive adoption of the social part I think cause you can link to eg HN, which already links to other sites. It's simply extending the backlinking concept to add personal pages.

Most long tail queries work well enough with Google and indeed most of the benefit of this approach would be in the head where ranking matters more. Furthermore nothing prohibits users from liking domains, not just individual URLs.

"next big idea" is probably "selectable algorithms".

Create 3-5 different search algorithms, with names and understandable characteristics, then let me choose which one when searching. You may have some algorithm that prioritizes recency, or rapid changes, or more interaction, and others that prioritize longevity, citations/links, others that prioritize that type of information - prioritize regional or noteworthy events over historical info, etc. Have some algorithms that penalize spam/linkfarms more heavily than others. The git/so spam farms that come up when I search for an error message are simply useless, and would love ways to switch away and try other algorithms, without necessarily having to switch entire search systems.

Have algorithms that explicitly take in to account the search/click behaviour of others. Let me choose to have the search behaviour of friends/colleagues/famous-folks explicitly included or excluded in search results ranking.

Let people easily switch between multiple algorithms, and let 1000 gardens bloom (or something like that).

Also, determining better ways to 'search' non-text stuff. If virtual worlds become a thing, finding ways to 'search' those (whatever that might mean) will be big.

Image search still has a long way to go, IME.

This could be a niche search engine for advanced users, but I feel that to average users this would be a strictly worse experience because they have no interest in figuring out how to "select an algorithm." They expect to type a query and the result they want comes up at the top. The search engine should be able to infer whether the user wants results that are more recent, more interactive, more authoritative, etc. based on the query, past search behavior, and anything else that is known about the user. If that isn't happening, the search engine needs to be made "smarter."

The way it's likely seen by many tech companies is that the more decisions you have to offload to the user, the less advanced your system is. An old car with a lot of knobs and dials and levers is not more advanced than a self-driving car with a single control, select destination. Some people will still prefer the old car, but it will be a niche market.

3 big buttons at the top after search results

"You're viewing through FOOBAR lens. Click here to view results from another lens".

Doesn't have to be that complicated.

To some extent that's already implemented in the form of the various tabs - if you want very recent results, go to the News tab or click the Tools button and select a time range. If you want academic results with citations, go to Google Scholar. If you want information from books, go to the Books tab. If you don't know/care, the main tab will mix results from all tabs.

You can add three more buttons on top of the tabs that are already there, but would their value for more advanced users be worth the extra friction/confusion for average users? And how would it scale if someone wants a fourth, fifth, or tenth "lens"?

I agree, it’s the classic example of “people don’t really like choice”. When you are presented with the option of a search algo, there’s always this feeling that you chose wrong. It’s an additional thing to think about, and makes things more complicated.

it doesn't have to be an irreversible choice. clicking between 'filters' should be something that takes an instant, and is one click away.

> Image search still has a long way to go, IME.

Try images.yandex.com: it's vastly better than most others. It's not too good at identifying specific people but it will find prior exact uses of an image (like TinEye) but is especially very good at abstracting what's the abstract subject, color, pose, setting etc in a photo and show you similar.

Also Bellingcat recommends Yandex: https://www.bellingcat.com/resources/how-tos/2019/12/26/guid...

This is what Brave Search calls “Goggles” [1].

[1] https://brave.com/static-assets/files/goggles.pdf

EDIT: adding on to previous post.

We now have a couple digital generations of folks who've grown up with 'web', and we've generated billions of 'documents'.

TIME ACCURATE QUERIES need to be mainstream.

"Let me search for FOOBAR and get results that would have been accurate/returned for Dec 11, 2006. Now search for FOOBAR and get results from Dec 11, 2014", etc.

I agree with you on this. I search for tech topics a lot and google offers an option to search "scholarly articles". I like that option as sometimes a tech term gets confused with something else. It would be great if there was a way to select from some check boxed, like an advanced search, "search only for semiconductor datasheets" or "search all Jazz music" or upload a picture and "identify an insect" (agree that does not work well yet) or whatever someone might be interested in.

The question is if such an approach would lure a lot of people away, I could see an entry angle might be a front end to google with an app that does that.

Whatever Algolia is doing, it is amazing. Would it be possible for them to scale as a general search engine for the web?

It's not going to be 'search' that displaces search, it'll be impossible. It will be made irrelevant through something else.

For example, someone comes up with a 'Finder' thing that's barely a comprehensive directory of the internet, and that, along with 'site search' aka for FB, Amazon, does better for the use case.

Most people are not 'searching' the web, they're often typing URLs or searchers for things they want answers to.

When I type a technical thing I'm really just searching 'stack exchange'.

On mobile, we use apps, which are different than pages.

Apple could come up with something that renders Google less than necessary.

Google will not be replaced directly.

> One thing a competitor needs is a new and innovative technical algorithm.

Not really, because Google can do algorithms. For a new competitor to really succeed they have to start with something Google can't do. Perhaps:

- device/platform related. Google owns chrome & android but a competitor could emerge somewhere out of their reach, like only on iPhone, only on airpods, alexa, oculus, etc.

- being worthless to 99% of the market to make it 10X better to a 1% audience (e.g. developers, nurses, or even more niche like college athletes, jewish moms, etc)

- privacy/ad related

Google pays Apple ~$10bn a year to be the default search engine on Apple devices. They pay Mozilla a significant percentage of Mozilla's revenue to be the default in Firefox. Obviously Google is the default in Chrome.

You can't compete with that by building a better search engine. Even if you had the infrastructure, capital, and you offered Apple and Mozilla more money than they get from Google, Google could beat your offer.

The only way someone could beat Google, or even just compete with them, would be if they could make everyone understand that they can change their default search engine and give them a reason to do it and have Google screw up how they respond. I wouldn't bet on it happening.

You could lobby the govs to require search engine selection in browsers / device on first install. I don't think that would change anything though. Google's results might be declining but they are still arguably better than their competitors and ATM at least the majority of people would still choose them.

You could beat them in a small segment market though. EG: A group tired of being banned by google.

> EG: A group tired of being banned by google.

Huh? I don't understand what this means.

Duckduckgo wrote about this a decade ago. So much investment needed to just get on their list of search engines.

Meanwhile on Android, there's no practical way to completely move towards ddg for searched info unless you basically get rid of some voice and search features.

Without laws, Google set up an ecology of everything search related hits google one way or another. On Apple there are ways around it because Apple isn't a search company.

Having said that......... Most people don't care. It's why Microsoft pushes so hard for ie. Because they get to set the default which nobody will change.

What if an entity like the EU mandated that google make a public search index API, which would allow any competitor to create a low-cost clone. Google gets to stand as it is, relying on its brand and extra suite of tools to compete.

It's heavy handed, but if you view search indexing as a natural monopoly, it's relatively simple for how effective it would be at curtailing market power.

Contrary to many posts here, I think that Google completely controls the quality of the results. The feeling is that quality has gone down because a lot of top ranked website are SEO ads filled websites. I think this is true and on purpose. Two years ago top results were very often Wikipedia, Stack overflow or reddit. Sending users to those websites is not profitable for Google.

I also think that's why it's very hard for competitors to compete. As soon as one is going to get credibly menacing they'll improve on the quality again to preserve their position.

I think that has happened with McDonald's for instance. A few years ago the quality was at it's lowest, the food definitely started to feel fake, air filled and not satisfying. Since the small local competition of hand crafted hamburger popped up a bit everywhere, and credible international competitors ramped up (5 guys, in n out...) they had to improve the quality. And my impression is that it's been a bit more than a year that the quality dramatically improved, and their hamburgers actually taste like food again.

I think it's simply the quality versus profits cursor and it can be adapted any time.

> And my impression is that it's been a bit more than a year that the quality dramatically improved, and their hamburgers actually taste like food again.

Huh, maybe I don't eat there enough to notice differences, but on the occasions lately that I've had a McDonald's burger, I haven't noticed the quality being anything but terrible. Not entirely unpleasant, but just not at all a high quality meal, even compared to some of the other fast food burger places.

Well it's still McDonald's we're talking about, it's still going to be 3mm thick patties and very sweet sauce. But with less soy derivatives, non caloric sweeteners and corn instead of wheat I have the impression.

I don't quite understand what you mean when you say "sending users to those websites is not profitable for google" - referring to Wikipedia, Stack, Reddit etc...

Organic search results only ever profit Google if they answer the user's question, because it makes that user more likely to use Google again and click an ad in the future. If a user searches something and the top result doesn't answer their question, then Google hasn't done it's job and the user is less likely to use Google again and therefore less likely to click an ad in the future.

Surely the most profitable option for Google is to answer people's questions quickly and easily, whether you think these questions are answered by Wikipedia instead of a business site targeted to answering that question is a different story.

I just don't understand how Google's goal of "answering people's questions" is somehow opposite to "providing quality" for users. I think for most of HN, they have a very narrow view that "quality" means antiquated wiki-type sites.

> If a user searches something and the top result doesn't answer their question, then Google hasn't done it's job and the user is less likely to use Google again and therefore less likely to click an ad in the future.

This is simply false. It would only be true if there was a better alternative to Google, which there isn’t.

Because those websites don't contain ads provided by the Google network, while Google can choose to send the user to websites that contain ads for which they are going to take a cut.

Obviously the best for Google would be to send the user to a website with the best content and that contains Google ads but that's often not both the case.

It's an optimization problem, and generally you cannot optimize for two variables, profits and user satisfaction.

Maximizing only for profits would probably lead to completely garbage results so they probably have to compromise.

They would need to be Apple, they would need to understand how to write software for this purpose and then deliver it, and they would need to be able to make this the default search engine for iPhones without antimonopoly distractions.

Beyond that, Google’s dominance isn’t going away due to their own “monopoly-like” positioning, their business relationships, the inertia of their massive public adoption, all of these applied within three or four other vital areas (ie: YouTube), their ability to pivot in response to anything novel that appeared to undermine their position, no apparent stories of Google executives hosting puppy kicking parties for the entire company to Satanic Panic everyone away from their products, the problem of promising companies being acquired because the owners (VC or founder or otherwise) are happy to be bought, the general “ick” factor of someone like Facebook attempting to enter the fray, some other things none of us have ever considered, random luck, and the initial conditions of the universe.

You have to have good results against the inconceivably vast amount of content out there, you have to be easy to use, you have to be free, and you have to be able to do all of this and more for the yeeeeeears it would take to wiggle into the space and expand while resisting the pressures above. Doable, wildly improbable.

Sounds like a government intervention is needed and the monopoly should be broken down for a better functioning market.

However, I guess big tech companies have become part of the superpower games (USA vs. China etc.). Breaking up Google might just mean a Chinese company takes over. Can't trust the other governments to enforce similar market conditions.

So yeah, like you said, conditions of the universe :)

> Sounds like a government intervention is needed and the monopoly should be broken down for a better functioning market.

Well-crafted arguments showing where Google (or ilk) are anti-competitive are likely to gain some traction.

>...Sounds like a government intervention is needed and the monopoly should be broken down for a better functioning market.

You can break up the phone systems because the child companies can provide similar levels of service. But how do you break up a single search algorithm?

You could discourage them from buying out smaller competitors en Masse.

In my mind it shouldn't make sense to found a company with the explicit goal of being purchased by one of the tech giants in a few years.

Many are never even really trying to get s sustainable business model and venture capital is fueling this machine.

Why shouldn’t it make sense?

One point of view is that it’s a more efficient way for the tech giants to develop new features. An internal team trying to do greenfield work will inevitably be slowed down by bureaucracy, where a startup can iterate more quickly without all the friction of things like performance reviews, HR exercises, and if I’m being cynical, pesky issues like user data protection frameworks.

It’s risky, but the payoff for founders is significantly larger than what an equivalent employee would get for leading an internal project.

Google seems to have optimized its search engine with the goal of making it as easy as possible for potential consumers to connect with Google advertising customers. Google also collects data on consumer search patterns again with the goal of connecting the consumer to the advertiser. That's the business model as far as I can tell. If you don't want to be sucked down their engagement hole, you have to clear your cookies after every search and probably going through a VPN is better.

There is an obvious problem here - companies who don't advertise with Google could easily find themselves blocked from appearing on the first pages of Google search results. Would anyone really be surprised to find that the internals of the Google search algorithm have a weight factor that assigns better scores to sites that advertise with Google? Hence requiring Google to expose the internals of the algorithm to public scrutiny really seems like the only way forward. Yes, those are 'trade secrets'. So are the contents of your favorite hot sauce, but government regulators require those contents to be displayed on the product label (for good reasons).

On top of all that, Google's also under pressure from governments and media corporations to push their information content (aka propaganda, influence, etc.) to the top of search results, burying anything like independent content in those areas (world events, domestic politics, etc.) far down. This is particularly obvious on Youtube incidentally, but Google has the same problem. Some of this can be avoided using the 'verbatim' option and some interesting word choices in your search string, but it's a fairly tedious exercise.

Again, exposing the internals of the Google search algorithm would be interesting here, as it seems clear certain 'authoritative sites' are assigned better scores in the search ranking - not because they have more backlinks or more accurate content, but merely because of governmental and media pressure.

My opinion hasn’t changed much since writing this (just disregard the blockchain commentary at the end):

https://www.quora.com/Is-it-possible-to-beat-Google/answer/M...

A few competitors such as you.com are in the works. I think there’s a lot of potential, but the ranking algorithm is going to need to be decentralized (aka user-configurable) in some way. This configuration would also need to be sharable, maybe something akin to an App Store, because not all users can be asked to be technical enough to configure everything.

There is an enormous amount of wasted value stemming from the disconnect between what Google thinks the user wants to see, and what the user actually wants to see.

If I had to guess why this happened, I’m betting that Google and others had a notion that AI and other advances could eventually approximate a user’s needs to a degree that would keep up with the demand. But the monopoly set in and there was no longer an incentive to innovate that far. I’m sure Google tried, but not to a degree that would have happened if the competition had really stepped it up, or even existed. However, I’m sure by now there’s less optimism that the above effort is even a solvable problem, or one worth solving. One-size-fits-all is extremely suboptimal, but that paradigm is Google at its core. Without a paradigm shift, it’s diminishing returns all the way.

What Google is good at, however, is monetizing what they do have. If you’re trying to challenge them in the monetization game, with the same business model competing for the same customers (advertisers), you most certainly will lose. Seriously, don’t do that. Get to unicorn status first.

Focus on the tech, make sure the business model isn’t competing with the giants, and say no to any buyout offer from Big Tech.

the ranking algorithm is going to need to be decentralized (aka user-configurable) in some way

What do you mean? What configuration options would you like?

Okay, one example:

Suppose you're a gamer - You could configure search results related to gaming to be limited to games you own, games you want, the subset of platforms or genres you care about, games your friends own, and so on. It wouldn't be a priority for most search companies to make something like that, but it would certainly provide some value for certain people. Decentralized configuration would allow for communities with given interests to configure and distribute customizations for that audience, in a way that would appeal to that audience. It's not that farfetched of an idea, very realistic both from a tech and adoption POV.

You.com is starting the building blocks of such a thing, but I don't know how far they're going to push the envelope. https://about.you.com/apps/

I've been pretty stoked on you.com so far. I do like the ability to pick my search sources (apps). It would be cool to see search results for different genres/communities.

`see search results for different genres/communities`

Can you elaborate?

I just like and agree with conceptually Nuzzerino's idea above.

"Decentralized configuration would allow for communities with given interests to configure and distribute customizations for that audience, in a way that would appeal to that audience."

Couldn’t the user just use a !bang for that?

I think 99.9% of users would hate to have to configure their search results.

From parent comment:

> This configuration would also need to be sharable, maybe something akin to an App Store, because not all users can be asked to be technical enough to configure everything.

That's still configuration. Sharing a config? Asking a friend for the best search algorithm? Most users are accustomed to just typing in their search query into random input fields and getting perfectly serviceable results.

They're free to keep using Google if they want. My grandmother preferred AOL for many years after its prime. No shame in that.

Or, as with any other software, there's this thing called a default, which usually works for the average user.

The Google hate is so thick that people don't even think about what a Google competitor or replacement (that isn't Google rebranded) looks like. No user is looking to pass around configuration file, or have discoverability issues because they haven't joined the right community that has what they are looking for. People want an omnibox that takes them to what they are looking for, even if they don't know what that is, full stop. Needing to essentially have a Makefile to find out "Who is the main actor in Spider-Man" is dead on arrival.

Absolutely no need to provide a makefile. But... there can be multiple selectable algorithms for people to choose from, and eventually some ways to configure those to your needs.

Having results come up that are also explicitly labelled "You are viewing results through the FOOBAR lens - click here to use another lens".

Our results are already skewed by factors that we don't have much visibility in to - being more explicit/transparent about it, while giving people some ability to influence/change the lens they view results through would be useful.

> the disconnect between what Google thinks the user wants to see, and what the user actually wants to see.

Results being what Google wants the user to see seems to be the real problem.

Youtube for example in the past would show suggestions from small channels that were interesting and fun, but now you can't even find them in search using an exact video title. I've seen Google News completely shut out a breaking news story for half a day and the only reason I can think of was the ranking AI saw some 'bad' words in every story - even from NYT/WaPo, whose articles were among the many dozens on Bing News. Presumably they've applied their anti-"I hated it" ideas to Search as well.

So now is a great time to compete with Google because they're handicapping themselves and even though they could just turn off the bias and return good results, they probably would rather lose than stop trying to influence users.

DDG is absolutely emerging, usage year over year is rising rapidly. I’ve personally switched over to ddg after years of jumping back and forth. It’s finally good enough. For the Indernet as whole to switch over will take time. One thing that keeps customers coming back to google is the integration across their applications, i.e. email, collaboration (google docs) video (youtube) etc. That being said, I don’t really feel theres much of a benefit being logged in when searching, so I think search engine traffic could switch over to e.g. ddg while people still are heavily invested in other google services.

Their Android browser is great. It kills ads and floating videos, and has excellent features.

However I just can't rely on the search results. It's rare for me to get a decent result for a search so I switch between chrome. I'm not sure why because I thought DDG used Google search.

DDG uses Bing (like most other alternative search engines). I believe Brave's new search engine uses its own index though.

Exactly, they use their own index, which AFAIK can be seeded by other search engine results. Here's an overview: https://support.brave.com/hc/en-us/articles/4409406835469-Wh...

despite the HN party line on this subject I think DDG is practically useless for long-tail searches. I was reserving the history of a city owned property in Oakland, California on DDG, using the address and the assessors parcel number. DDG results were all unrelated real estate advertisements in other cities. Google returned many pages of official city records. It’s almost as if Bing is not capable of indexing PDFs? Whatever the cause, this put me off Bing for the time being.

The complaints regarding google's quality isn't because google is being bad or doing something anti-consumer it's just that spam and ranking has become harder and harder over time. A new competitor wouldn't be free from those issues.

I vehemently disagree with this. Gmail is an excellent example of worst-in-the-business spam filtering today. Literally everyone else is better at it now. Google is getting worse at things that were originally their core competency.

Spam is only hard if you're accepting low-quality content and not pruning it heavily. But Google can handle the scale of the spam, they just don't want to reduce it because the spam increases their ad real-estate. That's one theory, the other I would argue is that once you peel away the genuine spam and the blogspam, there really isn't much content left on the web (besides a small handful of big sites like Reddit) and Google knows this.

Everyone complaining about google’s “declining quality” is living in some fantasy world where there’s something better to compare google to. Its results are significantly better than any competitor, check for yourself. If you think you can do better, go for it.

“declining quality” means current google vs past google, not other search engines

This is also not a fair comparison, because Internet is changing constantly.

Seems fair to me. They did improve quality for quite a long time. The quality only seemed to start dipping when shareholder expectations started driving what SERP pages look like.

Or, roughly, "The goals of the advertising business model do not always correspond to providing quality search to users"

Yeah, and large near-monopolies grow complacent.

In 2014, I was cursing the shitty search results of GDrive - and asked aloud why can't a company known for their search engine make all their search systems equally good.

It's 2022, and Google search is just as shitty as their GDrive search.

“There is nothing noble in being superior to your fellow [search engine]; true nobility is being superior to your former self.”

- Ernest Hemingway, paraphrased

I think that's a good point. Google knows nobody is reasonably able to do better. So they are free to optimize for ad revenue, and leave organic search "just good enough" to stave off anyone else.

In fact, great organic search could lower ad revenue. I imagine there's some balance there.

Nope. Living in the real world with experience of how good Google used to be.

The high number of posts on the subject suggests real decline, and stopping that decline won't be helped by a rose-tinted assessment of the current situation.

> Its results are significantly better than any competitor, check for yourself.

Maybe I don't have very high standards, or maybe it is the way I write search queries, but I almost only use DDG and the results are good enough for me. I just find what I am looking for. There usually is just no reason for me to use something else, so why would I want to put my privacy into the lion's cage?

The few times I landed on Google's result page during the last couple of years, I found the results more confusing than helpful. Maybe I unlearned how to use Google.

> Despite all the posts about the declining quality of Google

We assume people in general are dissatisfied with Google, because there are blog posts about it, and some people on HN agree, and because we have certain strongly held beliefs and technical knowledge that lead us to this conclusion. What we have not validated is whether people outside the tech community are as dissatisfied as we are. If they aren't willing to switch, then there won't be a notable competitor to Google.

No, I don't buy that at all. There are way more factors that play in to this. To me, the biggest one is humans just are lazy. In face of factual evidence, people continue to ignore it. This plays out in many different facets.

We know that Big Tobacco maniuplates their formulas to make things unnaturally addicting and are incredibly bad for people's health. People continue to smoke.

We know that processed foods are just not healthy for people, yet because of it's readily being available and cheap prices, people continue to eat it.

We know that the Earth is not flat, but people continue to believe that it is.

Just having the knowledge that [Google|Facebook|TikTok|$socialMediaPlatform] is manipulating what you see/read/consume in ways that benefit them and at times is actively harmful to the users doesn't mean people will stop.

I'm dissatisfied with google for my normal use cases too... But that is more with the state of the Internet in general. But on other hand I hate fuzzy search, trying to help me... Those keywords are there for reason and might even be spelled right...

I don't know anyone outside of HN who talks about bad google search results. The ones I know using other search engines are doing it for privacy or anti-monopoly reasons. Only HN seem to remember the glory days when Google always delivered exactly what you need before the continuous decline that warrants several of these 100+ comments threads a week for 10 years. For me today as in yesterday it is just a decent search engine that works as I expect

Kagi keeps showing up here, perhaps check that out.

https://kagi.com/

I can't figure out what their business model is. Do you know?

Yes they are planning to charge a monthly fee.

https://kagi.com/faq

How much will Kagi cost?

Kagi will be completely free during the beta-test period for all users. Once we officially launch, we are dedicated to providing the best possible search experience to our users at a reasonable price. We plan to offer entry level plans for as low as $10/month, unlimited plan at around $20-$30/mo as well as have bundles (to include Kagi email and other services), organization/team plans, family plans and annual payment discounts available.

We understand that for some user this may sound prohibitively expensive, but unfortunately we are not in the position to set the price point by consensus or market expectation, but by realistic cost of providing the service at a given level of quality in a way that potentially ensures sustainability and serving our customers long-term.

You can't be just a bit better than Google and you certainly can't be worse. It's very difficult to be better than Google, that's an enormous challenge unto itself. That's greater than a billion dollar problem just to get warmed up if you're talking competing with them at large scale. If you listened to HN, Google sucks and it's easy to produce a superior search engine because of how bad they are now. That's false; even if Google's quality has eroded, they are not a mediocre search engine. That notion comes from the same place wherein people proclaim they can create a serious Uber competitor in a weekend (and mysteriously these people never do anything of the sort).

You're going to need a quantum leap improvement over Google to unseat their positioning. It has to be very substantial to overcome all the various moats they have, not least of which is consumers being used to using Google, the brand awareness.

The next great search engine will emerge from a niche and conquer one segment after another from there. It won't be a massive general search engine that shows up one day (which is what the Google watchers have been waiting for forever - that new behemoth comprehensive competitor is never going to arrive fully formed). There's a decent possibility consumer Web search will be a later stage addition to said new niche competitor, consumer Web won't be its primary or initial target. They'll add on general consumer Web search as a "we might as well" offering once they conquer enough niches.

> The next great search engine will emerge from a niche and conquer one segment after another from there. It won't be a massive general search engine that shows up one day (which is what the Google watchers have been waiting for forever - that new behemoth comprehensive competitor is never going to arrive fully formed). There's a decent possibility consumer Web search will be a later stage addition to said new niche competitor, consumer Web won't be its primary or initial target. They'll add on general consumer Web search as a "we might as well" offering once they conquer enough niches.

I love this because I want it to be true. However, iirc Google wanted to sell itself to Yahoo! for a million dollars and about five years later again for a billion dollars.

I know there are quite a few millionaires here but for me, five million dollars would change my life. I can’t imagine being able to turn it down.

> The next great search engine will emerge from a niche and conquer one segment after another from there.

Even if by some lucky circumstance Google doesn't become aware of the new kid conquering their beachhead, as they start down this expansion path Google will acquire them or throw resources at competing. A deliberate strategy of only going after segments that Google avoids (like social) almost inevitably attracts the attention of some other MANGAM.

Still, given enough dice rolls someone will eventually have a streak, but we'll be waiting a while for that to happen.

I think there may still be a quality issue. Every then and again I change my search engine to Bing. Bing works perfectly fine for most of my searches. However, every few hours there’s one search that reminds me “you’re not on Google” because I don’t get the results I expect. I tried DDG as well with a similar outcome, although it doesn’t perform as well as Bing for me.

I realize I am a sample size of one, but I run this exercise about twice a year, and sadly always go back to Google.

There are some. Neeva is one I've heard of, and there was another similar one mentioned on HN just recently that I've forgotten. There was also Cuil. Bing exists of course. And then there are the language specific ones, Yandex, Naver, Baidu, and others I'm sure. And that's just the ones that make their own index, there's a sea of competitors like DuckDuckGo and StartPage that mostly license results from someone else (including Google). And there are also verticals with competitors like Amazon for products or Yelp for food, which Google very much considers as competition.

As much as people claim otherwise, it's not that Google has no competition. People are trying! Google's product is legitimately good, despite the widespread perception of declining quality.

Kagi is also good, in fact I get better results on Kagi than on DDG.

Add Brave Search to your list: https://search.brave.com/

Private, anonymous and independent search index. Generally, search has a lot of attention on it right now, lots of great efforts to compete with G.

you.com has a lot of useful apps for developers.

I will argue that he fact that Google is a monopoly is the only thing holding back competitors.

https://www.justice.gov/opa/pr/justice-department-sues-monop...

The only thing that would have a chance is if Apple developed a search engine and used it as the default instead of using Google. People are busy/ignorant and do not care and will eat whatever search results they are given.

But this might be a risk to Apple based on how many people use Gmail on their Apple phones.

It is not about quality anymore, it is only about market domination. The only thing that will bring about new search engines will be anti-trust legislation.

How does anti trust action solve this? Forcing to divest cloud and Gmail will not make competition have an easier job with their search engines.

Google search itself is irreducible, at least in any productive way.

I don’t know, that’s for smarter people than me to figure out. But they are a monopoly so anti-trust action is called for.

If you're not sure how it can solve the problem, why does it even cross your mind to recommend it in the first place?

I do not know how to cure cancer, but I am sure it is not good for me. So I go to someone who might know more about how to get rid of it.

You can know there is a problem without knowing how to fix it. That should not stop you from talking about the problem.

I never see this one mentioned: Google crawls the web at a high rate and is able to index new content rapidly. This is possible because every site lets googlebot through. But if you are not already an established search engine everyone will throttle your crawler, and you will have no chance to compete with Google on keeping up with the latest content.

Google's search results are bad by default because they're hijacked by SEO farms. Despite this, I find that there's almost always a way for me to find what I want with Google. If I want thoughtful discussion about a topic, I'll append "from:news.ycombinator" to my search. This also often surfaces great personal blogs or old documents or other things that don't rank high in search results.

If it's not covered on HN, I'll append "from:reddit" instead (they're not always wise but as at least they're not getting paid per word written, and the upvote system gives me a sense of what real people think). If I want a broad survey of a topic, Wikipedia or some other wiki are usually great. If I want a really deep dive, I'll append "book", usually in conjunction with "from:news.ycombinator". Lastly, I sometimes add a time filter, e.g. "after:2021".

Clearly this is an awkward pattern. But it really works, and I'm having trouble thinking of something that would be meaningfully better. I suspect most people have found ways of using Google that work well enough for them. So I'm afraid Google Search will stick around for awhile.

So I agree that the quality of Google search is declining, but long story short the others are still worse (IMHO obviously).

I tried DDG for a couple of weeks and it was just awful. For almost all non-trivial searches i had to result to !g. I finally switched back after two weeks or so and suddenly I'm much happier with the quality of Google search results.

Obviously this is just one man's opinion.

I think the technical challenges are big enough to make this a really difficult task.

Because Google is the best you're going to get with ad-supported search.

If you're willing to pay for search there is an alternative emerging: Kagi. It's not a Google competitor, though. It's a niche product for people (like me) who are willing to pay a significant amount of money for access to a search engine whose creators make money by providing value to their users rather than by providing their users to advertisers.

Kagi’s pricing strategy is ridiculous. A single search can cost 5 cents in charges if I click through a few filters.

IMO Kagi can only appeal to people who care about search enough to pay and don’t search enough to make it too expensive, which is probably a contradiction.

From https://kagi.com/faq, emphasis mine.

> We plan to offer entry level plans for as low as $10/month, *unlimited plan at around $20-$30/mo" as well as have bundles (to include Kagi email and other services), organization/team plans, family plans and annual payment discounts available.

I don't know where you're getting 5c per search from, but that unlimited plan would cap it at something I could live with and I doubt I'll need it.

> I don't know where you're getting 5c per search from, but that unlimited plan would cap it at something I could live with and I doubt I'll need it.

I said a single search 'can cost' 5 cents, but in reality it depends on the search and how many filters you apply.

You can view the cost of your searches on your account. Each 'user search' can be multiple 'charged searches', which is why the cost can be >5c for some searches (e.g. if you search Dog, then click images and size -> large this is 3 searches charged to your account not one, and if you click through various options to refine your image search each click is chargable for instance).

If you click on the top result, find out it's not what you wanted so hit back, then click on the second result, this is charged as two searches if your browser prompts a reload - which depends on the page you visit (even though I would class it as a single search).

This is not a single search, in your example the user just made three distinctive searches.

If you have a better business model, that does not involve building yet another ad-supported search engine, we are all ears!

How do you propose I search for a large image of a cat within your UI that does not take 3 distinct searches then?

I don't think most users would think this was 3 distinct searches, if it's just going through the steps you are required to take to get the single search you want.

There are two steps needed instead of three as you can search images directly. Searching for a large image (or any other search where filters and modifiers are used) are probably less than 0.1% of all searches so this use case is not worth optimizing for yet as it would not have a meaningful impact on overall average monthly cost.

I think there is a difference between what you (as the creator) seem to see as a search vs what a user would understand a search to be.

No Google alternative will be able to raise funding because top VCs own Google stock and they have no interest in seeing their Google shares lose value.

Secondly, the media would not allow it because many top media executives own Google shares. If a better Google competitor came along, media companies would do everything to suppress information about it so that nobody would know it exists.

It's kind of a facile answer perhaps, but I think the answer is as simple as 'you cannot out-Google Google'.

The declining quality of Google results seems pretty clear, but only in relative terms. That is, Google now compared to Google in the past. In an absolute sense the service remains pretty great.

A confounding factor is that maybe the quality decline is in fact in internet content in general. That is, Google per se is as good as ever, it's just it's become much more difficult to find good results.

So, simply, the answer is perhaps that no matter the effort a competitor could put in they simply cannot outdo Google at their core competency. Perhaps for fundamental reasons, but even if not in some niche, surely then for sheer funding and scale reasons when trying to expand beyond that niche.

If a competitor to Google emerges, it'll be in some non obvious thing that is not core web search. Basically, when core web search becomes less important and Google can't pivot quickly enough to the new more important thing to compete.

Because google's search engine is still better than all of its current competitors and the barriers to entry are extremely high.

It used to be, at least. As years pass, Google's search quality deteriorates. It's been quite dramatic in the last few years especially - I won't pretend to know why.

A lot of it started happening when they took away the ability to search for exact terms, etc.

>when they took away the ability to search for exact terms

You can still do this. Put quotes around the terms.

This doesn't work like it used to.

It does work. In the past when I mentioned it, someone provided an apparent counterexample, a search with a term in quotes that wasn't found on the page. When I reported it to the search team, they found that the quoted term was actually in a hidden submenu on the site. So the term was still on the page, but not findable with ctrl-f, except in the source. Try it out and if you find a counterexample, let us all know.

Okay, I had this issue today and looked it up. Sure enough the term I had to quote and add a plus sign infront of was in the linked page after clicking through.

But that's it, it was just "there" near the bottom. They put in all this fancy AI understanding and ML and NLP effort but when I go out of my way to tell the search that "this word right here is super important and critical", they just go ahead and append "page.includes(word)" as a filter to their super algorithm. Instead of using that signal to drive the search. No better than when sites used to stuff their html pages with keywords to trick search engines.

#grumpy

The problem would be that you probably wouldn't want to exclude the page from the results in case people expect to find it. Did the page get ranked above other more relevant pages that had the quoted term more prominently? If it did, you're right, that should probably be a stronger signal.

It sometimes works. Maybe more so than when signed out than signed in? I spent a huge amount of time at one point trying to search for a "tost ring" when signed in (as an account that has a history of searching for developer-related content, which this is not), and no combination of quotes, verbatim mode, etc could prevent it from being "corrected" to "toString". Eventually ended up going to Bing, and it was the first hit. This was a while ago, though, and today I hit tost rings immediately. No idea if this is a change in my profile or a change in the algorithm, of course.

All the posts here suggesting that the thing that is needed is some technical advantage, or a business model, or a niche, are all true, but miss the point; a competitor that actually beat Google due to one or more of those "ingredients" would simply be acquired by Google, or crushed in the marketplace with due application (mostly fairly, prima facie) of Google's warchest (which may be inherently unfair).

Managing to grow enough in an unappreciated search niche while staying under the radar in order to better withstand that kind of eventual attention seems unlikely. Targeting a segment that Google has been burned on (like social search) may help, but that probably just draws attention from a different MANGAM.

Eventually, someone will be both smart and lucky enough to carve out some of the search space, but don't hold your breath, it is going to take so much luck it won't be soon, and may look accidental.

I had worked out some costs. If you want to build an index-based search engine that is at least competitive on coverage, if not relevance, it would cost you about $2B/yr for at least a period of 4 years just to boot things up. So, that is at least $8B of upfront investment. This is similar to the problem of if someone wanted to match TSMC on 5nm feb. This cannot be achieved by traditional VC funding and startup model.

Remember, search is not just web search. It includes vast array of things such as maps, images, videos, knowledge datasets, discussion groups, real-time news and so on. Each of these segments is an enormous effort on its own requiring massive capital investments. Relevance algos these days are mostly driven by thousands of tweaks, ML models and legions of rules. It’s a complex beast that takes thousands of PhDs and years to perfect. Even than performance is pretty subpar than most expectations. So, what chance do you have as a new entrant in same game? The key is that you don’t want to play same game if you want to win.

A lot of companies have came and gone announcing themselves as Google competitor and trying to play same game. Things like DDG survive only because they can lower the COGS by offloading real work to other people like Bing which itself is fine example of what you might be able achieve if you only had a tiny fraction of Google’s budgets. If you consider dollar for dollar capital investment, Bing is actually quite good.

This is not to say there will never be a viable competitor for Google search. I just don’t think it can be through the traditional framework of index serving. No one really knows what other alternatives can arise in future. One very possible thing is language models. If we can figure out how to scale up serving of massive language models that effectively “memorizes” whole index in them, they can provide quality, capabilities and experience that cannot be matched by simple index serving. If this is viable route, I think we are still at least 3-5 years away.

What you need as a competitor is awareness by users. But it's hard to beat a dominant player with lots of cash.

Google owns one of the major mobile platforms, where it is the default search engine. It pays competing Browsers like Firefox to be the default and is so tied in public perception that "to google" is a verb.

Also Google is fast and mostly reliable.

Also Google search integrates with other services like Maps for localized search so that one is torn back to them easily.

And if you were to get close to it they have tons of money to fight you.

The interesting aspect is that outside the broader search domain their approach often enough doesn't work. They didn't get social networking, they didn't get messaging (except mail) ... so they way to beat them likely is to find new segments and occupy that space (like Zuckerberg wants to do with "Metaverse," whatever that shall be)

The thing that replaces Google won't look like Google at first. This makes it hard to predict who will dethrone them. In the past we use to say how do you beat Microsoft? Or why isn't there a Microsoft competitor (besides Apple etc...). Now we have Google and chromebooks and android. That's a big step beyond a search engine and competitive w/ MSFT. Amazon basically owns cloud computing. Arguably there are competitors in different markets. These challenges to MSFT happened due to the massive force of the Internet which basically reduced consumer costs by transferring it to advertisers.

If we are talking about search then it will likely be something AI based probably operating under a different model than Google search. AI is a huge force probably akin to the Internet.

The problem with monopolies is they can easily purchase serious competitors. That is why we need the government to regulate these companies and spur competition.

Even if the competitor tries to hold their ground and refuse to be bought, the monopoply can still bend the competitor to their will in other means.

When google got started, they indexed ~25 million web pages. Then they grew together with the web, compounding over ~30 years.

Today, it’s totally possible for a single person or a small team to build a domain specific search engine that indexes 10s or even 100s of million documents.

Building the index v0.1 that kind of working is not hard. But maintaining the index to handle countless edge cases is tedious and need non-stop investment (eg, infrastructure + paying salary for talents)

Also, a search engine is more than “keyword matching”. You need to do search result ranking. And it gets exponentially difficult to do as # of indexed documents increases.

Empirical question: Can any kind of cluster analysis result in identifying robust selected subject-domains/content-areas for search specialization, still allowing for good coverage with a set of these? This could fight what you identify as an exponentially difficult problem. It could also allow for selective marketing and mind-share.

TikTok algorithm applied to search. “I’m feeling lucky” but with quick swipe up/down user feedback on results.

So Stumbleupon

I think this is a neat idea. Google has tried a lot of products and models, but "gamifying" (at least in this sense) search itself isn't one I can recall.

Google Discover is the closest they have to this.

As a side note, I'm a publisher and Google Discover traffic alone has made me about $1k in the last week alone to just one single article by itself

Thanks, now I know this 'personalized feed' seen in the Google apps on mobile actually has a product name!

https://developers.google.com/search/docs/advanced/mobile/go...

Some of the content on there is, however, fairly obvious clickbait, but there's also a super easy (working!) 'not interested' button in there.

Google Search enjoys multiple economies of scale. To list several:

1. Data about what the population is interested: More people use Google today, giving Google more data to train its AI with.

2. Server capacity: It's cheaper for Google to store 1 image than for me to store 1 image, because it stores many more images. Same for compute, such as training AI. Google can get one A100 GPU cheaper than I can.

3. Ads: If I had the exact same search quality and even the exact same number of users, Google will make more money from the same operation, because it has more advertisers.

They set up their business well!

The principal step will be to move on from keywords and indexing, which are now a legacy technology, almost 26 years after Google started it all.

Returning blue links is a thing of the past, as the Web of yesterday is long gone. Blue links always were about surfing i.e. following hyperlinks just for the sake of it since the main premise was most of them were of high quality and quickly proliferating.

All that is gone now and the links are a promotional thing how to get paid in one way or another. This is why Google results have been deteriorating, regardless of tens of trillions of archived pages on the Web. Google has lost the principal ranking signal years ago.

The next huge scale smart information system will be based on dense vectors (a few hundred dimensions) such as in AI but the key will be much bigger scale, of (tens of) billions of vectors. Contemporary AI works won datasets 4-5 orders of magnitude smaller, getting bogged down in gigantic transformer models such as GPT-3 with 175B+ parameters that take weeks and millions of dollars just to train. One might wonder what is innate knowledge of such a huge model, and it is not much as one can see for themselves as GPT-3 is now open (until Apr 1).

The future will be based on embeddings that are NOT contextualized i.e. no separate vectors for different senses in superpositions. Such systems will not be based on ads nor tracking as the resources required will be orders of magnitude less than what is currently required at Google.

Google built a moat around its essential product: searching on the Internet.

Speed of access, an Internet browsing engine , contracts with non-GGL browsers, cloud data centers, maps/locations/navigation, fiber cables/internet pipelines, social networks and personal management software, AI/ML research, personal devices OS ...

Impossible is nothing, but any incumbent will likely take in capital linked to Google. GGL is dominant across the world's hundred and something countries. Excpect places where it is banned(China) or shunned(Russia), anywhere it is equivalent to a monopoly of internet services.

From a technical perspective, a challenger can rise up. From an economic standpoint, a challenger would not last long (maybe locally if favored by regulation). Even the most successfull others: Apple or Amazon would not dare take on Google's Ads or Search Engine. Microsoft tried with Windows Phones and Bing, then ended failing to dominate the market. Google is not singular business. It has so many tentacles in the tech sector that wave a web of interlocking tech products. Growth numbers in Search are boosted by integration in all of Google's own products and the rest of internet users and companies.

To answer your question is to figure out what advantage it has on any competitor starting from zero... Sadly, its network scale is too large to allow an equal challenger.

A viable business model. DDG has done a spectacular job of navigating the search space with arguably a very good product and look at how long it has taken them to get as far as they have. There isn't likely room for a 2nd DDG-type approach.

Setting aside all of the specifics related to building and running the engine itself, how do you make money doing it? Unless you are servicing a very specialized and lucrative vertical that doesn't mind paying, everyone else gets their search results for free so you generally can't charge users for it. This typically leaves advertising (which Google arguably owns the market for re: search) and/or selling user data (which is both unpopular with users and depending on jurisdiction, illegal). Figure out a new way to monetize search and you may have something...

This is a problem with today's world dominated by mega-tech companies: to compete in many areas of it you need to essentially be a monopolist in some other area so you can afford to compete in something like search as a loss leader at least until you get established. Look at how much Microsoft spent on Bing[1]... and they got how much of the market? Facebook wasn't initially profitable either, but they had a distinctly different approach to aggregating a pool of data Google couldn't search and users they couldn't monetize.

[1] https://money.cnn.com/2011/09/20/technology/microsoft_bing/i...

I've been using Kagi as my default search engine on my phone to test it out. Honestly, it just doesn't hit the convenience mark. when I search for a restaurant + hours, Google shows the hours in a card at the top of the results. Kagi can't or doesn't do that, and often doesn't even find the restaurant. I haven't used DDG so I'm not sure if they're similar, but for the average user Google just consistently hits the mark (of convenience)

Thats because many companies directly type in their data on Google My Business to make their business easier to find. There is also a community doing this (same with facebook). Smaller competitors could do the same with a community or at least manually typing in the data at least for bigger companies.

I also use Kagi and I think it's good enough for most things. But you're right, for local search results Google still is better.

This is because we are still in beta and simply do not have local search developed yet. Work in progress. Glad you are comparing a year old startup to a 20 year old, $2T company - keep us honest!

The biggest barriers to competing with Google, IMO:

  * Google (and all of FAANG) have a stupid high percentage of the worker pool.
  * Google's biggest asset is that they are Google.  Kind of like competing with Coke.  Intrinsic quality of what they do is kind of irrelevant.
  * General lack of value in most things internet.  Google is getting a lot of value out of what they do, but they are having to put stupid levels of manpower into squeezing out that value.

The internet has become from being a source of information to a marketplace everyone wants to make a quick buck instead. The times where you can just search for information without being tracked what and when you search, is over. I never received spam or when, i had known it came from the shady website i signed up with. Today, my whereabouts, my age and sex are exploited as information for useless and silly spambots. They are not even trying to entertain me. And all that because using a mobile phone, goole knows where you are and where you are working. That estimates the income, making more stupid wine offers possibel, im non-alcoholic, that info they didnt get because i called help hotlines and those numbers are not listed and info is not legal to share. But well. If one wants to build a free search engine and knowledge base, has to fight with users not knowing that the world is round and alternatives available. It starts in school, kids end up in front of Windows Computers, Word is a text processor and google-ing stuff is the way to go. It goes to that extent that recommending alternatives, people refuse it, even if its better because they dont know it. Humans are sometimes stuck in their behavior.

User privacy should be #1 and some search engines do provide it. https://en.wikipedia.org/wiki/List_of_search_engines#Privacy... Another not listed above is StartPage which may use Google https://www.startpage.com/

Brave Search [1] is the best alternative I’ve come across so far. They have this concept called “Goggles” [2] on their roadmap, which I think has incredible potential to disrupt the status quo.

[1] https://search.brave.com

[2] https://brave.com/static-assets/files/goggles.pdf

Brave Search is really impressive indeed.

There is also Kagi

https://kagi.com

Kagi is just showing google and bing results mixed together.

Also there is no actual path for them to be profitable (their pricing strategy isn’t viable imo) and they are currently loosing money on every search.

> Kagi is just showing google and bing results mixed together.

It's not _just_ that: https://kagi.com/faq#Where-are-your-results-coming-from

Anyways, Kagi results vastly outperform Google results in quality for me. I'd happily pay the rather steep price they are currently targeting, should that end up being how expensive it is.

Kagi founder here - this needs to be corrected.

We have a path to profitability. Indeed, the entire business model is engineered to be sustainable. Whether the price will be accepted by the market (we do not need many users to be profitable, maybe around 50,000) is another matter. I am sorry if you do not feel like on. But one thing we will not do, is loose money on every search.

IMO you need to start charging users now to learn the lesson on pricing quickly.

You have a theoretical path to profitability, but as you haven’t yet charged a penny it’s not proven to be viable. I genuinely hope you prove me wrong, and turn on paid subscriptions and it’s a huge success. If it’s not, better to learn that lesson now while you still have a chance to change course.

Resources matter: you can craft few better services, but if you do not have Alphabet resources your better service can't compete.

For instance how can you offer a better maps for PND if you can't elicit traffic data from the density and speed of android smartphones that are around of 80% of phones traveling around the world?

To defy Alphabet you do not need a better Google, you need a different kind of solution, for instance instead of competing in modern web crap to lock down users and surveil them you can propose classic desktop computing with decentralized tools. Try looking for instance at Jami, Retroshare, ZeroNet, you can integrate them in a suite and say "hey, instead of depending on Zoom, Meet or Teams, proprietary services with limits that might change, surveil, etc use this system, there is no SPOF, no service behind".

Some have tried something, for instance DeltaChat seems to be a WA clone, but it's actually a MUA, in that case it doesn't took off much because most people simply do not care, you have to know your public. Starting from CS and humanities courses [1] to attracts students and plant the seed of something new, being prepared to face years before a success simply because no empire born quickly and when some are against you it's even more complex.

The old adage: people have the power, but they do not know how to use it it's unfortunately very true, you can only bend people, use them, to elicit a slice of power :-)

[1] most CS students haven't enough skills to comprehend the world, they just explore few aspects ignoring the rest and that's why dictatorships like STEM, because they help to generate "ancient Greek's 'useless idiots' to be employed" instead of Citizens (disclaimer: I am an engineer, I learnt than personally in years)

No one is going to beat Google at their game. But the game will change. And Google is unlikely to win the next one, if past similar examples can teach anything.

There will be a dramatic change in how people ask questions and access information.

Maybe it will be personal assistants. I doubt it.

Perhaps these kinds of Neuralink implants? Could be...

Whoever nails it, will be one of the near-future trillionaires.

I wonder how much of the declining search quality can be attributed to the rise of walled gardens and social networks. It probably was a lot easier to index blogs and personal websites back in the day vs. TikTok/Snapchat/etc.. today. It's ironic that Google itself directly contributed to this.

Because imo it would require being more than just a search engine. Google has Chrome, Gmail, YouTube and Search. People like having a lot of their services consolidated, just look at the Apple folks.

Also while the quality of search has declined it is nowhere near the point where casual users would be bothered enough to switch

And Android, that keeps users tied to Google services through the power of defaults.

One strategy might be to monetise Googles failure/difficulty/broken business model with regard to SEO, promoted, and brand sites. How might one do this? Create a market for filter add-ons to Google’s results. A market for quality filters (subject to different user groups subjective view of what quality means). For example, let’s say I am a specialist in some particular area, and I’ve devised a “quality filter” or “screen” for my specialism. A user, an ordinary user might choose to pay for my filter to cut down the firehouse of junk normally coming out of a google search. The platform owner, for this value added search market might have an App Store for filters and skim off some of the recurring subscription revenues for the 5, 10, or 50 filters the user might subscribe to.

Google's competitor in what? As far as search goes they pay billions to device manufacturers and software vendors to have Google as default search everything, how would you compete with this? The actual search functionality, its speed, intuitiveness, etc. of any competitor is irrelevant.

> how would you compete with this?

The implication of OP's question and perhaps the heart of the matter is whether Google is unfairly applying some impediment or anti-competitive behavior. My personal opinion is that they're not. I'm not a software guy so maybe I don't know the facts. I would very much like to be educated on that here, the same way I've tried to educate people here about display industry facts on the ground.

brave.search is pretty good. BUT it is not the default search engine on most devices. AND google is still better for some specific queries. A competitor must be not as good but much much more better. Don't see it coming, especially it is very expensive to query the whole web.

People don’t seem to realize how good Google is in regional results. I live in a European country and every other search engine I’ve tried in the past is lacking when searching for local content. I can search for news articles that date back two decades in Google and get results.

DuckDuckGo does quite well here for me, as long as I enable the button for localized results. I very much like that it's an option there.

Bing is becoming fairly popular in the US https://gs.statcounter.com/search-engine-market-share/deskto...

Microsoft is pushing extremely hard for it. In Windows if you miss a barely visible button you will find yourself switching to Edge and Bing without realizing it.

That Google is still at 85% show that people really want to use Google, it is not just "whatever comes with the computer".

..."you will find yourself switched* to...." Ftfy.

I believe a competitor would need to be such a better experience that by saying you are "googling" something, as in the common word for searching, an essence of the competitor is left out.

If bing was successful at it, for example, it would cause the sentence "I tried to google it but ended up needing to Bing it" to make tangible sense to someone who has used both services.

Even if you create a better search engine, there is too much of a marketing gap to compete on that alone. "use bing because the results are better!" simply doesnt motivate enough people to switch off google.

Quite frankly, it is rare that google does not deliver expected results for the generic use case.

Search is kind of an investment dead zone. Even if you've come up with a completely paradigm-breaking method, getting money to develop the idea at scale would be tough since Google (and others) have so many patents that litigation risk would drive off funders.

If you have a great idea and access to deep enough pockets to develop it, there's still the problem that much of the information people want to find are behind various walled gardens and not suitable for public search. Incrementally more helpful arrangement of the same publicly available data that Google already has, isn't worth the effort of challenging the behemoth

Lots of competitors have emerged or are emerging. No single one can or maybe even should take over the mantle.

Google’s death will come not from some upstart search engine, but from many walled gardens that each have perfectly good topical search inside them. We’re already in a case where different kinds of information are checked at different destinations.

Google will keep trying valiantly to more aggressively turn into a knowledge base instead of a search engine, and ultimately will probably settle at being an interactive encyclopaedia.

The internet has moved past search engines. Now if you want information you need to know or figure out where to ask the question.

Google has a TWENTY year head-start of building server farms all over the world and hiring PhDs to develop the complex algorithms and software to do the indexing.

Can't just buy time on someone else's cloud network and hire a couple of coders to compete with that.

You could throw a lot of money at it but not compete on years of investment.

My brain is swiss-cheese these days, I wish I could remember the name of that one-man startup a decade ago that actually had a lightweight high-speed crawler and was making a serious attempt at competition. Was covered around here several times. But obviously they didn't succeed if I can't remember the name.

You really think DDG gets 2% as much traffic as Google? There's no chance

> what would a competitor need to defy Google?

A competitive business model. You can create tech that is easier to use and more powerful than Google's services, but you won't last unless you actually make money.

google competitors already exist they're just not obvious - When you want to find a place to stay you dont search google you go to airbnb.com when you want to buy random stuff you go to amazon, even coming to hackernews is a competitor. You are seeking something and find it. I do get your point tho about a direct competitor, at the end of the day - all the innovation around search is happening in that we are unbundling the search engine and turning each category/search result asset into a monetizable marketplace. My .02

What do you think of a p2p marketplace for search queries? You post your query to the p2p network that consists of indexers. Indexers interested in servicing your query would place a bid (i.e. the charge), from which you select one. The selected indexer would then service your query.

I think this would allow a competitive market for indexers in different verticals to emerge.

Most probably you wouldn't want to use it for every small query. But in situations when you need high quality results in some domain, the p2p marketplace would be a good choice.

Bing and Yahoo (who is pulling from Bing) are supposedly competitors. The problem is that Google is kicking their butt. Even if Bing became more of a true challenger, Microsoft is doing much of the same, so just replacing one name with another. Most would agree that Microsoft doesn't have any more respect for user privacy and would trample (ban, shadow ban, violate, spy, sell date of) users just as much.

Probably the only hope is some type of peer to peer search engine, that actually catches on and works well.

First you need to ask, why everywo is using Google:

Habits and ignorance. The casual web user is just fine with what Google has to offer.

And besides that: Google is convenient and everywhere. To get rid of it it requires not only the will but also the technological skill that the common user just does not have.

And that leads to the answer of your question: Create an alternative that is easy to use,ofcourse, that can be implemented by just "a click" and is has to be everywhere. On your phone, TV and computer.

> Habits and ignorance. The casual web user is just fine with what Google has to offer.

I think that’s a contradiction. If it’s just fine for their needs, the they have all the relevant information they need. They know perfectly well finding another service would take considerable effort, might well lead to a worse experience, and there’s a high chance they’d switch back anyway. That’s an informed decision, not an ignorant one.

So why did everyone start using Google when before that everyone had been using Yahoo or AltaVista?

Because Google at the time was a dramatically better experience than AltaVista, and Yahoo was more of a directory than a search engine.

A better question might be, what do you want that Google doesn’t already give you?

Personally I wouldn’t mind a search engine that was focused only on code results. Not only results that that showed code, but code focused results - tutorials, samples, books.

For example, if I search for “c web server” in a private tab on on Google… well… I get exactly what I want. If I search for “water shader”… I mostly get what I want as well (youtube videos not code though).

Hum, I think you might have a hard time with this.

...because antitrust authorities have been asleep at the wheel.

I think the way you'd compete is unbundling it.

Building a general search engine better than Google is probably a non starter, but...

Could you build a better image search for graphic designers? Could you build a better search for specific document types, such as academic papers or legal precedence?

I think that's what we'll see - similar to how nothing replaced Craigslist, but a bunch of startups unbundled features in Craigslist and made companies around them.

I don't feel the need for another Google. Individual search engines of the major sites are proving decent enough. The search engine within StackOverflow, Reddit, Twitter have been good enough for me. For arcane stuff, DDG is ok. For images, I use Bing. I sort of know which site can answer which query of mine, hence I rarely use GOOG these days, unless it is some mass-market, in-the-news kind of thingy.

What would a competitor need to defy Google? – A billion users who have not already chosen a search engine.

I slightly better experience should get a larger portion of new users, but existing users do not say "Today I will develop a methodology to conduct a survey of internet search engines and select the best one for me."

At this point everyone has either chosen a search engine or had one chosen for them.

You.com is one new entrant to the search engine space. It's ad-free, private, and customizable with preferred sources.

I've been a happy kagi.com user a couple of months now. It's like lo-fi Google, less results, no ads anywhere and similar to duck duck go bang commands if you want to use other search engines to check different results.

Google’s advantage over competitors isn’t that the search is great it is that they are much better at montesization.

I vaguely recall part of an interview with Warren Buffett, where he was talking about tech investments. At the time he was interested in cloud providers and not interested in search. He said that "search is winner-takes-all," while infrastructure-as-a-service is not. Maybe that's part of it.

Actually, I don't think that search is winner-takes-all. Rather the piece that is winner-takes-all is monetization of search through advertising. Because advertising networks are winner-takes-all. ...so the path forward leads through monetizing search in a way other than through advertising. And this is precisely what the next generation of challengers like Neeva and Kagi are doing. Another alternative might be public funding.

> Apart from Google's monopoly and the big technical challenges, what would a competitor need to defy Google?

Which monopoly? Email? Search?

Regardless, the answer is time. It took Google a decade to win the search war completely. It will take a competitor time to dethrone Google.

Because I believe a search engine for the internet should be nationalized, like the internet itself.

Google "solved" the internet search engine problem. Other competitors are just replicating it poorly. There's nothing good for the consumer here.

Chrome is currently a major component to Google's hold over its users' web experience. When it is measured in terms of market share, it appears to me that the browser wars and search engine companies fight in the same battles.

This article has insights from folks who actually tried:

https://0x65.dev/blog/2019-12-06/building-a-search-engine-fr...

TL;DR you need a lot of money and users (catch-22). Also, massive scale and massive noise.

You can also search HN for 'cliqz' for discussions about Cliqz shutting down.

Cliqz' blog is really insightful.

In case you weren't aware, Brave Search is led and built by key team members from Cliqz: https://search.brave.com/

The growth and usability of Brave Search this past year has been strong, with some significant features planned throughout 2022.

A possible alternative was discussed on HN half a year ago:https://news.ycombinator.com/item?id=28550764

Check it out, it's great!

Here's a thought, if Reddit fixes their terrible search engine, they would steal a large chunk of my search.

I've been attaching url:reddit.com to my Google searches to bypass all the blog spam thats taking over Google.

The realist in me knows google would likely not be allowed to just buy reddit for anti-trust reasons.

I guess the, pragmatist(?) in me thinks, "google kind of legitimately provides a solution to a problem that reddit has had like, forever. And reddit provides an organic source of knowledge that I feel google is at least trying to get back in the mix. Why should that combination not be allowed to exist?

You are comparing google with other search engines but they are instead an ads monopoly. If someone else invests big on an ads marketplace, a search engine will spontaneously form around it

How many Larry Page competitors are there? How many Sergey Brins, to listen to an idea without any validation, and judging it with independence? Why do they have to do everything?

Are people funding founders who want to make new search engines?

Have you all used presearch?

Presearch decentralized search engine, powered by blockchain technology.

https://presearch.org/

The network effect, the Lindy effect, and the Schelling point. If you wanted to learn about these concepts, what is the first thing you would do? :P

A super tight mobile os. Search, Maps. Excellent developer tools. Long term vision. This 10+ year android user is disappointed recently.

Have you tried Brave Search? It’s not 100% yet but it’s better for many searches. I made it my default and only use Google as secondary now.

Because nobody is writing good benchmarks for search engines. You can't improve something if you can't see what you're doing.

I find it interesting that you mention the word benchmark because reviewers used to rely on benchmarks for new phones and such but then iirc at least OnePlus and Samsung devices have been caught and banned from multiple benchmarks.

Banning them is silly. The benchmark programs reported unthrottled speeds, but when the phone has per app throttling (to save battery) you may not get the speed you measured in each app.

I seem to recall that some manufacturer's have been found to programmatically alter behaviour of hardware & background programs/services when a benchmark is running, creating results that are not achievable otherwise in other programs.

You can use benchmarks to sell a product, but you can also use them to improve your product. I'm thinking of the latter.

The absolute no 1 prerequisite for building a good search engine is having good benchmarks.

well... i've been "binging" and "DDGing" things a lot more often in the last couple of years.

Bing isn't the perfect google replacement as it seems to have gone the google route of not returning hits for things you search for, but for things it thinks you wanted to search for.

Maybe the "site:reddit.com" hack works well enough for people who care about search results.

Willpower and technical ability. It's a hard problem. Look at how many resources Microsoft has thrown at it with limited success.

Some platforms are complicated enough to require at least $100m to compete. search engines , OS , autonomous vehicles , voice assistants

Did you notice Google's monopoly over online ads? Competitors can't make money. It's literally called anti-competitive.

I think that has more to do with the information source than the market.

It's possible HN is an unreliable guide to Google's actual prospects.

For search I think a big issue is you can’t crawl the web like you used to. Cloud flare and others have made that difficult.

Who would fund you if you said you wanted to be a competitor?

A competitor would need to avoid being bought by one of the tech oligarchs. That's pretty hard to avoid.

If by some fluke, you find a breakout opportunity, you would need to somehow avoid getting crushed like Parler. The tech oligarchs will use any excuse to crush any competition.

That episode basically shows that to compete you would basically need to build your own internet first, and also your own banking system.

Why isn't there an Amazon or Microsoft or Boeing competitor emerging?

These are systematically entrenched monopolies of sorts.

> what would a competitor need to defy Google

Google is hiding important information on purpose... so don't do that.

Duck duck go, and bing are two alternatives off the top of my head. You just need to look for them.

2 cents,

With so many bots, a lot of sites run a whitelist process of user agents, so every new entrant is automatically blocked in those cases, and for the rest, any large scale crawling is quite likely to get blocked.

Anecdotally it seems a relatively new search engine entrant is using Chrome user agents and follows Googlebot robots.txt directives so are impossible to block. If true, that's an absurd way for a new player to enter the market and just annoys webmasters.

A recent anti-competitive report by the UK government estimated that it would take about $20bn dollars for a new entrant to compete on Google's level.

Then there's how Google pay Mozilla/Apple very large sums of money based on their ROI/search which no one else can compete with, partly due to the invasive surveillance capitalism that Google follows. That ultimately means that anyone who wants to compete has to pay browsers and operating systems to enter the market in a substantial way.

All that said, encouraging people to find alternative sources of information is most likely the best route to diversification with or without government(s) intervention.

One can be doubtful that googles downfall will come from another search engine.

because they Peter Thiel'd 10x'ed search and ads and competing against them would require extraordinary cost and a breakthrough in either.

I've been using Neeva, it works well.

That's funny:

Their slogan:

>try Neeva, 100% ad-free search engine, Totally private

Their founders:

>Neeva was founded by Sridhar Ramaswamy (ex-SVP of Ads at Google) and Vivek Raghunathan (ex-VP of Monetization at YouTube). They met in the early days of search ads at Google, and came up with the idea for Neeva over hikes and coffee.

They have founded a company that explicitly doesn't need the knowledge that they had acquired in their job.

[1] https://neeva.com/about

Yep! It is ironic but also makes sense in my opinion. I know more about why the things I'm an expert in are bad than most people do. That doesn't necessarily mean I know how to fix the issues though, but at least I have a good sense of the problems, which is a start.

DDG simply needs to raise awareness that they have a good enough alternative that most people would be happy with if only they knew it existed.

DDG is just a Bing front-end/proxy. They do not have their own crawlers.

If microsoft quit search for some reason, DDG would go away with them.

A business model.

Bing.

I have a broader answer for this:

By the very nature of solutions: a new company cannot beat google at general search. Google was and is search.

Like a tree that has grown up and shadows the entire land around it. There will be random small pockets of sunlight with smaller plants in it. But the big tree owns the area.

There can be a short term competition via extremely specialized search engines, but they will not rise to the same dominance. Google, in a way, is the entire idea of searching the web.

The next dominant generation has to be from a new paradigm, that makes web search obslete.

---

A relatable current example could be: youtube being dwarfed by tiktok. The nature of the new thing is such that it very naturally dethrones the old thing, without directly competing with the incumbent.

There will be specialized providers like vimeo, But at this point, youtube IS the idea of video on the internet.

---

Just note the context of this understanding: This understanding came from struggling to change the education system of my country. And then it seemed to apply everywhere I saw.

I'll round-off my reply with this quote:

“You never change things by fighting the existing reality. To change something, build a new model that makes the existing model obsolete.”

– Buckminster Fuller

DuckDuckGo just ended their value proposition:

https://twitter.com/yegg/status/1501716484761997318

Not that there are many great alternatives, but the whole point of a SEARCH ENGINE is not censoring results. I understand ranking makes it searchable, but omitting views / results based on politics is self-reinforcing and makes anyone using the service blind to views and in a sense reality.

I think search can’t really be “improved” until it’s decentralized and uncensored. The quality decline IMO has to do with the social networks and “trusted news sources”, which I also think is a walled garden limiting accuracy of info.

See “trusted news initiative”: https://www.bbc.com/beyondfakenews/trusted-news-initiative/

The main reason a lot of people use DDG is privacy, not uncensored search results. I do think that this move could have been handled better.

Both go hand in hand with trust. When you compromise one you poison the other due to lost trust.

I think your mistaken, the only reason DDG took off, at least in the circles I’ve visited (open source communities, privacy focused, politics, etc) is the freedom to research anything in a safe way.

Think of it another way, “who cares about privacy?” Probably those searching for “unacceptable” ideas.

If they censor, who’s to say they wouldn’t turn you into the authorities for “wrong think”.

Their value proposition of privacy was damaged by this BECAUSE they want to limit your search results.

thank you for articulating this properly -- privacy & censorship are intricately intertwined, like yin & yang

Spreading lies is a value proposition? I disagree. I don’t think “censoring child porn” is a bad thing either, which DDG does. Just because information of some sort exists, doesn’t mean it’s inherently valuable to spread.

> but the whole point of a SEARCH ENGINE is not censoring results

Disagree. I hate it as much as the next person when Google censor results due to DMCA request (a law i couldn't care less about which shouldn't have jurisdiction over me), but you have to admit there are things to limit. Should a search engine return child porn results? (Yes, it's an easy and egregious example, but that's the point - there are such subjects).

Furthermore, censoring low quality enemy propaganda is a public service.

> Furthermore, censoring low quality enemy propaganda is a public service.

I think anyone looking to keep people from understanding is an enemy. You fight propaganda with truth, not censorship.

Did anyone seriously believe that DDG or its data sources did not filter content before?

Reply on news.ycombinator.com