Google exploited exclusive search engine deals to maintain its advantage

Not wanting to excuse Google here, but I wonder if a certain kind of moat can build by itself if a company is big and well-known enough.

E.g. many websites have restrictive robots.txt files or other anti-bot and anti-scraping measures in place - however, I'll claim that most sites will make a specific exception for Googlebot. Not because Google pays them or anything, but because it's common knowledge among web developers that search engine bots have to be let through. The problem is that there is no public registry for search engines, so developers just go by the engines they know and deem most important - and that's primarily Google.

If they feel generous, they'll add few of the other biggest players to the list as well, such as Bing or DDG, but they for sure won't add some obscure startup they never heard of.

The effect is that anyone trying to create a new search engine automatically will have a harder time creating an index of the same quality as Google's, even if they had access to the exact same hardware, algorithms and expertise as Google.

So in effect, the quasi-monopoly perpetuates itself, exactly because everyone else is acknowledging it and adjusting their own behaviour towards it.

In hindsight, the victor claiming they won because they were better than Bing et. al is exactly what I would say as well if I rigged the game.

We are so commonly assuming that Google has the secret juice that makes search work, and we are so deep in that we can’t trace our steps back and see if this statement still holds in 2023 or not!

Same goes for companies perpetuating the myth that they only hire mega-brained geniuses. That is far from the truth! Most exceptional companies are bound to regress to the mean at some point :)

As a consumer, having my freedom of choice so severely limited appears decisively anti-capitalist. Ffs Firefox please remove Google as the default.

„Competition is for losers“

Peter Thiel on WSJ https://archive.li/2014.09.12-221631/http://online.wsj.com/a...

What kind of world is this? Capitalism run rampant? Shocked.