To Break Google’s Monopoly on Search, Make Its Index Public
bloomberg.com
bloomberg.com
This proposal won't do very much. Indexing is the (relatively) easy part of building a search engine. CommonCrawl already indexes the top 3B+ pages on the web and makes it freely available on AWS. It costs about $50 to grep over it, $800 or so to run a moderately complex Hadoop job.
(For comparison, when I was at Google nearly all research & new features were done on the top 4B pages, and the remaining 150B+ pages were only consulted if no results in the top 4B turned up. Difficulty of running a MapReduce over that corpus was actually a little harder than running a Hadoop job over CommonCrawl, because there's less documentation available.)
The comments here that PageRank is Google's secret sauce also aren't really true - Google hasn't used PageRank since 2006. The ones about the search & clickthrough data being important are closer, but I suspect that if you made those public you still wouldn't have an effective Google competitor.
The real reason Google's still on top is that consumer habits are hard to change, and once people have 20 years of practice solving a problem one way, most of them are not going to switch unless the alternative isn't just better, it's way, way better. Same reason I still buy Quilted Northern toilet paper despite knowing that it supports the Koch brothers and their abhorrent political views, or drink Coca-Cola despite knowing how unhealthy it is.
If you really want to open the search-engine space to competition, you'd have to break Google up and then forbid any of the baby-Googles from using the Google brand or google.com domain name. (Needless to say, you'd also need to get rid of Chrome & Toolbar integration.) Same with all the other monopolies that plague the American business landscape. Once you get to a certain age, the majority of the business value is in the brand, and so the only way to keep the monopoly from dominating its industry again is to take away the brand and distribute the productive capacity to successor companies on relatively even footing.
Google simply has the best search product. They invest in it like crazy.
I’ve tried bing multiple times. It’s slow, it spams msn ads in your face on the homepage. Microsoft just doesn’t get the value of a clean UX.
DuckDuckGo results are pretty irrelevant the last time I tried them. There is nothing that comes close to their usability. To make the switchover, it has to be much much better than Google. Chances are that if something is, Google will buy them.
Building the index itself is relatively easy. There are some subtleties that most people don't think about (eg. dupe detection and redirects are surprisingly complicated, and CJK segmentation is a pre-req for tokenizing), but things like tokenizing, building posting lists, and finding backlinks are trivial - a competent programmer could get basic English-only implementations of all three running in a day.
- Hardware infrastructure and data center presence for extremely fast search from anywhere in the world. - Near real-time search suggestion. - personalized search results based on past search + geolocation. - Search to get instant results without having to go to a website.
Just to name a few. Google Search is the gold standard of a search engine, not because its Google or because they have been around for a long time and the brand name sticks (I am sure it helps too), but for the simple fact is no search engine is even remotely close to being as good as google and I have tried them all more of the less and given them shot. They are just not good at all.
I also don't understand the hate towards google being in charge of so many products so many people use, ie, Mail, Maps, Chrome, Android, Docs (to name a few). It's simply because they are damn good at it. If its a crime to make a product so good that people continue to use it, then I don't know what else people are supposed to do. As if we are asking google to make shit products, I just don't understand the reasoning.
The best available doesn't necessarily mean the best possible. And Google is far from it, and it's getting worse, not better.
Using a bit of Google-fu I'm usually able to find what I need quickly but it's still more of a hassle than it used to be.
First, I think search results at Google have gotten worse because people are not actually good at finding the best example of what they're looking for. People go with whatever query result exceeds some minimum threshold. This means when Google looks at what people "land on" (e.g. something like the last link of 5 they clicked from the search page, and then which they spend the most time on according to that page's Google Analytics or whatever), they aren't optimizing for what's best, they're optimizing for what is the minimum acceptable result. And so what's happening is years and years of cumulative "Well, I suppose that's good enough" culminating in a perceptible drop in search result quality overall.
Second, Google has clearly been giving greater weight to results that are more recent. You'd think this would improve the quality of the results which "survive the test of time" but again, Google isn't optimizing for "best" results, they're optimizing for "the result which sucks the least among the top 3-5 actual non-ad results people might manage to look at before they are satisfied". So this has the effect of crowding out older results which are actually better, but which don't get shown as much because newer results have temporal weight.
My warning is this, too, which you've surely noticed: Google search has created a "consciousness" of the internet, and in the 90s it used to be that digitizing something was kind of like "it'll be here forever" and for some reason people still today think putting something online gives it some kind of temporal longevity, which it absolutely does not have. I did a big research project at the end of the last decade, and I was looking for links specifically from the turn of the century. And even in 2009, they were incredibly hard to find, and suffered immensely from bitrot, with links not working, and leaning heavily on archive.org. Google has been and is amplifying this tremendously, by twiddling the knob to give more recent results a positive weight in search. Google makes a shitload of money from mass media content companies (e.g. Buzzfeed) and whatever other sources meet the minimum-acceptable-threshold for some query, versus linking to some old university or personal blog site which has no ads whatsoever. So the span of accessible knowledge has greatly shrunk over the last few years. Not only has the playing field of mass media and social media companies shrunk, but the older stuff isn't even accessible anymore. So we're being forced once more into a "television" kind of attention span, by Google, because of ads.
It's easier to return the most relevant 10 results when there's only 10 thousand options than when there's 10 trillion options with 10 thousand new ones created every day.
My guess is that it's because Google Search now also has to cater to queries from Assistant. Being required to handle web, mobile, and assistant probably necessitated tradeoffs in quality of one over another.
More generally I feel like as the company gets bigger it just gets much harder to handle all the complexity and keep things focused.
I know it's not google proper, but I'd guess a significant number of their searches are specific to youtube.
But what about when Google fails? Unlike DDG, there is no culture of switching between search engines when googling. Typically, you'll just rewrite the query for google. And as rewriting the query is an entrenched part of googling, you are less likely to notice this as a failure. It is this training that's the core advantage nostrademons points out.
I've been using DDG exclusively since I was a contractor at Google years ago and have never had a problem finding things with it...
I read somewhere that someone used a skin to make ddg look identical to Google. After doing that, they never even thought about using Google again.
Interesting I would have thought that crawling at this scale and finishing in a reasonable amount of time would still be somewhat challenging. Might you have any suggested reading for how this is done in practice?
>"It costs about $50 to grep over it, $800 or so to run a moderately complex Hadoop job." Curious what type of Hadoop job you might referring to here. Would this be building smaller more specific indexes or simply sharding a master index?
>"Google hasn't used PageRank since 2006." Wow that's a long time now. What did they replace it with? Might you have any links regarding this?
$50 gets you basically a Hadoop job that can run a regular expression over the plain text in a reasonably-efficient programing language (I tested with both Kotlin and Rust and they were in that ballpark). $800 was for a custom MapReduce I wrote that did something moderately complex - it would look at an arbitrary website, determine if it was a forum page, and then develop a strategy for extracting parsed & dated posts from the page and crawling it in the future.
A straight inverted index (where you tokenize the plaintext and store a posting list of documents for each term) would likely be more towards the $50 end of the spectrum - this is a classic information retrieval exercise that's both pretty easy to program (you can do it in a half day or so) and not very computationally intensive. It's also pretty useless for a real consumer search engine - there's a reason Google replaced all the keyword-based search engines we used in the 80s. There's also no reason you would do it today, when you have open-source products like ElasticSearch that'd do it for you and have a lot more linguistic smarts built in. (Straight ElasticSearch with no ranking tweaks is also nowhere near as good as Google.)
I can no longer find anything remotely good quality, I discover new and quality stuff from social media like Twitter and HN.
The search results seem to be too general and too mainstream. Nothing new to discover, just a shortcut to the few websites like Reddit, StackOverflow for more techie thing and Wikipedia and the few mainstream news websites for the rest.
I usually end up to search HN, Reddit or StackOverflow directly as the resulting quality is better as I can get easily specific. Getting specific is harder on Google because it just omits or misinterprets my search query keywords quite often.
It's probably possible to build a search engine for a specific vertical that's better than Google. However, you face a few really big problems that make this not worthwhile:
1) Speaking from experience, it's very difficult to define what "better" means when you don't have exemplars of what queries are likely and what the results should be. The reason search engines are a product is that they let us find things we didn't know existed before; if we don't know they exist, how can we tweak the search engine to return them?
2) People go to a search engine because it has the answers for their question, no matter what their question is. If you had a specific search engine for games, and another for celebrities, and another for flights, and another for hotels, and another for books, and another for power tools, and another for current events, and another for technical documentation, and another for punditry, and another for history, and another to settle arguments on the Internet, then pretty soon you'd need a search engine to find the appropriate search engine. We call this "Google", and as a consumer, it's really convenient if they just give us the answer directly rather than directing us to another search engine where we need to refine our query again.
3) Google makes basically 80% of their revenue from searches for commercial products or services (insurance, lawyers, therapists, SaaS, flowers, etc.) The remainder is split between AdSense, Cloud, Android, Google Play, GFiber, YouTube, DoubleClick, etc. (may be a bit higher now). Many queries don't even run ads at all - when was the last time you saw an ad on a technical programming query, or a navigational query like [facebook login]? All of these are cross-subsidized by the commercial queries, because there's a benefit to Google from it being the one place you go to look for answers. If you build a niche site just to give good answers to programming queries or celebrity searches or current events, there's no business model there.
I can't get better products by searching Google, I can get the best-spammed products or most promoted products only.
The fact that I am getting low-quality service and Google is printing money means that there is a place for good a good service and if that service cannot emerge due to Google's practices, it probably means that the regulators need to take action.
Or maybe the search is dead, long live social media.
The gist is, I am not happy with a service but the company that makes that product makes a lot of maney. Can't tell if I am an anomaly or if other people feel the same way because Google is a monopoly and maybe the regulators should make it possible to compete with Google and see if there's a space for a better service.
Yes yes, I am the product but I am the product only if I am happy with the stuff I'm getting in return.
Keep in mind when Google started, Yahoooooo! Was the big player and Google overtook them by simply being better
Funny, I don't disagree with this, but my perception has been that Google seems to detect when I've switched roles from one type of programmer to another. I don't know if that's organic from the topics I'm looking up or not, but if I'm looking up a generic string search, it seems to return whatever language I've been searching for recently. (very recently in fact)
My point is, it seems like the search engine intuitively understands my "vertical" already. Maybe it's just because developer searches are probably pretty optimized.
Google Ads (used to?) lets you target by "bahaviour" vs "in-market". They can tell the difference between someone who is passionate about beds, maybe involved in the bed business (behavior) and the people who are making the once-in-a-decade purchase of a bed (in-market).
Google can tell devices apart on the same google account and keep together search threads. I might be programming on my desktop making engineering searches but at the same time I'm googling memes on my phone; both logged into the same account.
Definitely have seen malicious adds for "facebook login", though that was probably 2016 or 2017.
Better is a search engine that takes your queries more literal. This is what everybody means when they say Google used to be better. The query keywords and no second guessing.
When you insist on Google using verbatim mode or something, you often don't get results. Which is bullshit because I remember 10 years ago, queries like these had me plowing through the results, so much that you actually had to refine the query -- you can't do that in Google any more, at least it's not refining, it's more like re-wording and re-rolling the dice. But it all feels very random and you don't get a feel for what's out there.
I mean sure there is a place for a search engine like this, if it works well. And in its own way, Google works well.
I sometimes do want my query to be loosely interpreted like I'm an idiot, and I head straight for the Google. Ever since I saw the "that guy wot gone painted them melty clocks"-meme, for certain types of queries I have indeed found that if I formulate my question like I got brain damage, I get superior results. Because that is the kind of audience Google wants you to be.
But sometimes you don't feel like the lowest common denominator and you don't want to be treated as such. And there should be a place for that, too.
I can't help but think it's partially due to people using tools _specifically designed_ to make Google's job harder (FF SandBoxes, uBlock, etc) and not understanding the implications of using them... and then blaming Google for returning "bad" results.
Like, a lot more seo spam though.
I have this problem too. Google often thinks that I made a typo and presents me results for things I didn't searched for or care about and I have no way to force it to search for things I really want.
Maybe it just feels more successful because it lowers my expectations. But at least you get to mash the keyboard like a maniac, do no corrections, press return and watch it just work.
It's kind of like watching Google do a "customer is always right" squirm.
This is not a proposal to just share the crawl data, but the actual searchable index, presumably at arms length cost both internally & externally.
The same ideas could be extended to the Knowledge Graph, etc.
IMO the goal here should not be to kill Google, but to keep Google on their toes by removing barriers to competition.
API access to either the unranked or ranked index in memory wouldn't do anything useful, BTW. To have a viable startup you need something a lot better than Google, which means that you need algorithms that do something fundamentally different from Google, which means you need to be able to touch memory yourself and not go through an API for every document you might need to examine. Remember, search touches (nearly) every indexed document on every query - if you throw in 200ms request latency for 4B documents your request will take roughly 25 years to complete.
Knowledge Graph is already public - it was an open dataset before it was bought by Google, and a snapshot of its state at the point Google closed it to further additions is still hosted by Google:
https://developers.google.com/freebase/
(It's only 22G gzipped, too - you can download that onto a personal laptop.)
Doesn't it only touch ones with at least one of the search terms in, or stemmed/varied words relating to some of the terms? And does that via an index?
Basically, yes, it uses an index and touches only documents that appear in one of the relevant posting lists. However, after stemming, spell-correcting, synonyms, and a number of other expansions I'm not at liberty to discuss, there can be a lot of query terms that it needs to look through, covering a significant portion of the index. Each one of these needs to be scored (well, sorta - there are various tricks you can use to avoid scoring some docs, which again I'm not at liberty to discuss), and it's usually beneficial to merge the scores only after they have been computed for all query terms, because you have more information about context available then.
There's a reason Google uses an in-memory index: it gives you a lot more flexibility about what information you can use to score documents at query time, which in turn lets you use more of the query as context. With an on-disk index you basically have to precompute scores for each term and can only merge them with simple arithmetic formulas.
You mean like Wand or BMW?
But, reading through the other comments, leaving out this part would make it better than Google.
Maybe stemming. I remember when Google added stemming (somewhere in the early 2000s). I was conflicted about it because I didnt want a search engine to second-guess my query (can you imagine??), but I also saw the use because I was already in the habit of trying multiple variations.
Auto spelling correct is a no-no. Just say "did you mean X?" and let people click it if they misspelled X. No sense in querying for both the "typo" and "corrected" keywords, because the "typo" would rank much lower, right?
Similar for synonyms. Either it should be an operator like ~, or maybe it should just offer a list (like the "did you mean" question) of synonyms to help the user think/select similar words to help their query.
That dump is outdated, not supported, and very incomplete comparing to what google has now.
If I type in “<name> Pentagon” on Google, the first link is LinkedIn. DuckDuckGo doesn’t even list it at all. There’s countless examples where DuckDuckGo just can’t find basic information. DDG is just unreliable beyond it’s silly name.
I feel it might have even been better at one stage ?
(Search for bang queries with, not surprisingly, "!bang".)
But also, thank you. I didn't realise there were so many bangs.
The real power is in the "bangs", though; you can use the `!` to immediately jump to the first search result without seeing a search page, or use `!g` to switch to Google for this particular query, among others. It enables a sort of power-user usage that one wouldn't get with Google.
I understand that for some people that's not enough of a time savings to make a difference, but I know DDG well enough to be able to `!` things and almost always immediately get to a successful result. I treat it as an extension of my brain at this point.
If I'm working on python and numpy and I want to look up `argsort`, I know I want to search the numpy page, so !numpy argsort takes me right there.
Any kind of web dev is !mdn whatever and I don't have to scroll through a dozen BS tutorials, I just get the specs.
I used to configure those as search keywords in Firefox (and before, Opera), which do roughly the same without the exclamation point. But on a new browser, even just configuring your favourite top 5 searches is a lot of hassle compared to just setting DDG as the default search and using their bangs.
I've been using Bing for the past few months; it's not great or terrible but is it "viable" enough for people to shift to over time? Or is it not viable because it's backed by a major corporation?
I'm sure there are search quirks with each engine but I've seen issues with Google too and yet it's the "devil we know" ... so people unconsciously work around them.
Maybe it's because I don't have tracking enabled in Google (I'm not logged into my account when at work) and opt out of tracking where I can. Maybe this is the difference between the lack of difference I see and the huge difference so many others see. But I still don't see it as an issue because I generally find what I'm looking for with one search. Might be the third item, but that's not an issue to me.
I hear this so often that I assume something has to be different. I'm curious if others have ideas as to what it might be, or if I correctly identified them.
I'd estimate I'm doing that maybe 5% of the time. It seems to be even odds that I find a satisfying match, though, obviously those are all the hard queries.
The hardest queries are trying to dig up details about stories in the news.
Google knows what you want before you even ask. You might find that convenient, I find it unsettling.
I guess it’s not as bad as Facebook; at least Google doesn’t spoon feed you.
I have no problem with advertising etc. but the tracking and selling of data is such an idiotic thing. We as consumers should have a global internet-law, and be reimbursed for data leaks or usage outside the scope of the application.
By no problem with ads I mean the original ads of google. Very clear they were ads and not intermingled with the results. Scrolling down for the results is nuts. I will click ads if they’re relevant, regardless of if they’re on the right or in the results. So please stop supporting this fraud against advertisers.
Instead, what might help DDG is a plugin that detects when you go past the first or second page of Google search results, and suggests that you might get better results on DDG. It's a little intrusive, but the mental nudge becomes "$oldReliableStrategy has flaws, try $newRiskyStrategy". You get a positive emotional interaction with DDG rather than "forcing" yourself to use it all of the time and "failing back" to Google.
That's quite a claim considering they were reporting PageRank in their toolbar until 2016, and toolbar PageRank was visible in Google Directory until 2011.
Are you talking about PageRank from the original patent?
https://searchengineland.com/google-has-confirmed-they-are-r...
This is the weird thing -- it feels smaller. Back in the early 2000s it really felt like I was navigating an ocean of knowledge. But these days it just feels like a couple of lakes.
(also, I'm pretty sure it was billions already quite early on?)
For instance, if you were to forbid Google from operating on every odd-numbered day, then 50% of the search engine market and revenues would immediately be distributed among competitors and furthermore users would be forced to test multiple engines and they could find a better one to use even when Google is allowed to operate.
Obviously this has a short-term economic cost if other search engines aren't good enough as well as imposing an arbitrary restriction on business, so it's debatable whether this would be a reasonable course of action
This has other drawbacks of course.
Depends on how you count the date, this could create markets where people in different countries will sell Google search results to each other. New VPN providers pop up with the promise of 24h Google coverage. Software startups switch to a system where you bing work 16 hours straight, then get the next 32 hours break and repeat. "Breaking news" has a new Oxford definition, since newspapers change plan to publish news 5 minutes before Google opens for search. Electricity price increases for the first 2 hours of the odd-numbered day to combat the spike in demand. Comcast introduces a new fast lane at only $199 a month that has no slow down access to Google. University groups lobby for a new exemption in the law allowing unrestricted weekdays access. Political parties lobby for also blocking Google on the day of debates, regardless of whether it's an odd-numbered day. It's kinda fun to keep going.
Common Crawl and other sources do in fact have a ton of data that can be used which is very affordable.
The DATA itself stopped being a real competitive advantage probably 2008-2010.
Google's major advantage now is its algorithms and the fact that they've proven it works and is reliable.
Most importantly, its the brand. Google MEANS search in the US and that won't change anytime soon.
PS,... if you need tons of web and social data Datastreamer can hook you up too :)
This may help to explain the poor quality of some the results on queries I run on Google lately that return content obviously written for ranking in SEO but that have very little value.
I have 2 questions:
- What make"the top 3B+" the top ones?
- How can I "force"a search on the other 150B+ pages?
I agree about consumer's habits, but not about quality - i mean google of today is worse search engine than google of 5 years ago.
Now google tries to guess, badly, what you meant, instead of giving you what you asked for. The pleasure of dealing with IT systems is that they give you what you ask them for, not what you meant - it introduces extra error, and worse - one that cannot be fixed by user.
I can rephrase my querry, and google will still interpret it - leading to same batch of useless results.
Just let "google" become the generic term for search, as it's already well on its way.
The main problem is, I think the author is wrong about what Google's "crown jewel" is. Yes, Google has a huge index, but most queries aren't in the long tail. Indexing the top billion pages or so won't take as long as people think.
The things that Google has that are truly unique are 1) a record of searches and user clicks for the past 20 years and 2) 20 years of experience fighting SEO spam. 1 is especially hard to beat, because that's presumably the data Google uses to optimize the parameters of its search algorithm. 2 seems doable, but would take a giant up-front investment for a new search engine to achieve. Bing had the money and persistence to make that investment, but how many others will?
If a government was serious about getting more players in the search industry, they would force Google (and all other players) to make this data public.
Simply say "All user-behaviour data used to improve the service must be freely published".
Make the law apply to any web service with more than 20 million users globally so small businesses aren't burdened.
If the data cannot be published for privacy reasons, the private parts must be seperated and not used by google or it's competitors.
> the private parts must be seperated
This means literally making legal interpretation of all documents on the net, to determine whether each of them is private or not.
As a user that notices the impact of this data: please no, thanks though.
Have you ever visited youtube's home page in incognito mode? It's... bad. Really bad. Not allowing any company to use this (obviously very private) information in ranking would simply make their products suck, horribly, compared to today.
Do you like the personalized recommendations because of channel subscriptions?
I always get the "anonymous default" home page with YouTube and don't care. The home page is just a wasted load before I can start typing in the search bar. As a bonus, staying incognito means all the videos on the right-side panel are related to the current video. Not related to a music video I have playing in another tab.
But that's where differentiation occurs. Every search engine will get short tail results correct. We go back to Google because it also performs with the weird queries.
I agree that algorithmic superiority will probably perpetuate Google's dominance. But making its index public is (a) legally precedented, (b) conceptually simple and (c) a small step in the right direction.
On the other hand, if I have a health-related search, I run it through Google. DDG has the proper content. It's just that it priorities the blog spam. That's an algorithm problem.
Relieving the former, as the author's proposal would do, makes DDG more competitive. As a second-order effect, it would also let DDG priorities resources towards the second problem, making them more competitive still.
Bing does have the reputation of being better for NSFW searches than Google, so I guess that it's normal to have more NSFW false positives as well.
This is what makes me wonder why we don't have a LOT of competing search engines. Perhaps i'm vastly under-estimating the technology and difficulty (I could well be - it's not my domain) but it surely it can't be THAT hard to spawn Google-like weighted crawl-based search results?
It's a long-since solved problem - heck, pageRank's first iteration recently came out of patent protection - it could just be copy'pastad. Why aren't all the big companies Doing Search?
Google pays some large number of people to do search and grade the various results they get to see if the answers are good, which then helps feed back ML.
Heck, according to this article[0], google has been paying people to evaluate their search results since 2004.
[0] https://searchengineland.com/interview-google-search-quality...
The spammers won. Google gave up and settled for "we like the right kind of spam—the kind that took a little effort, and makes us money".
Searching for "north face glaciation" did help as the first page of search results did have one entry on the topic I was actually searching on!
Maybe they should have a "I'm not buying anything" flag!
Many of the top search queries are navigational searches for brands.
And so if tons of people are searching for your brand then if there is a potentially related query that contains the brand term & some other stuff then they'll likely return at least a result or two from the core brand just in case it was what you were looking for.
Google for general search. Duckduckgo fir general if you want something a bit more private but not extreme enough to run your own spiders. Bing mostly for porn search - not being snarky some people do consider it to have better results.
Google custom tailors results for each and every machine. Even if you're not signed in, Google uses your browser fingerprint, the OS it's reporting and location/IP data to custom fit results. There is no "stock" google result.
This is something DuckDuckGo et. al. can't do if they want to focus on a privacy model. DDG does offer location specific searches, which can be helpful.
It's easy to gather the necessary data, but it's hard to know which parts of that data are the most relevant for finding good content and avoiding bad content. Is it more relevant if key words show up in links or titles than in the body of the text? If so, SEO spam sites will include a bunch of keywords in links and titles. Is it more relevant if keywords show up in the first 200 visible words of the page? If so, spam pages will make tons of pages with relevant keywords at the top.
The hard part about building a search engine isn't indexing the internet, it's adapting to spam. Spammers are continually adapting to changes in the algorithm, so the algorithm needs to adapt as well. And the more popular your search engine is, the more money you make and the more able you are too adapt to spam (and the more spammers focus on your engine).
So, the problem isn't that Google has a better index (though I'm sure it does), the problem is that nobody else has the will to spend the money necessary to tune the search algorithm to stay on top of spammers. When Google started, companies didn't care as much about improving their index and instead focused on building their other content (Yahoo, MSN, etc). Google saw the value of search and got a lead on everyone else in terms of curating results, and now they have the momentum to stay in front and have shifted to building content to improve monetization. Nobody else has the monetization network for search that Google has, so they'll continue having the problem that other companies had (Microsoft wants to point you to their other services, DuckDuckGo is limited by their commitment to privacy, etc).
In short, Google wins because:
- it was better when it mattered - it makes money directly from search - its other services improve their ability to understand what users want, which improves search quality and ad relevance
You can't make a better algorithm by being clever, you make a better algorithm by having better data, and that's hard to come by these days. The only way I can think of a competitor stepping in is if they target an underserved demographic and focus data collection and monetization there, and DuckDuckGo is close by targeting privacy conscious power users.
The irony there is that DuckDuckGo can't collect much of that data precisely because of their privacy focus.
You didn't just hit the nail on the head; you drove it all the way in with a single blow. Bravo.
Adaptive crawlers?
Consider DuckDuckGo, which sells itself on privacy, but after more than a decade has only 0.18% market share. Without the power to make it the default in an OS or browser, you'd have to have a really strong value proposition to convince people to switch.
Outside of ad revenue, search has always been seen as something of a "charity" effort for the internet. It's "boring" infrastructure work that can be critically useful but doesn't really make money directly on its own. No one wants to pay a "search toll" and there's no government agency in the world that the internet would trust as a neutral index to run it as actual tax-basis infrastructure.
They are.
This is far more valuable than the general page rank algorithms that were initially developed and have already been duplicated many times in academia and business.
I'm not quite sure about that. 15% of Google searches per day are unique, as in, Google has never seen them before. [1]. That's quite an insane number.
[1] https://searchengineland.com/google-reaffirms-15-searches-ne...
Some of the new things are probably variation as well - as others have mentioned, sentences and voice commands can give lots of new stuff.
Without some clear numbers on that from a major search engine, I think this might be very difficulty to infer.
Or do you mean queries forwarded by home assistants trying to parse inputs?
The calling card of the developer realising that real users never act like you expect :)
As a developer, I search using keywords; for example, if I was looking for property for sale in Inverness, I might search for "property Inverness", whereas I've seen and heard "typical" users use something like "find me a 2 bedroom house with a garden for sale in the North of Inverness" - much more verbose, and containing stop words and phrases unlikely to help (I think!).
So did I.
Around the time I left Google behind I had started to search like my wife did, using full sentences. it sometimes worked better I think.
Natural language is likely the preferred search input method for kids under a certain age, who cannot yet type fluently. My kids formulate very long, complex queries verbally. The other day my son asked Alexa why the machine gun is such a deadly weapon. She replied with a snippet from Wikipedia that was surprisingly relevant.
is not atypical.
http://commoncrawl.org/ http://commoncrawl.org/the-data/ http://index.commoncrawl.org/
related.. Mark's blog is amazing and worth more than any data science degree imho.
https://tech.marksblogg.com/petabytes-of-website-data-spark-... https://tech.marksblogg.com
[edit] in my experience yacy works really well. You have it crawl the sites you frequently visit and their external links and it quickly accumulates to something more accurate than google.
I mean with the newest advantages like machine learning it's more and more possible to _semantically_ link queries. If that's the case, those 15% could become 5% truly unique searches or even less.
"how dumb is trump" and "how dumb is donald trump" are two different searches but they semantically belong together because they mean the same.
A unique search query could still land you on Wikipedia.
That is impossible, and therefore wrong (I'm wrong, please see below). To know if a search is unique, as in Google has never seen them before, Google must be able to decide if a query it receives was seen before or not. Even if we assume Google needed only one bit for each message it has ever seen, and assuming it only saw 15% of new messages each day since its creation more than 20 years ago, it would need to store more than 2^1471 bits.
What could be true is that each day 15% of all searches are unique on that day.
Edit: I'm wrong. The 15% of completely unique messages per day are in regards to the messages per day, and not in regards to all messages it has ever seen, therefore exponential growth doesn't apply. To see that, assume Google just received one search query each day for 20 years but it was unique random gibberish, then Google could easily save that even though 100% of all messages per day are unique.
Assume Google receives 1 trillion queries per year, and has been around for 20 years. Using a bloom filter you can achieve a 1% error rate with ~10 bits per item. So a 200 terabyte bloom filter would be more than sufficient to estimate the number of unique queries.
If you have a list of 20 trillion query strings, and each query string is on average < 100 bytes, you're looking at a three line MapReduce and < 1 PiB of disk to create a table which has the frequency of every query ever issued. Add a counter to your final reduce to count how often the # times seen is 1.
A bloom filter is the most appropriate data structure for this use-case. How is it overkill when it uses less space and is faster to query?
1: https://en.wikipedia.org/wiki/HyperLogLog 2: https://en.wikipedia.org/wiki/Count-distinct_problem
To find the fraction of queries each day which are new, you would want to add a second field to your aggregation (or just change the count), the first date the query was seen. After you get the first date each query was seen, sum up the total number of queries first seen on each date, compare it to the traffic for each date.
You could still hand the problem to a new hire (with the appropriate logs access), expect them to code up the MapReduce before lunch (or after if they need to read all the documentation), farm out the job to a few thousand workers, and expect to have the answer when you come back from lunch.
Of course, Google knows better but to treat every search query literally. Slight deviations and synonyms work for the majority of the people, even if us techies highly oppose them and look for alternative solutions (like DDG) that still treat our searches quite literally.
(And they don’t even need to store all searches for all time for this, thanks to Bloom filters.)
I haven't thought this through, but take all the queries as they're made and create a bloom filter for every hour of searches. Depending when this process was started, an analytics group could then take a day of unique searches, and run them against this probabilistic history, and get a reasonable estimation with low error. Although the people who work on this sort of thing probably know it far better than I.
The real question though might be assuming the 15% is right, do we care about those 15%, are they typo's that don't merge, are they semantically different, are they bots search for dates or hashes, etc.
Tangential - but does anyone else feel that google results are useless a lot of the time? If you search for something, you will get 100% SEO optimized shitty ad-ridden blog/commercial pages giving surface level info about what you searched about. I find for programming/IT topics its pretty good, but for other topics it is horrible. Unless you are very specific with your searches, "good" resources don't really percolate to the top. There isn't nearly enough filtering of "trash".
I suspect that the AI models powering the search results develop a sort of symbiotic relationship with the spam - if the user actually finds what they are looking for by clicking through an ad on an otherwise spammy page, everyone “wins”; the user found what they were looking for with minimum effort, google got their ad revenue, and the spammy page got a little cut for generating content that best approximating the local minimum that links the users keywords to actual intent...
Some of the sites that were hit like Suite101.com went offline. eHow is still off well over 90%. ArticlesBase sold on Flippa for like $10k or some such. One of the few wins hiding in all the rubble was HubPages, but even they had to rebrand and split out sites & merged into a company with a market cap of about $26 million ... and the CEO of Hubpages is brilliant.
Even with IAC on some sites they are suggesting ad revenues won't be enough http://www.tearsheet.co/culture-and-talent/investopedia-laun... "As Investopedia charts its course as a media brand, it’s coming up against the roadblock all publishers eventually hit — the reality that display revenue alone won’t be enough. ... Siegel said he expects course revenue to exceed what’s generated from the site’s free content. While he wouldn’t say what the company’s annual revenue was, Siegel said it grew an average of around 30 percent for each of the last three years."
There is also other factors which parallel the panda update that further diminish the quick-n-thin rehash publishing business model - Google's featured snippets & knowledge graph pulling content into the SERPs so there is no outbound click on many searches - programmatic advertising redirecting advertiser ad spend away from content targeting to retargeting & other forms of behavioral targeting (an advertiser can use a URL as a custom audience for AdWords ad targeting even if that site does not carry any Google ads on it) - mobile search results have a smaller screen space where if there is any commercial intent whatsoever the ads push the organic results below the fold
There are 2 issues, I think.
Firstly, the SE-optimised spam, which has become very good as masquerading as genuine content.
Secondly, Google has dumbed search syntax down a bit, and often seems to outright ignore double quoted phrases, presumably thinking it knows better than I what I want.
As a dev, I do accept I may be an outlier though - with the incredible wealth of search history and location data that Google holds, it seems likely things have actually improved for typical users.
But for increasingly more basic searches about a product I'm interested in or a medication or anything else non-complicated that would have gotten me a clean list of decent, non-paid results even 5 years, I'm now getting half a page of sponsored BS and then another half a page of 'created content' written by a bot or shyster explicitly for gaming Google's SEO.
Not only has Google lost almost all their good will (i.e. Don't be evil), but their products aren't even that good anymore, at least not so much better than alternatives where the negatives of using Google outweigh the difference in quality.
I’ve been using DuckDuckGo and have found I have this problem less. I don’t always find what I mean on DDG, as of now I’d say Google is still better if you’re not sure exactly what you’re looking for is called, but if you know the keywords you need DDG is often better.
Google doesn't make money from you finding what you're looking for. Google makes money from you searching for what you're looking for.
For example, suppose I do a google image search for "pear", because I want images of pears obviously. The first result is indeed a pear, good job google! Except the first search result just happens to come from Amazon, and also happens to be a pretty shitty thumbnail quality photograph (355x336). It's a pear alright, but why is this particular image of a pear first? Google didn't try to give me the best image of a pear, they tried to give me the pear image they thought most likely to induce a financial transaction. Or alternatively, google let itself get cheaply manipulated by Amazon's SEO. Neither is a good look.
A much better pear image, 3758x3336 from wikipedia, is further down the search results. So it's not like google was unable to find good pictures of pears. And a non-image search for "pear" returns the wikipedia page first, so it's not like google failed to noticed the relevancy of the wikipedia article about pears. Yet the shitty amazon thumbnail of a pear shows up higher in the image search results than a high resolution photograph of a pear from wikipedia.
I would agree that programming search results tend to be quite good, but I think this is likely in large part because the average person attracted to programming both has a high IQ and has experience building some part of the web stack. Thus the sites that are quite manipulative in nature would have a hard time trying to fake it until they make it in such a vertical where people are hard to monetize and are very good at distinguishing real from fake. And even if a fake site started to rank for a bit it would quickly fall off as discerning users gave it negative engagement signals.
This is also perhaps part of the reason sites like Stack Overflow monetize indirectly with employment related ads targeted to high value candidates versus say a set of contextually targeted ads on a typical forum page or teeth whitening gizmo ads on the Facebook ad feed.
The lack of filtering of "trash" probably comes from a bunch of different areas
- I think there was a quote that people are most alike in their base instincts and most refined in areas where they are unique. some of the most common queries are related to celebrity gossip & such. There are also flaws in human nature where inferior experiences win based on those flaws. For example, try to buy flowers online and see how many layers of junk fees are pushed on top of the advertised upfront low price. shipping, handling, care, weekend delivery, holiday delivery, etc etc etc
- some efforts to filter trash based on folding in end user data may promote low quality stuff that people believe in. a neutral & objective political report is less appealing than one which confirms a person's political biases. and in many areas people are less likely to share or consider paying for something neutral versus something slanted toward their worldview.
- as the barrier to entry on the web has increased some of the companies that grew confident they had a dominant position in a market may have decided to buy out other smaller players in the vertical & then degrade the user experience as real competition faded. there was a Facebook exec email mentioning they were buying Instagram to eliminate a competitor. Facebook's ad load is now much higher than it was when they were smaller. But the same sort of behavior is true in other verticals too. Expedia & Booking own most the top travel portals.
There has also been a ton of collateral damage in filtering all the trash. So many quirky niche blogs & tiny ecommerce businesses were essentially scrubbed from the web between Panda, Penguin & other related algo updates.
The user data helps/ed Google create the superior UX, as you say. The reach is what makes Google & FB valuable to advertisers. A search engine with 0.1% of Google's user volume cannot charge advertisers 0.1% of Google's as revenue. Returns to scale/reach/market-share are very substantial in online advertising.
I'm glad we're talking though. Those tech giants are too powerful.
Ultimately, the old antitrust toolkit is near useless today, for dealing with tech monopolies. It's not obvious what "break up Google" even means. There are strong network effects and other returns-to-scale. It's a zero-marginal cost business, which was rare enough in the past that economists a ignored it.
We need fresh thinking, a new vocabulary, new tools, but we do need to deal with it.
* an Office suite / enterprise company (Google Cloud + Docs + Gmail + Business)
* a phone company (Android)
* a search company (Google Search + Advertisement)
* and a media company (Google Play Movies, Music, Books and YouTube)
The names would probably become different in time, but you get the gist.
Amazon and Microsoft could be broken up much the same way, in neat categorical 'silos'. Facebook should be trisected into Facebook, WhatsApp and Instagram again. I have no idea how you would break Apple up without utterly destroying their core principle, vertical integration. There is no way to do what Apple does with MacBooks or iPhones if they don't control the entire stack. I'm not saying they shouldn't be, I just see no way.
Granted most of your examples wouldn't be, except for search, but it still seems more interesting to me to just have a bunch of mini googles made from cleaving teams. Certainly that would make for some crazier competition.
The biggest blow to Google wouldn't be to break it up into lots of small companies, you just need to separate the advertising business from everything else and you've effectively neutered the monopoly. Google's genius isn't in hiring the best engineers to providing a ton of services, it's in convincing people that they're not an advertising company, and that is where Facebook has been falling out of favor recently (I'm guessing that's why they bought Instagram, and why Google bought YouTube).
But my thinking was that if you simply cut off advertising all the products still have massive marketshares and could lean on each other, as long as some succeed. Not to mention investors probably willing to prop up such a massive aggregate marketshare (one only has to look at Uber).
If you 'silo' them, success of one division of previously-Google won't lead to all of them dominating.
This is a supposition - while perhaps it seems to makes sense, seems true, “must be true”, it doesn’t mean it is true!
Unless you worked on search quality at google you really aren’t in a position to know if, say, google cloud, or android provides useful signals to search (outside of the signals they’d collect anyways if they were different companies).
One thing people are obscuring is just how crazily effective AdWords are. They work for the advertisers, and they earn google like 70+% of the revenue. Confirmed via sec filings which does break that out. Go play with creating an AdWords campaign and try to infer just how much data google really needs to deliver those ads - it’s less than you’d think.
In short: this overall move is more wishful thinking than solidly reasoned. Surveying the field of streaming video, given the amount of studio driven consolidation, is there really a tons of competitors being held down and will spring up? I am skeptical.
(1) This doesn't actually reduce market share, since each of these are basically different market categories.
(2) Almost all the revenue is from search. That company is the revenue generating arm for the other ones.
You are right though, it doesn't deal with the dominance of the search directly. My hope is a complimentary effect to the above also happens: Google no longer gets gobs of personal data from its other services, allowing other search engines to approach its efficacy.
As is clear I'm not really a fan of direct intervention in a single market, I see it as more of a problem when these giants muscle their way and control more and more markets, creating a vicious feedback loop.
I'm still not sure how this would work on Apple though, since their main differentiator is their design sensibilities and integration rather than their platform monopolies.
I guess iMessage and the App Store do rely on monopoly rents, but I can't think of any way to sever those links without making the iOS platform less secure.
I think it's instructive to look at the rest of the market. How is Mozilla funded? Basically a single gigantic contract with Google. Even Apple accepts payment from google to become the default, and it's not cheap: https://fortune.com/2018/09/29/google-apple-safari-search-en... The same logic applies to pretty much anything Alphabet spins off -- there's little difference between ownership and those contract.
About the only competition this setup produces is the ability for Mozilla to walk away to a competitor bid, which they did for like a year before bailing out at the first opportunity. There's a huge incumbency bias in these contracts. The first parallel that comes to mind is employer provided health insurance. Everyone gets to bid, but the incumbent knows the claims history far better than the competition and we'd only expect them to lose bids to companies overly optimistic about that history. Google knows how valuable various traffic sources are, but their competitors have to guess, and only when their guess is higher than Google's does it pay off. Does anyone think Yahoo winning Firefox was a good deal? I haven't seen any analysis to support that.
> My hope is a complimentary effect to the above also happens: Google no longer gets gobs of personal data from its other services, allowing other search engines to approach its efficacy.
Wouldn't the most profitable thing for these broken up companies be to sell their slice of the personal data pie as many parties as possible? This seems like a net loss for privacy. How much extra would it be worth to set up an exclusive arrangement?
In contrast to most people here, I think breaking up Amazon is far more important to Facebook, Microsoft, Apple and many other tech companies. Only Google is as bad.
Before Google Maps we had a few online map services and they were terrible. Google Maps redefined what it means to have free access to web based interactive global maps, it changed how people find things and it was all payed by the ad business. Later on some monetizing efforts were made for it and competitors started to appear, mostly trying to catch up and copy what Google Maps did, but without the huge cash infusion of the ad business none of this would have happened.
A decade later, people take these things for granted and just want to split services up. I guess it makes sense from their point of view but to me it's not that clear what should happen while still allowing for the type of creativity and speed of development that allowed things like Google Maps to appear because I'm afraid "the next big" thing that could redefine our lives (and improve them) would be slowed down or simply made non-feasible.
This is not true. MapQuest revolutionized things almost 10 years earlier than Google Maps. Google search is what allowed Google Maps to overtake MapQuest. Also, Android providing real-time traffic data of all their users gave them the winning formula.
As was, like, an app that could reroute live instead of relying on pre-printed paper instructions.
Gmaps had both before MapQuest.
I've seen mid-roll ads on songs on YouTube.
Hosting costs & delivery costs (per byte) drop every year. Every year their compression gets better. Every year their ad revenues goes up. YouTube ad revenues have been growing at something like 30% a year for many years.
I think one reason Google doesn't break out YouTube profitability is because as soon as they show they are profitable they end up getting some of their biggest partners (like music labels) using those profits to readjust revshares.
Also if Google claims YouTube is not profitable they can be painted as the victim for extremist content or hate content they host, whereas if they show they were making a couple billion year a year in profits these narratives would be significantly less effective.
Core search ad prices haven't really been falling yet Google blended click prices keep falling about 20% a year while ad click volume is growing about 60% a year. This is driven primarily by increasing watch time on YouTube and increasing ad load on YouTube. Google blends video ad views in with their "clicks" count.
Who is going to sell those ‘neat silos’ cheap advertising to survive with no other business model? Google?
The cloud services (Enterprise apps, hosting, etc) don't need it.
I think we'd probably see a worse Gmail (worse ads or aggressive upsells).Even though a lot of these corporations build offices, hire non-Americans, and pay tons of foreign taxes in countries in which they do business, the main executives and talent still live in the US, the IP is developed here, and the majority of profits end up back in the home country.
It's better for everyone who actually matters - shareholders, intel agencies, government officials, associated businesses, etc - that these companies remain large and globally dominant, even if it screws over US citizens by having to pay the monopoly taxes and suffer the privacy invasions. We're an insignificant sacrifice in the decision-makers' minds.
> 2) 20 years of experience fighting SEO spam.
That's probably a key issue here though. Providing an API potentially makes it easier for spammers to identify ways to boost their content in a well automated manner.
How so? Unless you give reasoning for the scores, or provide live updates etc, just putting an API on search wouldn't change much - you can APIfy search now, there are multiple services offering it as a service. Granted, at some point it's getting expensive, but for SEO research, you're probably not running a million queries.
I also happen to think that is the search engine I would prefer. I think I could build that pretty quick if I had the api access.
Some argue (not necessarily me) that Google isn't necessarily purely optimizing for quality using that 20 year click-and-search log, that they're accepting some inefficiency by biasing for political (left-leaning) gain or "censorship by obscurity". If competitors could more easily build alternatives, which, say, didn't have those biases, then arguably that'd put more competitive pressure on Google to not use their monopoly for bad stuff.
I think we've reached an equilibrium state on this that has significantly degraded the educational quality of search engine results.
The total garbage SEO spam we used to get is gone, which is nice, but what it's been replaced with is technically relevant but mostly manipulative advertising. Product searches will basically give you a bunch of no-name blogs who are almost definitely paid off by one vendor or another.
Even actual inquiries are inundated with search results that do answer the question, but do so in extremely cursory and incomplete way. Or, in the case of recipes, Google seems to prioritize results that give you long, meandering narratives before they actually talk about their recipes. It has some very weird ideas about what people actually want when they search.
One of the most annoying things is how impossible it is to actually find the website of a local business, especially a restaurant, by Googling. Your hits are always Googles' own cobbled together dossier on the restaurant first, then some combination of Yelp, Grubhub, Postmates, AllMenus, etc. pages. If the restaurant has a website you can't tell and it's probably way on the bottom or on a second page of results.
In the past it was a handful of very decent results amidst a sea of total garbage SEO spam. Now it's a sea of mediocre content farm stuff, but it ranged from difficult to impossible to actually dig into detail on things anymore. The old spam we could at least dismiss as crap within a fraction of a second of seeing it. The new spam you have to actually read most of it before you realize it doesn't have what you're looking for.
Having a full text index on that is more involved but hardly impossible. You're completely right that it's not at all Google's secret sauce. Bing has clearly indexed much more than that, plus invested a ton in actually returning good results from their index. And still nearly nobody cares. It's just not easy to make a better Google, and the people most likely to figure out how to do that already work there.
I'd actually advocate for making public an anonymized list of actual search queries.
Domain specific search engines could evolved based on the demand of what has already been searched for.
It depends which sense of "better" you mean. It's nearly trivial to make an ethically superior search engine by just not building the spyware bits of Google.
It's difficult to make a search engine that's "better" along the dimensions of speed, profitability, etc.
As far as I remember Google is actually shrinking its index in terms of number of indexed websites because 90% of the internet are irrelevant for the majority of searches. Basically "quality over quantity" if you can say that.
This is even more depressing. Google was such a wonderful tool for us nerds because we could finally find those usenet posts, personal blogs, tech mail lists, etc. of all the esoteric subjects that had been hard to find previously. Before Google, you'd use lists of curated links (e.g. Yahoo) for a given topic that had been traded back and forth between various sites and other interested netizens.
It's apparent that Google is becoming worse and worse for these types of searches, while it concentrates of more popular queries like "When is the next <my show> on" or "What is the current sports-ball score" or "How big are Kim Kardashian's boobs".
Just like Craig of Craigslist recently came out with an article saying the internet has actually made the news media worse, not better for informing citizens - something he did not predict correctly - it's apparent that Google is pushing us in the same negative direction in the ability to find quality information on non-consumer knowledge.
* https://www.theguardian.com/technology/2019/jul/14/craigslis...
Same reason people won't buy electric cars with 100 mile ranges, even if they very rarely travel more than 100 miles.
From what I can tell, Google cares a lot more about recency.
When I switch over to a new framework or language, search results are pretty bad for the first week, horrible actually as Google thinks I am still using /other language/. I have to keep appending the language / framework name to my queries.
After a week or so? The results are pure magic. I can search for something sort of describing what I want and Google returns the correct answer. If I search for 'array length' Google is going to tell me how to find the length of an array in whatever language I am currently immersed in!
As much as I try to use Duck Duck Go, Google is just too magic.
But I don't think it is because they have my complete search history.
Also people forget that the creepy stuff Google does is super useful.
For example, whatever framework I am using, Google will start pushing news updates to my Google Now (or whatever it is called on my phone) about new releases to that framework. I get a constant stream of learning resources, valuable blog posts, and best practices delivered to me every morning!
It really is impressive.
So it could very well be that as more users adopt the new language/framework in the first couple of weeks they have taught google those associations.
Google isn’t a search company. They are a distributed machine learning company that make most of their money from learning what people want and showing relevant ads to them.
Really good or really bad only exists if there is something else to compare it to.
For the same reasons you’re exalting them, I have non-technical friends who asked me how Google knows so much about them (and suggestions on how to avoid it) because they found it too creepy.
I don’t think people forget Google’s results are useful; some just think they’re more creepy than valuable. You seem to have picked your side in that (im)balance, and other people prefer the other side.
There’s also the relevant consideration that no matter how useful they may be, they should have no right to impose themselves on you. By this I mean that one should be free to refuse their creepiness, understanding the price is their usefulness. Yet, Google is the subject of privacy violations all the time, and they are caught time and again lying about what they collect on users.
Also, for incognito stuff, it'd be nice to have read-only basing on stock profiles related to various activities or people.
But I have a hard time believing google truly partitions everything in a multi account setup.
Tell that to the people who had their privacy violated by Street View[1]. And the people who specifically disabled location services on their Android devices but were still tracked[2]. Or all the people who have no idea what Google Analytics is and never consented to it, but are profiled by it everyday.
> All of their services are a choice you are making.
I do my best to avoid privacy invading companies, and as a technical user I find it tiring and know I deal with consequences (e.g. broken websites). It perplexes me that comments like yours still pop up. We’re not the only segment of the population that exists; non-technical users are the majority, and they have the same right to privacy as we do, with a modicum of transparency. If even technical people are regularly tripped by privacy invasions we didn’t know about, what chances do non-technical users have?
[1]: https://www.nytimes.com/2013/03/13/technology/google-pays-fi...
[2]: https://qz.com/1131515/google-collects-android-users-locatio...
1.generally speaking I would think VERY few people care about an image of their property being on street views. 2. It's not really illegal to take pictures so even from a legal standpoint it seems like a gray area. 3. I understand there can be individual reasons for not wanting this, but it seems to be a very large net positive. And I would apply that statement to most other tracking and data policies they have.
If they are lying about how their services track people, that is definitely grounds for concern. The transparency can definitely be improved, but still these are people with Android phones and people using Google Analytics. No one is forced to use these things they are free to use any other service or create their own.
And my attitude is out of pragmatism and how I think privacy issues should be handled. I don't have any problem with the way Google uses my data so I don't care to fix a non problem. And I don't see it as their responsibility to change a way of business when anyone is free to use any other service or create their own, since I don't find it offensive.
> Google on Tuesday acknowledged to state officials that it had violated people’s privacy during its Street View mapping project when it casually scooped up passwords, e-mail and other personal information from unsuspecting computer users.
That answers your first three paragraphs. There’s no “if” to their lying and privacy invasions. They’ve been caught and admitted their actions time and again.
> No one is forced to use these things they are free to use any other service or create their own.
It is here I will respectfully give up on continuing the conversation with you. You’re either ignoring my main point or truly don’t care for the majority of users. Most people don’t understand the ramifications of these choices and for good reason; they are hard to understand. By suggesting non-technical users create their own services and devices, I’m now wondering it you’re trolling me.
> And my attitude is out of pragmatism (…) I don't have any problem with the way Google uses my data
Which is valid, but irrelevant. I’ve already mentioned in the top post different people make different choices. I presented another side and used facts to justify it. If you’re going to answer with mere opinion, you’re not adding to the points made by the original poster.
My first points were about the streetview product. Scooping up passwords is obviously not the intent of that product, maybe that was an error or they changed the core product at some point? I can't read the paywalled article.
I'm not suggesting non-technical users create products... you're reading so far out of context. Just because user X can't create a new product does not mean that we should place sanctions on company Y. I'm glad you used facts somewhere else because in this post you just illogically connect a bunch of dots.
Yes some of it is my opinion and alot of this is yours. But a fact is still no one is forcing you to use these products, then you went off about stolen passwords and trolling and resigned yourself from the argument. That sounds like a rationality of a completely one-sided biased individual in itself, respectfully.
Yes everyone agrees transparency is good and lying is bad. Google is not Evil Or Benevolent. They're just people...
"And I don’t use them. I hoped that by continuing to mention non-technical users you’d get it, but this was never about me. You keep bringing up that argument, but read what you replied to in the first post — I recounted the experience of non-technical people I know, not my experience. Stop telling me I have a choice; the point is not us, it’s non-technical users who don’t have the knowledge to make informed choices!"
Haha you are so ridiculous. This was your first post:
There’s also the relevant consideration that no matter how useful they may be, they should have no right to impose themselves on you.
Then you say you don't know why I bring up that you don't need to use Googles services... C'mon man get real. That's why the point about using alternatives or creating new ones is very relevant and this entire thread is about sanctions. Don't start a convo you can't participate in and then just claim you won and leave, that's childish behavior.
That is an insane extrapolation, and the reason I don’t want to continue the conversation with you: you’re answering points I’m not making. I haven’t even hinted at sanctions; I have no idea where you’re getting that from.
> But a fact is still no one is forcing you to use these products
And I don’t use them. I hoped that by continuing to mention non-technical users you’d get it, but this was never about me. You keep bringing up that argument, but read what you replied to in the first post — I recounted the experience of non-technical people I know, not my experience. Stop telling me I have a choice; the point is not us, it’s non-technical users who don’t have the knowledge to make informed choices!
> That sounds like a rationality of a completely one-sided biased individual in itself, respectfully.
Believe what you want. I just don’t want to keep wasting my night arguing with someone that started a discussion but refuses to address the points originally made. Why reply, then?
Maybe I’m not explaining myself well enough, or in the correct way for you to understand, or maybe you’re the one not grasping what I mean. It doesn’t really matter where the problem lies, just that it’s clearly not working.
Maybe if we ever meet in person we can resume this conversation, but tonight it’s not being productive, so I genuinely wish you a good week and sign out here.
This is information that Google doesn't have any need for (noise) and didn't want in the first place.
They also self-reported the failure, where they could have just nuked it and we wouldn't be having this conversation.
Some writer from Gizmodo tried that last February. Let's just say that you are technically correct that you don't need any of the big tech firms.
https://gizmodo.com/i-cut-the-big-five-tech-giants-from-my-l...
Just as a general observation without taking either side:
People routinely fail to recognize both sides of a particular thing. It's why we have sayings like "You don't know what you've got til it's gone."
Sо I wonder how much of the magic you perceive might be just your interests matching the interests of the most other people using Google and thus it's not just Google magically guessing you're into Javascript (for example) now, but Javascript being popular and this being the cause of both Google returning matches for it and you starting to use it? Did you ever do a clean experiment - e.g. try to learn APL or some other relatively obscure language and have Google return all results about APL and none about Javascript?
Same when I owned a Pixel after hearing about Google Now and their ML magic there. Nothing more magical than an iPhone in terms of suggestions. The camera was amazing, but not all this supposed contextual stuff.
There's no arguing what you're describing is useful, but it's nice to keep in mind that there are downsides even if you ignore the privacy argument (which, IMO, shouldn't be ignored).
https://myactivity.google.com/
For many people it is enough to be totally creeped out about Google.
Also, that Google remembers context can be handy but it is not essential. Without context, I am sure you would be equally capable of finding what you are looking for, although it might take a little more typing since you'll have to supply the context yourself. Imho, convenience is not a good argument for giving away your personal information.
I hypothesized once with an ex Microsoft HIGH up that it probably took 10B to launch bing. He said I was almost exactly on the nose.
Also this is a ridiculous thing to ask for. How much money do you think Google pays for the bandwidth to crawl the web? How much do you think it costs to run the machines that create indexes out of that? How do you value the IP involved in the process?
Google should give away the fruits of that labor for free, plus invest in a reasonable API to download that index? Plus the bandwidth of sharing that index with third parties? It’s probably not even feasible aside from putting disks or tapes on multiple semis to send to clients. The index is 100 petabytes according to [0]. With dual fiber lines, and no latency for mind bending numbers of API calls, that would take 12.6 YEARS to download a single snapshot.
[0] https://www.google.com/search/howsearchworks/crawling-indexi...
As someone who was operationally responsible for a search index (formerly VP Ops at Blekko) the kinds of things crooks tried to do was pretty instructive on how they use search in advancing their efforts.
What do folks even mean by "Google's index"?? Google results combine tons of signals, including personal histories for each user. Sharing metadata for the top billion urls wouldn't cover half the functionality, or make a competitive engine. And on the other hand, there may not be a single other organization in the world prepared to manage a replica of the entire data plane that impacts seatch. The proposal is somewhere between underspecified and nonsense.
Robert Epstein (born June 19, 1953) is an American psychologist, professor, author, and journalist. He earned his Ph.D. in psychology at Harvard University in 1981, was editor in chief of Psychology Today,
He has also made some questionable claims about google manipulating search results to favor Hillary Clinton.
https://en.wikipedia.org/wiki/Robert_Epstein#cite_note-15
His research is based entirely on his own experience
“It is somewhat difficult to get the Google search bar to suggest negative searches related to Mrs. Clinton or to make any Clinton-related suggestions when one types a negative search term,” writes Dr. Robert Epstein, Senior Research Psychologist at the American Institute for Behavioral Research and Technology.
https://www.vox.com/2016/6/10/11903028/hillary-clinton-googl...
https://www.politifact.com/punditfact/statements/2016/jun/23...
(Disclosure: I work at Google, but this opinion is my own)
I searched from a Firefox private window over a VPN from the Netherlands. But since the results are the same (regarding presence/absence of "email" as an autocomplete term) I don't think it matters much.
I'm not ruling out the possibility that google actually is manipulating search results, but this is not proof of that.
* donald trump email * donald trump email address * donald trump email list * donald trump email newsletter * donald trump email list signup
And just to drive the point home I tried "root_axis emai" and got "root_axis email". Try anyone else and you get similar results, 'barack obama emai', 'george bush emai', etc etc. So yes, this is proof that the results are scrubbed for Clinton email.
Since Hillary Clinton is not unique, then it's not proof that her results are treated differently.
Compare these results:
* https://www.google.com/search?hl=en&q=hillary%20clinton%20em...
* https://www.google.com/search?hl=en&q=will%20smith%20email
I’m not saying google is manipulating auto complete intentionally (though they might be), I’m just saying your counter examples are irrelevant.
It would be like “Donald trump Russia” NOT auto completing then someone saying “but neither does Taylor swift Russia, so we’re good.”
Claiming that she's unusual, since you expect it to work for her based on stories written about her is a different claim.
1. I said "only for Clinton" in the contect of Trump vs Clinton. Then I compared to other U.S. politicians, where the example still holds. The intended meaning is perfectly clear.
2. Obviously the fact that Clinton was the topic of a scandal involving email is the assumed context here. That's why I said "if anything, we would not expect "email" to be autocompleted for trump" (implied: but for Clinton, email is a more relevant search term based on published news, etc.).
Also, if I type "hillary e" I get "Hillary Emails PDF" as an autocomplete suggestion. If I type "clinton e" I get several email suggestions "clinton email PDF" ,"clinton email film", "clinton email FOIA", "clinton email download".
Please include these confounding results in your analysis
This is the problem with search query anecdotes, it ultimately produces a subjective and pointless debate about how one should interpret search suggestions for arbitrarily selected one-off queries. There is no methodology here, and we don't even know how widespread any particular suggestion results are. Any person with an agenda will be able to cherry-pick search queries that confirm their narrative.
I think the bigger problem is that these adjustment appear to be done manually. We know that's what they're doing to avoid racist autocomplete phrases (and reasonably so, I think most people would agree). Having such judgments made on specific political topics/events/scandals will inevitably result in political bias, especially in an organization whose workforce is so politically skewed to one side.
It is a rather suspicious result. Suspicious enough that it is hard not to imagine a deliberate act is behind it. I admit that I don't have any proof.
I just tried searching again a few times with new private windows, and "email" alternates between first and fourth suggestion for trump. But the more important point is the absence of the suggestion for clinton: we know it's been in the news extensively, we know people searched for this phrase a lot, and now "email" has been removed from the suggestions only for Clinton. I tried searching a few more U.S. politicians, and for all of them "e" autosuggests "email" somewhere between first and fourth place. So the complete absence for Clinton does not look like a generic algorithm change.
Curious....Did you get any results for "hillary clinton e-mails"?
what's up with that
If the trend data is not being used for the autocomplete results, what is? And why?
https://trends.google.com/trends/explore?geo=US&q=hillary%20...
Despite it being off topic, can we define why those claims are questionable? Is their data proving those claims wrong? Because with all the Google political controversies over the past few years, and given the political donation history of Google employees, it’s highly plausible that search results are manipulated to favor certain politics over others.
If the “questionable claims” have been disproven or are inaccurate, then it would seem that you’d provide some proof. Essentially, it you are to claim the search engine was not biased towards Clinton, certainly there would be some proof of that? It’s more reasonable to suspect Google manipulating search engines than not, given the political environment at Google.
The real “questionable claim” is that Google is neutral in any way — which is kind of the entire premise of the article. If Google were completely neutral, then why would their monopoly on search need to be broken?
the claims are questionable because his methodology is questionable. If he claims google is biased, he should have a good peer reviewable study that proves this..not google is biased because google didn't auto prompt me with "created AIDS" when I typed in hillary Clinton....
And he's the one making the claim that google is biased...The burden of proof is on HIM.
This is a forum for people in the tech world, right? Shouldn't we question N=1 "studies"?
Even if the specific engineers and managers in the video clips don't have the level of authority to make the changes they're talking about, it's still chilling that their attitude could be common in Google and they see political ends of their great power as being some kind of great responsibility; instead of respecting the idea of equal/diversity of opinion.
Going to go out on a limb and say I don't respect "Hitler was a bad man" and "Hitler did nothing wrong" equally. Individual employees are allowed to have opinions...even opinions I don't agree with.
Maybe this Google 'expose' is the first time they're not guilty of that, but anyone who still finds them credible after their last several blatant mischaracterizations is far more forgiving than I would be.
" ... Project Veritas has edited the video to make it seem that I am a powerful executive who was confirming that Google is working to alter the 2020 election. On both counts, this is absolute, unadulterated nonsense, of course. In a casual restaurant setting, I was explaining how Google’s Trust and Safety team (a team I used to work on) is working to help prevent the types of online foreign interference that happened in 2016. Google has been very public about the work that our teams have done since 2016 on this, so it’s hardly a revelation. The video then goes on to stitch together a series of debunked conspiracy theories about our search results, and our other products. ... "
That's not really off-topic - The fact that the author still supports the claim calls into question their ability to make further claims about the subject.
[1] https://www.snopes.com/fact-check/google-manipulate-hillary-...
[2] https://www.vox.com/2016/6/10/11903028/hillary-clinton-googl...
None of these debunk the claims made by Google employees in the Project Veritas videos.
All you can do is disprove the claims of their existence. Someone tries to claim there's a pink unicorn in the garage, and you can check the garage and say that the it is pink unicorn free. Someone tries to claim political bias exist for hillary clinton in instant search results, and these articles disprove that claim as using cherry picked evidence.
Now, if you suspect that the instant search results are politically biased, then the burden of proof is on you to provide evidence of that existence - preferrably without cherry picking evidence to fit an agenda yourself. Otherwise it's just hand waving and click bait.
The proof is a senior Google employee admitting to bias and manipulating results in the Project Veritas video. There's also plenty of anecdotal evidence you can see for yourself as a user. In addition to that I know many people who work at Google and the vast majority of them have extreme political bias.
Likewise, the handful of people you know having political views, in an organization of 85,000 people, is not evidence that that organization's search results being biased.
If there is so much anecdotal evidence then it should be easy for you to prove, right? Or are you afraid of being disproven?
James Okeefe is publicly known for selectively editing video to make false representations, and this duck is no different.
You specifically claimed that there's plenty of anecdotal evidence. Where is it?
> anecdotal evidence > I know
So much hard data here...real hacker news material...
"The proof is a senior Google employee admitting to bias and manipulating results in the Project Veritas video."
>Project Veritas has edited the video to make it seem that I am a powerful executive who was confirming that Google is working to alter the 2020 election. On both counts, this is absolute, unadulterated nonsense, of course. In a casual restaurant setting, I was explaining how Google’s Trust and Safety team (a team I used to work on) is working to help prevent the types of online foreign interference that happened in 2016. Google has been very public about the work that our teams have done since 2016 on this, so it’s hardly a revelation.
https://medium.com/@gennai.jen/this-is-not-how-i-expected-mo...
I'm going to retread a comment I made when I first learned about Epstein a few months ago: https://news.ycombinator.com/item?id=19167084
In short, and ignoring my ad hominem attack on his motivations, I encourage you to read/skim his two "studies" [1][2] and see how absurd they are. You might dismiss my claims and summaries as biased, but I think I was pretty open-minded towards his conclusions until I read them.
[1] https://www.pnas.org/content/pnas/112/33/E4512.full.pdf?with...
[2] https://aibrt.org/downloads/EPSTEIN_&_ROBERTSON_2017-A_Metho...
Would be nice to know whether it was because of the search history bubble he lived in or not.
If I type "Hillary Clinton Emai" I don't even get a suggestion for email :/
https://sputniknews.com/us/201609121045214398-google-clinton...
I know Sputnik isn't a good source, but according to him, they were the only ones who would publish the findings without edits.
They only let him do that because it fits with their agenda. The real test is if they would let him publish an article on Putin's corruption...without edits..
"But what about those nasty filter bubbles that trap people in narrow worlds of information? Making Google’s index public doesn’t solve that problem, but it shrinks it to nonthreatening proportions. At the moment, it’s entirely up to Google to determine which bubble you’re in, which search suggestions you receive, and which search results appear at the top of the list; that’s the stuff of worldwide mind control. But with thousands of search platforms vying for your attention, the power is back in your hands. You pick your platform or platforms and shift to others when they draw your attention, as they will all be trying to do continuously."
But this is a huge problem. I'd rather have 10 independent search providers instead of 10 companies proxying the results of google. It's worse, if I don't even know from which index the results come from. I guess, many people don't know, that Startpage shows you Google results.
I don't want Google results! I want different web crawlers ordering the results according to my taste without tracking each and every page impression of me. Give me that and I'll switch in a heartbeat.
Okay, great, we'll see how you browse and use that to determine...
> without tracking each and every page impression of me. Give me that and I'll switch in a heartbeat.
Uh? How exactly do you want them to do that?
What google does really well are two things:
1) It ranks those pages according to the well-known PageRank algorithm, although the details are kept secret. They also learned how to punish sites, which try to mislead the algorithm (Duplicate content, excessive SEO, link farms, etc)
2) What they also do well is returning search results in a very short time, based on a relevancy of your search term combined with the precalculated ranking. It also looks at your history in this step and the data it tracks about you.
My preferred search engine would open up both bullet points, giving you the option how pages are ranked and how relevancy is determined. For example: Rank personal blog posts higher or rank news higher or only use an inbound link count. Or you could get fancy and crowd source ranking (similar to hacker news). You could also configure search term relevancy. For example: did you ever search google for something specific and it automatically adjusted your term to something similar, more popular?
You could open up all those decisions and give the user options. Google on the other hand keeps everything secret and uses your personal data in unknown ways. I would love to have a search engine which is configurable. Yeah, not an easy thing to do. But it would be awesome.
I would argue Google already supports some of what you want, like allowing you to search just for news, although additional filters based on something like the niche you're looking for or the type of content don't hurt.
The majority of Google's Revenue comes from Google, Youtube, Gmail and Play. They make so much $ because they have the biggest network effect of advertisers-eyeballs in the world along with Facebook. That. Is. Unbreakable. Even more than a social network's network effect, because the friction to switch budgets and people in a company is higher than a guy telling their best friends to download an app.
And then, YT is a network effect. And then, Play/Android is also a network effect. And then there's the branding. But presumably every big company has the latter. Still, what a brand. Everyone knows what Google or Android is. Every. Single. Human.
Finally, because they make all this money, they can pay to be the default on the other half of the devices, Apple's devices, to use Google as default. Last time I checked, $5B a year.
Hence, this article is so bad.
I don't even care about Google, just saying.
edit: did I mention Chrome? They've got chrome too, with the googleverse as default.
In order to compete, you would need good results and good performance for your organic results and your ad program in addition to a huge budget for traffic aquisition and patience and ability to execute on strategy consistently over multiple years.
Also, keep in mind that people will judge based on perceived quality, not objective quality. Simply being shown as a Google result increases the perceived quality for most results -- in order to be seen as equal, the competitor will need to have consistently better results.
Can you provide some context? Article does not mention Thiel, Russia or China.
And I don't even get why you say China and Russia is anti Google. They are anti-information more than anti-google. If google allows them to control what information people get they will have no problem with Google. They are further anti-west and are concerned that Google will play ball with western governments to give them access to Chinese and Russian data. Maybe I am missing what you mean by them being anti Google.
See: https://www.statista.com/chart/17109/us-digital-advertising-...
https://www.cnbc.com/2019/02/20/amazon-advertising-business-...
That said maybe my sources are not good? But it's better to think is someone being truthful or trying to manipulate me with a specific position...
You started with a decent premise, that google isn't an actual monopoly because it has "competitors" (and yes, those quotes are intentional). But then you go off the deep end and completely lost me to the point that I didn't even finish the entirety of your post.
There are many many reasons that someone could be against google's absolute market dominance.
Technically correct, but not truthful.
You should look at markets, not globally. Google doesn't have a global monopoly (and likely never will because China, Russia won't allow their web to be controlled by a US company), but it has a monopoly in lots of markets (read: countries).
Having a popular product != having control over the supply of a market. Market dominance through having a better product than the competition is how the free market works.
They have already been condemned for their Google Shopping service for monopoly abuse on search.
A company does not need to be a monopoly to face Anti-trust measures made in the public interest. In the 90s you could have run Linux on a PC, installed Netscape on a PC, or bought a Mac, but Microsoft was still treated as a monopolist. The same with IBM in the 70s. There were at least five to ten other major suppliers of mainframes, but IBM was so dominant in the market that they were aggressively regulated for anti-trust.
Google is not the victim of misinformation. Everyone is just starting to understand how important they are as the de facto gateway to information (regardless of choice) and mulling over the implications of it dominating add revenue, search, etc.
>Google is a profit-seeking corporation that needs to expand into China to generate more money and bends the knee as much as it can to make inroads there. Far from being a source of truth they follow all Chinese censorship laws (why, because they would be blocked from operating there and unable to make money off of the Chinese). China likes everything about Google other than the fact that it is not a domestic corporation.
Whew the irony
Thiel's criticism was that because Google wasn't working with the U.S. Government, and was operating in China, that Google isn't patriotic.
Not only is this rhetoric a false equivocation, but it's dangerously wrong.
Google definitely has a dominant market position.
This is not how it's suppose to work.
That obviously wouldn’t be a simple job, of course, and maybe there are some interesting reasons why it wouldn’t work well.
Outlaw ad tracking in some way.
Given the complexity of (a) Google's search and ad integrations and (b) the adtech landscape as a whole, this would be difficult to do legislatively. That leaves settlement with the DoJ, a costly and time-consuming path.
The author's suggestion is not exclusive against a break-up of Google. Its moderation and basis in precedent, however, make it something multiple agencies--not just the DoJ--could implement. Including Congress.
And breaking Microsoft's attempted browser monopoly required the DoJ.
To be clear, I believe we will eventually have to break apart--at the very least--Google and Facebook. But there are advantages to the author's proposed tactic of making the index public. It's a cleaner, cheaper, and quicker solution and doesn't harm the odds of a future break-up.
Apple and Google competed. Government intervention is rarely the right answer. A bunch of people who are both beholden to lobbyists and ignorant to technology won’t produce the outcome people think it will.
Besides, the last thing anyone should want is more government power. Given the choice between trusting the government - that has the power to take away my liberty and my money - with more power or trusting private corporations, I have much more to fear about government.
The slap on the wrist from the DOJ didn’t lead to decreasing dominance on Microsoft. Google/Facebook/Apple competing did.
A search engine is not life or death. Google doesn’t stop anyone from creating a website and reaching consumers other ways.
As someone outside of the tech bubble, this statement always confuses me. I see it once a week or so on this site. Someone will say 'well, company x is doing y, what's to stop you from just disrupting them and working around?"
Because people get their information in consistent, and predictable ways. And one of those main ways is to search for it using a popular website, so popular it's literally called 'googling' something.
If you are an unknown, a search engine is literally life and death for your company, from what I can tell.
Again, I'm outside the tech sphere, so maybe I'm just ignorant. But I don't think so.
I’ve worked for B2B companies that actually had a sales team.
It would be easy to write a law, not to pass one. Generally speaking, when a law is unpredictably disruptive it becomes (a) difficult to pass and (b) time-consuming and costly to defend in court.
This is a rare instance where anti-corporate leftists and dejected right wingers could actually do something substantive together.
In most cases I would agree with you the defenders of private industry are pretty fierce in the US, but all that ideology sort of dissipates when you feel like you are being oppressed (whether or not, or to what degree it is true).
Potentially. It's still more difficult than the author's index proposal. No reason they can't be pursued in parallel.
Regarding a search-ad break-up, I'd guess there would be lots of wrangling over (and lobbying around) defining search and advertising. For example, is Amazon's product search tied to an advertising business, given it sells third parties' products?
What would be the revenue model for their search then?
I read OP's proposal as separating Google search business (search + search ads) from its third-party ad business (e.g. AdSense).
Doesn't that make them an ad business?
Pre Split:
- Company A
- provides: search
- sells: ad space
- Company A
- provides: search
- sells: ad space
- Company B
- (re-)sells: ad space
If post-split that’s company B, I’d argue that is a genuinely new structure with some different properties.
Company A would still have an ads team to manage the integration, but they would explicitly outsource ad selection to (potentially) multiple partner companies.
What you're describing is exactly what Google ads are today.
The actual hard part is when it comes to ranking and sorting the data in any useful way, and doing it within like 100ms. Plus various other issues like spam protection etc. This is where Google excels (at least in my opinion).
Is there a reason to break up Google other than that they are doing well? Other search engines seem to have no problem establishing their own niche and doing well.
Then again, the concentration of power in a handful of operators likely threatens the open internet.
i.e. iOS (especially new versions) would fall under copyright protection [0].
PageRank is a patented technique for search. The patent apparently ran out about 6 weeks ago [0].
While both copyright and patents are intended to protect creators for a certain period of time, copyright protects a specific work and patents protect an idea. Patents should generally expire much more quickly since they cover a much broader topic.
I also realize both systems are completely crippled at the moment, but I'm trying to stick to what they're at least intended to be.
[0]: I'm sure they have tons of patents centered around iOS but this is about protecting the OS itself. [1]: https://pulse2.com/googles-pagerank-patent-expired/
The patent is expired but could you argue the index Google is accumulating and aggregating could be copyrighted work in the same way that iOS is? It has taken considerable effort, expense, and expertise to cultivate a useful index.
Full disclosure I’ve no idea what I’m talking about just thinking I can’t see what basis people have to force Google to open up their key money maker just because. Other people/companies can create their own indexes with the now expired patent, no?
One obvious search query string is: reddit + candidate and see some surfaced and some less so.
The author's proposal, making Google's index publicly accessible, would leave Google search intact. Google's algorithm would presumably remain proprietary.
For people who like Google today, nothing would change. For people for whom Google falls short, there would be new options. Looks like a clear win-win for consumers.
The limits of Bing, Qwant, DDG, etc. are not the number of the indexed pages.
It's that, give them the same pages, and the same search terms, and they don't return results as good.
But it's worth something. Giving DuckDuckGo direct access to Google's index, including the ability to train models on said index, would improve the competitive landscape.
This is not a good thing. Basic keyword searching on an index isn't viable to compete, then you go back to when keyword spammers were at the top. Google should be forced to distribute all their search technologies to the public with the threat of the Alphabet corporation being dissolved if they do not comply. That is the only way to get us out of this mess and break Google's stranglehold on the internet.
Thus, small businesses are less likely to reach the threshold of the index size required to make profit ratio comparable with that of their bigger competitors.
What would be the alternative ad-free business model for Google to pay for its datacenters? Paid subscriptions?
I've been performing google searches for 20 years. In an alternate past universe... if I was paying $9.99/month for paywall access to their ad-free search engine, Google Inc would have more sensitive data collection about me -- not less. Google would have decades of my (sometimes embarassing) search history specifically tied to my paid account.
Ads can reduce privacy but they can also increase privacy by making explicit logins optional. (In other words, the anonymous google searches I did at the office during work are uncorrelated with the searches I do at home or at school. Without explicit Google account logins, the searches I do on my smartphone at Starbucks can be uncorrelated with the searches I do on the desktop at home.)
I don't like ads but the alternative ad-free revenue model is worse: Google Inc having my credit-card payment info (which means my real identity) and all my private search queries tied to it.
The thing the article misses is that search isn't really Google's crown jewel anymore. It's their position at the top of the adtech ecosystem, and while they bootstrapped that with search, I kind of doubt that search is the main thing driving their ad views today.
Example: if you type "broadway shows phantom of the opera", the first non-ad link redirect to www.broadway.com. But www.broadway.com doesn't do anything: they just buy tickets from the legitimate website (www.telecharge.com in that case) with an extra ~$30 fee per ticket tackled on top. I won't go as far as to claim that Google is ok with it because broadway.com must spend tons of money on ads, and not telecharge.com... but I'm suspicious.
Counter example: "green card lottery" now points to the official us gov websites, while few years ago it was scams after scams all the way down.
How exactly "do* you break up a Google or a FB so that (only one possible reason, but the one cited here) they don't control too much media/mind share?^
Laws usually want to be general, and my suggestion doesn't necessarily lend to that, but I'll suggest it anyway.
Facebook doesn't need to be broken up into several companies. It can just be shut down.
I don't mean that it should (justice-wise) be shut down. I just mean that we won't lack for social media. We will have social media alternatives the day after FB shuts down. Theres a chance we'll get something more open instead. There's a good chance we'll get several small replacements instead. There is 0-chance that we'll lack for ways to share posts and post pictures. This isn't Bell, where we need to keep the phones working. The phones will work fine with or without Facebook.
There is no need for a company to generate $70bn in revenue, in order for us to have social media. That's a key difference from all antitrust cases of the past.
YouTube is another sort of example. If it shuts down, alternatives will pop up with immediately...maybe open ones.
There are justice questions (is it fair to shareholders/employees/zuck?) There are legal validity questions (why FB and not apple?). But, for the practical questions... the problem is an easy one.
^fwiw, I also think this is the most worrying part. These companies have a tremendous control about how and what people think. They make Murdoch media look quaint.
The problem isn't so much that google has a monopoly on search as that they abuse the monopoly to force their way into new businesses. Force google to spin search off as a separate entity from the rest of the company.
To me, the problem is power, not necessarily market/pricing power or even exclusion power. The problem is that the "public square" is essentially a handful of private monopolies. What Google considers "inappropriate" is banned or disadvantaged on YouTube. What FB (or Twitter) considers distasteful can become absent from the public or political debate.
It's no joke, at this point. Things "woke up" when it seemed that shady stuff happened during trump's election. But.. several revolutions had already started on social media, coups, counter-coups, civil wars. 10s of thousands of public officials around the world have been elected on FB.
That is a lot of power in very few hands, under a veil of secrecy, often with bad incentives. It was cheap printed paphlets and union halls that made fascists-vs-communist street fights a thing in the 30s... mediums. These days, an algorithm notices that telling (proverbial) fascists about communists is good for engagement... And the algorithm optimizes. I'm not accusing them of doing this deliberately doing this, but the feed algorithms optimize this way for a reason.. incentive.
A single large company is also easier to regulate than a bunch of smaller ones.
If you are concerned about monopoly power over social media, go ahead and break up Facebook. If you are concerned about privacy and the security of social media data, you are probably better off regulating Facebook and raising high barriers to anyone who wants access to Facebook's data.
My guess is that Google's not lobbying effectively.
It occurred to me that the two most powerful things Google has to work with are records of clicks, and the time users spent on the webpages Google returned. I've argued against Google monopoly before because I can throw together a web crawler and search engine in a weekend, so it's not like it's a hard market to enter.
> According to W3Techs, Google Analytics is being used by 52.9 percent of all websites on the internet
This is the real problem though. When a search engine sees a new query, it uses everything it's got to assert which pages the user wants, but with Google Analytics, they can test their assertions constantly to see if a user actually wanted that web page. Then your future queries could be compared against previous queries that were validated by a user spending several minutes of active time on the returned page.
I'm sure Google's algorithm is great and all, but I really think this is what sets them apart.
No: "Search does not use Google Analytics for ranking" https://www.youtube.com/watch?v=LLmO1GE4GvI
That's ranking.
If they use the data for ranking, they'll lose the huge amount of value it has for validating.
It's possible to achieve something similar by measuring bounce rate, but in any case, the sheer number of searches is what matters. For new entrants, this kind of expertise will be very hard to get.
I looked into it, and it seems DDG is using Bing and Yahoo search API and lots of other sources. I looked into the pricing of Yahoo's search API / Bing search API,it ranges from $0.80 / 1000 queries to several dollars per thousand queries.
It seems to expensive to be economically viable with ads, what am I missing ?
For comparison, that works out to ~$15 per year for the average users search volume.
If you can't make $15 per user per year on search, you have bigger problems...
https://www.theverge.com/2018/12/12/18137369/duckduckgo-duck...
There is also ddg.gg which I find even easier to type.
This isn't even touching on that Amp email garbage that they're pushing.
It’s rather obvious and there’s a lot of google employees on hacker news so I don’t really see the point of arguing further. You’re having a giant part of the defining tech of the 21st century controlled by one company. How’s that healthy?
I don't think that's a good idea, to let one company "have" search. Basically, you're letting a single company regulate discovery and information for a large chunk of the culture as a whole.
Why is a single company allowed to have control over essentially half the internet?
If that company has control of most of the discovery and monopolizes the information produced, they have one level upstream control of most of the Internet and the culture. It's like they don't just own the reservoir, but own a majority share in the river and most of the dams upstream.
So you're saying they're giving away too much good free stuff and need to be stopped?
My mistake. I had forgotten that Google is a taxpayer funded organization.
Why the hell not? Another company is free to make a better streaming platform, better map service, better browser, better email service... The government has no business telling companies which (legal) products they can or cannot create and own.
Just goes to show how easy populism is to push. Google and Facebook are in the news a lot and used by the average consumer, making them the perfect scapegoats.
Meanwhile, the average consumer just sees Samsung as the TV people (actual industries: apparel, automotive, chemicals, consumer electronics, electronic components, medical equipment, semiconductors, solid state drives, DRAM, ships, telecommunications equipment, home appliances) or Berkshire Hathaway as that old rich guy that drives a cheap car (actual industries (property & casualty insurance, Utilities, Restaurants, Food processing, Aerospace, Toys, Media, Automotive, Sporting goods, Consumer products, Internet, Real estate), so not a peep is made about them or every other horizontally distributed company in the world.
Samsung was also building military jets and artillery pieces. Imagine the media response if Facebook went into weapons production :)
The best way to fix the situation is to get users to pay. And this could be done. Users can pay negative dollars (e.g. get paid). Why would a search engine want to pay it's users? Why would a retailer want to pay me to look at their ad? Why wouldn't they?
Example:. My wife and I buy a not insignificant amount of stuff from LuLuLemon. So I have a deal for Lululemon. Pay DuckDuck half what you pay Google to show me your stuff. And DuckDuck, pay me half what they pay you. Right now DuckDuck gets zero. Google probably makes 20+ bucks a month off me. In this math, DuckDuck nets 5$!
There is a natural equalibrium here. Advertisers quickly start paying zero to people that don't buy stuff, so users can't farm ads. But relevant customers get discounts.
So now I actually care about making the trade off between second best but effectively cheaper or best but more expensive. E.g. a functioning market.
To do this though, you have to start tracking users in some way. But Google already does this, so the situation doesn't change. Some form of surveillance is required to improve relevance. The difference here is, I can choose who surveils me and to what extent rather than being stick with the winner take all.
So, somebody gotta do what Google did 20 years ago.
I'd personally do this to solve the problem:
1. Separate Google from Alphabet and separate Google Search from everything else they do.
2. Nationalise the Google brand and search engine along with Bing. Everyone else can stay as-is for now.
3. Allow applications for companies to become a search result endpoints. Ownership would need to be completely transparent. Prevent incumbents from being involved - Amazon, FB, Google, Apple, Microsoft, Samsung etc.
4. All Google and Bing result listings would then be evenly distributed across the search result endpoint providers.
5. The propositions for each of these companies can now be customised knowing that they will get a maximum number of likely searches per year.
6. Prevent any mergers or foreign ownership of these new companies.
Sure the search would develop at a slower pace but its power would devolve into utility again rather than as a way to control what people see.
If you want to enforce anti-trust actions against google's app store policies, I'm all for that. But forcing them to give away property is insane.
The author references precedent in the 1956 consent decree with Bell Labs [1]. If anything, that was more extreme. AT&T developed those patents.
The author's proposal doesn't involve Google surrendering its algorithm. Just the index it compiled from public resources using a publicly-subsidized Internet infrastructure. All without content owners' permission.
[1] https://economics.yale.edu/sites/default/files/how_antitrust...
This is a failed model. There is no incentive for Google to continue to update the public version, and it would quickly fall out of date while Google focuses on their own internal copy. It’s a solution suggested by lawmakers who fundamentally don’t understand how computers work.
How is this different from the 1956 consent decree? AT&T spent money developing its patents. But on the basis of longstanding law around public interest, it was forced to license them to third parties. (Note: not give them away.)
> and continues to dedicate to keep the index up to date?
The author explicitly contemplates, again within the context of the 1956 consent decree and many subsequent and preceding actions by the U.S. government (mostly around pipelines, et cetera), use fees paid to Google.
> There is no incentive for Google to continue to update the public version, and it would quickly fall out of date while Google focuses on their own internal copy
Google wouldn't be permitted to maintain dual states. Its algorithms would have to use the public index.
The article makes the conceit of equating an online data store to the telephone infrastructure. Google is not preventing other companies from indexing the internet in any way, shape or form. There is no barrier to another company building a similar data store from scratch using the existing internet infrastructure. On the other hand, laying out your own wires has physical limitations, especially when another company has claimed the most optimal route. In such a scenario, your costs will always exceed theirs (more infrastructure to reach the same consumers).
Regarding forcing Google to use the same index: How does DoJ plan to enforce this? How would DoJ ever be sure that Google is in compliance? I once again assert that the law magically assumes new technology to solve old technology problems without fundamentally understanding how computers work. Computers are copy-on-write by design. Delete always requires an extra step, and the verification that only one copy exists is impossible to make in the digital domain.
AT&T wasn't blocking anyone from laying interstate copper. It's just prohibitively expensive to do so. The analogy, down to the network effects, is quite apt.
Google can't have a monopoly on data gathered from the publicly available internet, that's absurd.
This is a rare instance where anti-corporate leftists and dejected right wingers could actually do something substantive together.
In most cases I would agree with you the defenders of private industry are pretty fierce in the US, but all that ideology sort of dissipates when you feel like you are being oppressed (whether or not, or to what degree it is true).
I think we're heading towards political ideologies having the same protection as religious institutions (which IMO, are exactly the same in all practical matters).
The government has never had the right to meddle in what publishers decide to publish (with exceptions for regulations on pornography and classified information). I can't imagine the government forcing Mother Jones or The Nation to print conservative viewpoints. If the platforms become "publishers" their power to select what information is available increases. They become liable for more of the content on their sites, but they also gain full control over content.
The only way I see to protect freedom of access to information would be to declare certain spaces (Facebook, Reddit, YouTube, etc.) as "privately-owned public forums" where suppression based on ideology would be heavily restricted.
For instance, if you're a newspaper, and you publish "Politician X is a rapist" and you don't have a way to back up that allegation in court you're liable. As a platform, you're not liable for the content your users post.
If you are allowing people to say "Politician X is a rapist" but not allowing people to say "Politician Y is a rapist", you've crossed into the land of publisher in some people's minds.
Simply being a platform on the web is 'doing business' if you're getting revenue from that operation. Can a platform specifically reject a religious group from their site? Can a platform demonetize a religious group?
I personally don't believe in protected groups, but if we're going to have protected groups, we should apply the same protection to all groups, equally, even if the group size is 1.
Liable is incredibly hard to prove in court, and usually cases are taken in civil court not criminal court.
Can a platform ban specific groups? I see where your argument is coming from. Commerce is allowed by the Constitution for the common good (to serve everyone). Could Walmart, for example, say that conservatives are not allowed to shop there? Definitely not.
However, it has recently been established, through a series of court cases culminating in the Gay Wedding Cake case, that you cannot compel speech from an individual that they don't believe in. Corporations in the US are often given rights similar to individuals. I can see this going to the Supreme Court under a similar argument with Google, etc. (you can't compel us to disseminate what we deem as immoral, anymore than you can make a Christian baker bake a gay wedding cake).
Ironic because leftists were livid with that decision, but it may be the one that allows private companies to censor you on their networks.
> Liable is incredibly hard to prove in court, and usually cases are taken in civil court not criminal court.
Going to court is expensive, even if you win. Thus, the protections for platforms allow an avenue of easy dismissal of cases, preventing from go very far in the court system in the first place.
> Google, etc. (you can't compel us to disseminate what we deem as immoral, anymore than you can make a Christian baker bake a gay wedding cake).
Sure, but I think then they would qualify as a publisher and not a platform according to some. Compare the results from Google to Bing for the query 'Trump is a'. Did Google hand-tune their algorithm for their results? Did they tune their algorithm to weight more heavily organizations that share their political bias? EG, Prefer CNN over Fox News (or vice versa). Do some sites have insider knowledge to better game the system?
I can remember a time when there were no news results in a google search. It was a better internet.
This is commonly repeated but not true. In fact, the opposite is true.
https://www.law.cornell.edu/uscode/text/47/230
Scroll down to section C.
1. No provider or user of an interactive computer service shall be treated as the publisher or speaker of any information provided by another information content provider.
2. No provider or user of an interactive computer service shall be held liable on account of—
2.a any action voluntarily taken in good faith to restrict access to or availability of material that the provider or user considers to be obscene, lewd, lascivious, filthy, excessively violent, harassing, or otherwise objectionable, whether or not such material is constitutionally protected; or
2.b any action taken to enable or make available to information content providers or others the technical means to restrict access to material described in paragraph (1).[1]What is 'good faith' and 'otherwise objectionable' in this context? Is removing all content from a religious group afforded in this definition, as long as the provider considers it objectionable?
'whether or not such material is constitutionally protected' has almost no meaning because almost all speech is protected.
Another interesting point is, while it's allowable by the law to restrict content you deem objectionable, is it allowable to promote content you prefer over other content? Or de-promote the content but not remove it?
Is it acceptable to remove religious content with no explanation as to why? Is it acceptable to ban or demonetize a religious group without explanation?
> is it allowable to promote content you prefer over other content? Or de-promote the content but not remove it?
Legally it is allowable since the law makes no distinction between "promotion" and "curation".
also this didn't set well with me
"Access. There might have to be limits on who can access the API. We might not want every high school hacker to be able to build his or her own search platform. On the other hand, imagine thousands of Mark Zuckerbergs battling each other to find better ways of organizing the world’s information."
Given that it has previous culprit of that exact same charge was gay marriage it is clearly a meaningless "family values" style euphemism to try to make their complaints seem valid.
Web index, while important, is by no means most important part of search engine (which I had say is relevance). The OP article is written by someone who has little clue that search engine involves massive amount of technologies besides index and even relevance - everything from spell correction to recommendations to query rewriting to answers to segment specific searches like images/video/maps/local/product/news/entertainment/events, so on and on and on. This is above and beyond the gigantic infrastructure needed to run all these at scale, speed and cost effectively. All of these needs to work harmoniously with each other and designed keeping in mind weaknesses and strength of each component. For example, index for news must be refreshed almost in real time and relevance needs higher emphasis on location.
Search is not free and no one has figured out if someone just can provide index and someone else can provide relevance and everything can just work as efficiently as before (data point: each query consumes 0.3 Watt-hour of energy at Google).
Even if we got Google's data, there's a whole lot of scraping to transform irregular and disparate. Typically you would have to google a keyword, look through the search results, visit different websites (research mode), and then consolidate separate sources of truths to build your own understanding. Building a scraper for each website in the search results and displaying a complete table data of the website. Why click through 100s of pages of profiles or data when you can have it all in one view? Why bother with HTML, when all a data-centric individual desires is data. Getting to the data is so tedious and a long journey. Scrape the website, clean the data, make it available for consumption, schedule & consolidate updates. For instance, a hedge fund that scrapes certain group of websites to execute market orders for an automated trading system.
A tabular data focused search engine would return tabular data, there would be no HTML medium, just straight up raw data. For instance, instead of seeing the comments rendered in a normal browser, imagine a tabular data that describes all the username, post time, comment minus the hierarchy.
To build a focused crawler quickly, I came up with Web Scraping Language (https://scrapeit.netlify.com), and essentially what I want to do is hire people to write WSL to scrape the web and then sell a subscription to data-centric customers.
What do ya say HN?
Partially bull__it - even forgetting about the $ that anybody would have to pay to Google for I/O traffic to scan its index (on which Google would put its % of fair earnings for at least the I/O generated plus maybe the "discovery" service for new URLs) that way any competing search engine would only have access to what Google decides that is worth indexing => OK from the point of view of the competing market (more search engines can rely on the same source data and potentially extrapolate different results) but in many cases NOK for the end-users as in all search engines that would use such service only the data that Google would decide that is worth to be indexed could be used (therefore a "closed bubble" set by Google).
If I were Google this proposal would probably be OK (more $ earned to cover the fixed costs of indexing by selling I/O to outsiders and at the same time it would lower the pressure/attention from regulators).
On the other hand non-indexed informations like all what comes from the Android OS (e.g. position of users, apps used by whom&when&etc), usage and actions on web-pages tracked by g-analytics, DNS access (e.g. 8.8.8.8), and etc... would be excluded - these are nowadays probably the most lucrative sources of infos of "what's happening now" + "what are people doing now where" which in turn generate the best answers to what "most" users want to know, which is mostly about "now" and their regional area.
1. The definition of the word "monopoly" is not applicable to this situation.
2. Property expropriation reduces the competition within the society, while also being unethical.
On top of this, Google search is as good as it is thanks in no small part to business practices that make Google as a company unsavory. Tracking your browsing habits, reading your emails, and watching your geographical location all feed the search algorithm, providing more seemingly prescient search results. For people who are OK with this, I don't see how an alternative can show up and compete with the years of data Google has already collected on a given user.
Toppling the Google monopoly is going to be more about a cultural shift than just giving competitors the data and algorithms they need to become viable. Users need to be made more aware of how Google uses their data, then alternatives can pop up marketing themselves on how they do (or don't) leverage user data. I'm skeptical that enough users will actually care to make this difference, at least in the short term.
Through hard work and good strategy they have achieved a dominant position, and I think that as a society we should both honor that and recognize that they have been pretty good players in the market. At the same time, if Google were instead run by the Koch brothers we could be in dire straights, and so we should also realize that we can't safely leave such power in the hands of private citizens without demanding accountability, transparency, openness to scrutiny, open dialogue, etc.
It's close. It's as close to a monopoly as MS was with IE built into Windows 98 - while you can find and use alternatives, it's on you to do this, and Google is the default all over the place.
Remember, too, how much power this gives Google - the top five search results for a keyword are intensely powerful, for everything from directing commerce to a particular site, to controlling news narratives, and more. Companies pay millions for SEO "professionals" and software like Moz / Stat to ensure they're in the top rankings. The latter in turn need to defeat google's anti-bot stuff to get search results, so they're in their own arms race...
The whole ecosystem is dirty and gross and gives too much power to too few people with too much money. It'd be fine if the average consumer was conscious of this and could reasonably make the choice to use a different engine, but this is highly controlled as well - think of the Google donations to Mozilla that they may not be able to live without.
Anybody can write a web-crawler and get a searchable indexed snapshot of the web. What Google is good at, is ranking your search results based on relevance and quality so the best results are at the top and you never see the terrible results.
Today the picture is completely different. Google, Amazon, Facebook and Apple are global - their infrastructure is virtual - not physical lines and cables. If a government wants to cause harm to one of those companies, they can decide to "move" somewhere else.
In addition GAFA is capturing dollars from everywhere in the world and bringing it to the US. It's in the US governments interest to tread softly here. No matter how the media spins things, GAFA is great for the US economy.
First googles competitive edge was never from its index. It was from its ability to give good matches to questions.
Second google being forced to share information like this might be considered a government taking of google property. Which would require a massive amount of compensation.
Third this has a condition that google isn’t allowed to remove anything from its index. However this flies in the face of the right to be forgotten, copyright law, child exploitation material law etc.
Fourth it says google must allow people to add to the index. Which is just inviting the index to be spammed with garbage.
How do these nominal fees hold a candle to the advertising dollars they rake in? Their dominance in advertising fuels their R&D to keep their algorithms superior. They have to maintain their superiority to fuel to keep the advertising dollars flowing. If suddenly anyone can offer Google results then a lot of that ad money dries up. Then the R&D money dries up.
So, say, people could use Google+ (if it was sticking around) to see Facebook content without giving data to Facebook.
Not sure how that would work with Twitter, I guess they'd just show media and clips of text posts.
Honestly I think I'd love it - just choose your favourite social network and only use one.
Up to each network to implement an interface that makes sense.
Would you want to use a library that filters its books to push an ideology? Would you want to listen to the radio if they only pushed top 40? Would you want to have a doctoral thesis that needs corporate approval? Research books that exclude facts based on corporate profits?
All three are terrible at serving me up websites to buy crap when I typically want to learn about something. I would love a filter that says "Don't try to sell me anything"!
I believe Google's biggest advantage is in their marketing & that the word "Google" now is a synonym for search.
In the future, when coming up with an opinion about something, I'd encourage you to look at statistics and combine that with your own personal experience. You'll often learn something you didn't know, and come up with a more grounded opinion.
Also asking for personal opinions will give me much different results than looking up a blind test research as brand loyalty will play a big part in ones like Duck Duck Go.
The real web crawlers are people who surf the web, Google likely takes information from analytics and outbound link tracking to determine the most popular (not the most informational) website.
Alternatively we could just publicly fund a crawler. A project like Common Crawl, just bigger and better, with some simple indexes prebuilt.
A classic antitrust breakup might.
- Google Search - no account required or offered.
- Doubleclick - ads on third party sites, not including Google sites.
- Alphabet Services - mail, docs, etc. - anything that requires a login or pay.
- Alphabet Heavy Industries - cloud, large scale business services, etc.
We don't exactly know how google indexes or ranks pages. It might be a completely different way of looking at indexing, it might be optimized for google's use cases such that we don't even have the software or the hardware to instantiate and search through it.
Let other indexes such as DuckDuckGo win through the market.
Search is virtually the only industry that Microsoft operate in where they aren't the Monopoly or biggest player.
Searching with an API is the same as searching from the browser. how the results are going to be sorted: garbage will be garbage, and should be demoted both in the API and HTML version. And of course, downloading the index is out of question.
I think that a better way to improve competition would be reducing cloud computing costs (bandwidth in particular). But of course, regulators in USA would say that it is communism or something... Sigh.
Second, that is a terrible idea. It would be a fucking crisis if Google's entire index was made public. I don't think the author understands the scope and nature of what Google indexes, as well as the protections Google has implemented.
Candidates with $Bs to spare (!) are Apple, Facebook, Microsoft. First two never started it, Bing is quite stupid so far. My default SE in Bing. I wish it was Apple :)
I've only tested with 1000 records, but the query times are all <200ms.
In short, 200ms is slow, not fast.
EDIT: Just to clarify, I'm not saying it's necessarily easy to be faster than 200ms (I'd need to look closer into fuzzy text searching algorithms to be able to make an educated guess here); I'm just saying it's not fast enough.
Basically it moves control to the client; if the client wants to query servers across the globe there's nothing stopping them, but by default you only query servers in your config file. Organizations like the new york times, universities, and governments would host servers for the same reason they host their own email infrastructure.
It also moves a ton of control to publishers; dindex doesn't crawl the web unless a client explicitly wants to.
The index take a lot of disk, memory, computing power, bandwidth... And then you have to handle attacks, spam, dead websites and not going bankrupt in the process.
I don't know you, but my best toy website is only capable of serving 100k reqs/second in localhost if serving pong...