The death knell of digital search
newyorker.com
newyorker.com
Which is one thing technologists often screw up: They misinterpret how people react to change; and they misinterpret how people really use their products.
Google Search used to be (and still is) a way to find places on the internet. As much as I like AI in searches, I agree; when I want to find the thing, I don't want AI getting in the way.
The issue with someone like Google, or Microsoft, is that they have a larger moat in many products, so the amount of degradation that internal actors will be happy to push in search for profit is much deeper.
This is the reverse of my desires. Most pages on the current web contain a few useful nuggets buried in a shit sandwich of overly verbose text and aggressive advertising. AI extraction and summarization distills the mess to concentrate the useful bits.
It probably won't be huge but isn't that kind of the point?
Then, they essentially bankrolled their enemy by giving websites a way to monetize spammy content. By the late 2010s, they started losing the battle for many types of organic queries - although many HNers forget that the most common and "important" search queries are things like "Nike shoes", not "an implementation of a Bloom filter in Rust". And then, in the 2020s, they pioneered LLMs and made the technology public, and drove a final nail in that coffin by making it possible to produce SEO spam on any topic, including Bloom filters, practically for free.
Now, they see AI answers as their salvation, steamrolling a variety of non-monetizable use cases that never really mattered to them. And this will probably work only until AI-poisoning SEO is in full swing, at which point Google will find themselves with dissatisfied users and diminished ad revenue.
Up until around 2015 desktop traffic dominated, google pre 2015 tended to favour big comprehensive articles that covered not just the question being asked by the searcher but also everything surrounding it. Reading such content would inform you not only of the answer to the question you were asking but also would give you enough context to understand if you were even asking the right question to begin with.
To me this was an absolute golden age driven by the ability for website users to create good comprehensive content and monetise it via ads.
Post 2015 google shifted to prioritising content that directly answered the question being asked and little else. I assume because mobile users just wanted their answers which ultimately lead to the "snippets" that directly answered the query in google itself.
Google invented the tech for LLM's but did basically nothing with it until OpenAI forced their hands because they knew that it would kill the ad driven web. Once OpenAI did force their hands we end up with search as it is now.
Pre 2015 there was a point to creating good information on the web, now there isn't, it'll just be stolen by LLM's, there's literally no point beyond your own personal altruism.
Thankfully for google it owns youtube and there is still an incentive to upload useful information there, so that's increasingly where new information will live, but the web as we have known it most of our lives, is done.
Or for your own reference later. Or for the fun of it (some of us are that type of person!). But yeah, little else.
(The requirement for monetization on YT is 3k "qualified watch hours" per month. If you upload 5-minute videos, and 10% viewers make it to the end, you need hundreds of thousands of views.)
But now, it's worse than altruism. The work your doing doesn't directly benefit the masses, it benefits OpenAI & co. You're not doing charitable work, you're a total chump.
If you're someone providing information for free, because information wants to be free or whatever, then you should be happy when a copyright agnostic (ignorant?) AI company scoops it up for their next model. In a way, you're directly shaping the zeitgeist for everyone that uses those models.
You don't have to wait for that one lone person to stumble upon your site. Now everyone gets an answer from an LLM directly shaped in some small way by small changes to a few of a few billion weights.
The entire premise of this thread is that search is fucked in large part because there's no incentive to upload any information any more beyond altruism.
The LLM's can figure their own shit out, I'm not feeding them for free.
Of course YT creators can do whatever the hell they want. They don't have to listen to me.
This is not fully true. From a seo prespective, it is not possible to generate a lot of spammy AI content, optimzie it and then publish it and expect google to rank it. They already discourage spam and such clear spam is pretty easy to detect. Otherwise all keywords would be gamed by now since everyone can just make AI go brr. Being at the forefront-ish of AI, google surely saw this coming and prepared for it in their ranking algos.
I think you're being a bit biased in your analysis here and coming at this from a conclusion you already made.
- Eric Hoffer
I've been playing the Ocarina of Time remake and whenever I get stuck trying to get a certain item or progress on a quest, I'll fire up Claude voice chat and ask it for guidance, and it regularly gets confused and gives me wrong answers (somewhere around ~30% of the time).
Given how long these models have been out and that it's a highly constrained game world from almost 30 years ago which is documented top to bottom, I struggle to understand how Claude is still so bad at straightforward questions with straightforward answers. This was a solved problem (albeit with a less modern UX) in the legacy internet and it doesn't actually feel like getting the right answer is necessarily easier.
https://gamefaqs.gamespot.com/n64/197771-the-legend-of-zelda...
Google used to return these documents, or at least forums and communities who knew where these guides were. Google no longer connects communities together however.
LLMs don't either but the hope was that they at least read all of the old internet and have it somewhat memorized.
Claude is absolutely killing it for me. Man-years and man-years of backlog work, hobby projects, optimizations, research, experiments, finally implemented. Bloated apps finally optimized to fit on micro instances, slashing my infra costs. Recovery from operations incidents in a third of the time.
If I could make Claude forget everything it every knew about Ocarina of Time to make it 1% better at what it already does so well, I'd press that button in a heartbeat.
Claude should be an order of magnitude better than that. So, I can't say I understand how your experience has been so bad. Very surprising.
For example in the earlier days of the net lots of people just sucked with the search engines that existed while I found what I wanted quickly looking for in the same topic.
How you ask the question is still important, and will probably always be important.
I ended up building my own search instead - whoever controls your search results also controls how you find information and what you see, so I felt I needed to start building my own index. If you don't want to go down that rabbit hole and don't mind Kagi's politics, $10/month is a small price to pay to get a better search experience.
I started with a metasearch, since that's (mostly) all Kagi is. I send prepaid-API calls to Brave, Mojeek, Marginalia & Yep for every query, and get about 200 results back on every search. The cost is far less than I paid Kagi. My first version was a single PHP file written by Claude Sonnet 3.6 (!), any modern AI would one-shot something dramatically better. Obviously you can iterate & improve it over time yourself. Grouping the results by domain and ranking by how many search engines return the same page/domain will get you ~60% of the ranking quality.
Over the years I added more ranking signals and had Claude build a ranking optimizer. I provide "golden queries" ranked as I would prefer, and the optimizer runs experiments to optimize the weights for each ranking signal. It isn't perfect, especially since I only use on-page signals. But it has been good enough that I don't miss Google.
My own index is just a couple of SQLite databases. It is ridiculous how far you can get with SQLite, FTS Full Text Search & its built-in BM25. Each webpage takes up 10KB of index storage on average, once trimmed down to just the indexable text. I never clear my Firefox history, so I went through my history cache to find which sites I actually visit, then made a command line indexer to run on my laptop and index them in the mornings. Most sites offer a sitemap or RSS file, so you don't need to worry about crawling the site. My tiny index doesn't answer every query, and that's what the metasearch is for. But surprisingly often, that tiny subset of the internet does have the answer I need, and it saves me a paid search API fee every time it does.
My system is held together with duct-tape and hope, so for most people it's easier just to use Kagi (or Uruky.com). If you do want to build it out, the blog at marginalia.nu has far more useful technical details than anything I can offer. But if you do go down the rabbit hole, I think you'll start questioning how a giant enterprise like Google can manage to screw up search so much.
I treat my Firefox History the same way though the way I use it is a bit different right now. I would like to harness it like yours, but first I think I need to pipe it into ArchiveBox or something
Aside from the obvious larger issues with Gmail (privacy, trusting that Google won't just deactivate your account, etc), is there anything in the day-to-day usage of Fastmail that you find to be an improvement over Gmail, in the same way that Kagi is an improvement over Google search?
This massively improves your security.
You can keep the actual email that you use to log into fastmail completely secret and never use it anywhere which makes it harder for people to attack your fastmail account.
- Bring-your-own domain name with lots of useful features for aliases and routing messages around based on that
- Real human customer service, and with agents that actually want to help you to boot
- Everything follows common standards (IMAP, CalDAV, CardDAV), no need to deal with Google/Apple-specific weird stuff
- Masked email
- 60 GB storage per seat
- Actually usable and good web UI
- JMAP (which has stable mail ids--as opposed to IMAP which does NOT guarantee that the ids of the mails will be the same on the next connect! That's why I went to fastmail in the first place: attempted massive two-way sync corruption because IMAP just doesn't guarantee what I thought)
kagi, for example, has "Quick Answer" mode, which means if you append "?" to whatever your asking about, it will generate a summary (sort of an on-demand AI injection), which is great for the user experience (nobody is shoving AI down your throat).
but the problem remains: am i really going to scour web results, or am i going to throw in a "?" and get my answer in less than 3 seconds? i find myself using this more and more, despite deliberately trying not to, the convenience of is just addictive like that.
the result for web site operators is the same: my click never arrived, the ads they served didn't reach anyone, and they are disincentivized to even run a site.
i'm not sure what's going to happen, but it's not looking great. the web was meant to democratize information. the current generation of LLMs/search/chats are reversing this, siloing the truth within the same 5 oligarchs of data.
> The thought of paying a subscription for search is abhorrent
I've found myself paying for more and more products that I used to get for free. Given how expensive running a search engine is ad revenue is necessary for a free product. Advertisers pay more if they are getting clicks and conversions... Obviously this ends up impacting search results. Additionally, the tradition SEO model went from bad to worse with LLMs.
Anyways, up until very recently I would've agreed with you, but ultimately this was a pretty easy expense to justify after a short trial.
I've been a happy customer of Kagi since 2022, and for me it's well worth the money. It gets better results than Google or DuckDuckGo, and I trust the results more because they're not pay-to-play like Google.
Just hope that the index quality does not get further deprioritized... It might make sense to make a public indexes if corporations aren't interested in maintaining them...
I don't know if it's the language piece, the geography and routing, maybe something else in the backend user profile... Idk
There is no other search engine that seems to come remotely close for me for actually showing me what I'm looking for, and on the first page. I've tried dozens probably over the years based on random comments from people, but never find anything else close to the results quality Google search gives me.
I wish I knew for sure what the difference is! Maybe query verbage, search modifiers, etc?
Point is maybe it depends on topic maybe?
These days I use a couple different search sources both paid and free and it’s been that way for a couple years.
I am wondering if something in their user profile or identifier is tainting the results.
I've had people on this very website tell me they searched for XYZ and couldn't find it, so then out of curiosity I will try and replicate it.
I have flipped on a vpn, in incognito mode with tons of anti-fingerprinting tools, and the XYZ thing will be the first or second result on the page.
IDK, I just find it weird as hell and would love to know why.
> Maybe Google search is like an LLM - if you’re asking for a code reference it works, but if you’re asking for something else (music theory?) it just doesn’t?
Possibly.
Maybe it is the way they structure their query? I've been using it since release and the death of altavista and others, so I may not even be aware of how the two may differ. I have heard some say that natural language search was what they always /expected/ from google search, so maybe there is something to that.
It has been a game changer, for me, for constructing magic incantations, er shell command lines, for doing stuff with ffmpeg. In this case the AI answer is fine. Paste, does it work? Yes, great, done.
Since no one will find that, you might as well write for yourself. And if you are wroting for just yourself, you will keep it for yourself or just wont bother.
Which is significant anount of people who wrote. People who dont care about either are a minority of those who wrote.
I can ask an LLM to give me the link, even write a custom prompt that lets me type a search query…but why?
Moreover, LLMs still use search under the hood, do they not? So search won’t die. It can be bundled in the LLM subscription, or its own subscription like Kagi for the minority using local models.
If you append &udm=14 to the url you get a traditional search. That doesn't help with all the SEO spam though.
have been a paying user for years, very happy with the service.
"What time is the football game on tonight? People often ask what time the football game is on tonight, and to find out what time the football game is on tonight. Tonight the football game is on at a time when the football game is on tonight".
People started to use LLMs for search because they actually return the desired content in a form the user wants to read. I unfortunately actually like AI mode because of this even if i recognize relying on AI summaries for search has its problems.
You can only scale moderation so far, and non-neutrality exposes you to liability (not moderating something can be construed as permitting it)
If Google really wanted they could've let us self moderate / let us install crowdsource block lists .. but I'm betting they decided against, as it'd help people block ad-funded sites their monopoly depends on.
Now they're giving away their market position ;) halting all the things inside Google that led to good products. Thanks Google!
We haven't seen it manifest yet because alternatives need to emerge to jump to.
But Google is rolling over right as it gets easier for a small hungry team to build an amazing thing. Here comes a new era of great and useful stuff for us all.
It'll take a moment. Monopolies don't crumble overnight.
If you believe this is how it should go, then wherever you can, do your part. Help alternatives get so good that they stop being the "alternative".
Is this truly a value-add? Like will Reddit make more money doing this? If so, I weep for the future.
The internet is going to close up as content becomes increasingly valuable. The free ride ended a while ago with paywalls everywhere, but even social networks have raised shields.
It sounds like it's working at least with respect to this author
He seems to have no alternaative but to keep feeding training data to a chatbot/agent
I am still using traditional search, done from the command line in textmode
No browser, no Javascript, no cookies
There's even way to still get "10+ blue links" from Google, it's "hiding" in plain sight, unchanged since the mid 90's, though I expect this will be shut off eventually
"But I would really prefer if Search just worked the way it did in the first place."
It's possible but the user might have to stop using Google, Anthropic (Google is an investor), Spear Street Technology, Inc., and so on
I think this author like so many other "tech" journalists will go whichever way the wind blows, whilst making futile complaints into the ether
"I find myself using A.I. to fight against aggressive A.I."
Here is what one security researcher said about "Instinct"
https://www.businessinsider.com/former-meta-security-chief-a...
In the past few months though, I’ve been heavily using the assistant set to use deepseek to assist with putting together a Minecraft 26.2 mod pack. It’s saved me so much time and frustration it’s unreal. Turns out there are certain things that are complex enough that regular search really isn’t all that great for.
For example, I can now do things like ask if there’s a mod that allows torch placement from the hotbar while holding something else, via middle click, compatible with Minecraft Java 26.2 neoforge, as the one I previously used is stuck on 1.21.1. And instead of spending 2-3 hours trying a bunch of keywords and combing through poorly named and documented mods, I can quickly get fairly reasonable confirmation that this doesn’t exist, along with some alternatives.
I can also hand it a GitHub source and asking it what needs to get changed to update a mod to newer versions, giving me an idea if doing so will get within my extremely limited coding capability. A real game changer is basically asking it for a tutorial, like what’s involved with fixing a model from an outdated resource pack for a recent version and having it work with Fresh Animations. This would previously have required many days of painful and frustrating searching.
I can also ask it for basic scripts, rather than spending time hoping to find something someone else already made. Such as taking ratings from my navidrome database, and writing them into tags in the related songs. Just the other day I asked it for a violentmonkey script that can take a Reddit comment with a video, and give me a url that yt-dlp can handle as is without having to dig through a bunch of nonsense manually. I would have never have had the expertise or time to learn to implement these things myself.
Of course, one must keep in mind that the knowledge the llm has has a time cutoff, but I’m frankly have so much of a better time than before.
I can now see myself using the next update of Claude to generate mods (the current version does not have much knowledge of Minecraft 26.2) with features I want. While I’m reasonably competent technically, actual coding is beyond me, and is far beyond my free time to learn.
While AI doesn’t replace my use of classic search wholesale, it does take a set of problems I was trying to poorly solve with classic search, and actually gives me what I want.