Hacker News Reader

Comment by paxys | Hacker News Reader

All the information Gemini surfaced was created with human effort and published on the internet with the expectation that humans would visit the website and the creator would get some reward - advertising dollars, bragging rights, popularity, subscribers or whatever else.

If the only visitors to websites are now LLM training bots then what incentive is there to publish anything new? For how long can we continue to rely on pre-2024 non-AI generated content?

View on HN

I've seen websites put up some draconian measures to try and get a grip on the scraping. So much for the sub-second loading experience when you have Cloudflare, Google, Anubis, and all these other captcha services trying to see if you're a human. It's made the web browsing experience so much worse.

Some of the proposals to address this include charging bots for access to web resources, but they will also have repercussions for regular users. I don't see how you solve this cleanly.

Me, I'm just scraping the parts of the internet I like, toying with local LLMs… ready really to just shove off.

Cloudflare specifically has a block for LLM and AI training bots now.

Not sure of the effectiveness but it's there.

Minimal. I'm behind Cloudflare and 90% of the traffic is still scrapers. I don't think they're serious about the long tail.

I think the main thing Cloudflare is trying to do is block direct traffic from frontier labs and then start charging them for access. They might end up shooting themselves in the foot, as this simply empowers sketchy residential-proxy outfits to undercut Cloudflare and sell the data to labs for less.

I think the other thing they're trying to do is get most of the internet to send them all of their cleartext traffic. Expect in 2040 the PRISM2 docs will get leaked by some Eduardo Rainedon and we'll find out Cloudflare was the NSA all along.

It still only blocks "well-behaved" bots that have proper User-Agents and respect robots.txt, so it's largely pointless.

The problematic bots are all disguising themselves as Chrome and sending requests from millions of residential proxy IPs, and the only real solution to those is some sort of captcha or PoW page on first visit.

While they themselves announced an AI bot, the irony is palpable. In reality they just want to control who gets access to what, themselves excepted.

Sure - it sucks, unfortunately the alternative is the sites going away entirely. When the load from scraper bots is constantly knocking the site offline the choices are literally to allow it to remain inaccessible for much of the time, put up a layer of defenses with all the user-annoyance compromises that entails, or just give up and unpublish the site.

The alternative is simple.. Go dark. VPN tech is known from like 30 years. Pretty much everyone can use it (VPN providers). But instead using it to browse net, build VPN overlay networks of interest for people. Gaming networks, R&D networks, Retro Networks. People will peer to PoP and use resources. Bad actor? BAN it from network. You have control. This could be done in Internet, but big corpos and big money won the battle. Just wake F*ing up...

Continuing on your suggestion.

There could be open source tooling to create custom private "closednets", with

- trust ring mechanism to allow invitations, flagging, banning, and banning those that invite people who were banned

- the rules of the closednet

- search engine with opt-in scraping

- portal (remember the 80s?) with all the registered nodes, perhaps by service category such as public git repo hosts, web sites etc.

etc.

The first closednet could be Hacker News.

It doesn't have to be an IP-layer network. A website that you need to log in to view works just as well.

Nah, it needs to be IP. IP is well estabilished protocol, everything speak it. Once you set it up, you can use it whatever you like. Web pages, gaming service, IRC, Mail, P2P confereces, everything. Everyone will bring it own slice to the pie. You love networking, became PoP and peer and provide access. You just want content? Connect to closest PoP, get IP + DNSproxy and vioala.

The networking part exists, it's called DN42. But it's hard to set up. I don't see why you'd use this instead of an application layer password.

I know DN42, but this network is more oriented toward R&D and experimenting. Yeah, its not for your avarage Joe. But your avarage Joe can buy connection from VPN provider and use it, and so we can provide user friendly PoPs with minimal skills needed to setup, supporting different VPN software.

or you put a password on a website

Spoken like a true websurfer. You know, networking is not only about WWW.

You can also put passwords on things that aren't websites.

I think you'd struggle to keep LLM bots off the network unfortunately.

If it had any real value, anyway.

Small, truly private communities could be an interesting thing though.

How I can strugle to keep them off? To peer to network, you need to talk to human. Arrange L2 connection, assign IPs (only static). If its leaf node, we are done. If its another network, we need to form BGP connections to exchange routing.

Yeah, Network by Humans for Humans. Thats why Im not interested in all those IoT/Auto networks when you just connect and stuff automagically configure. It looks nice at first glance, but you loose control. F2F works way better in that matter, like RetroShare, but I never investigated it much.

I just used a VPN yesterday and the NY times blocked me because they think I look like a bot. It gave a couple possible reasons, one being "a bot was also using this IP address".

VPNs are great for torrenting but any serious website like an online bank or web email provider will turn you away. They claim it's for bots but really it because they only want customers they can track.

You're conflating a VPN as in a virtual private network, with a VPN as in Nord/Atlas/ExpressVPN, which use the same underlying technology in a different way.

> Cloudflare, Google, Anubis, and all these other captcha services trying to see if you're a human.

Yep. IMO, this is so far the biggest AI-inflicted damage to the web. A bit of anecdata - wikipedia (and all other wikimedia sites) are blocking my Firefox since about a week, with a "please respect our bot policy" message. Outright block, not even a captcha.

It took me a while to figure out they don't like me disabling some SSL ciphers, so now "JA4 browser fingerprint" is not matching user-agent. Funnily enough curl (what I would imagine a bot would use) pulls exact same URLs from exact same client IP, just fine.

Maybe the curl thing is because they are happy to let you do some light scraping. What they want to avoid is bots directly crawling the page interactively. No one seems to be blocking chatGPT when I promot it to use it's web search skill anyway.

Exactly. What’s problematic about comments like your parent is the absence of mid-to-long term thinking.

It’s like bragging about a new highly addictive psychedelic drug that a dealer gave you a taste of for free. The effects are awesome today, you feel so fun and free! Never mind that it’s destroying your body and that the dealer will eventually charge you or demand you pay in other ways, that’s a problem for another day. Weeee!

> If the only visitors to websites are now LLM training bots then what incentive is there to publish anything new?

I write because I have ideas I want to share, and whether that happens with LLMs as an intermediary isn't important to me.

The danger that's concerning people (rightly or wrongly) isn't that LLMs are going to be an intermediary to your website. It's that they'll be the only thing reading it. No one will ever read your post or know what you wrote. The only consumers will be LLMs, they'll train on a version that strips out you as the author (probably more due to expedience than any sort of malice; it's not like you're famous, are you?), and your idea might get embedded into a set of model weights somewhere. No human will see a byte of it.

Are you actually saying you'd be OK with that?

Not the OP, but I suspect no one goes to my website anyway. (I write nonetheless.)

I do! You invented the crayon picker in MacOS..?

Yes, I did do that thing a long time ago, ha ha.

There's massive differences between "my site isn't hugely popular, but I get some readers", "my site is up and findable but genuinely no one visits except scraping bots" and "it's available, people would visit if they knew it existed, but they're not being offered it, they're being offered bot distillations with no reference back".

The last one is the worry.

The middle is... where we all start.

The first is not a bad place to be, all things considered!

Yes, that would be fine. I write primarily communicate ideas, not for credit or fame.

Empirically, however, LLMs don't strip out the author: the big models know a lot about what I've written even with search disabled. Ex: https://claude.ai/share/8cbcdf88-a360-421a-8c06-ae7b7992e866

Sure they know about you if you ask, but generally they won't credit you if they cite an idea from their latent space that came from you.

That's fine! Humans who read my blog typically won't credit me if I help inspire them either.

(I was pointing out that "they'll train on a version that strips out you as the author" seems to be is incorrect about how training works)

It knows you, but it doesn't know me. Perhaps you are legitimately noteworthy enough to not worry about this!

My experience being on the searching end is that these things are terrible about attribution of where they find anything. Which has bad consequences not just for authorship, but for correctness (which is the usual reason I'm poking at them -- they're being wrong again). This makes a lot of sense when you consider the massive, massive compression that's got to occur during training, but it's still frustrating.

I tried Fable, which is the biggest model I have access to: https://claude.ai/share/ab39ee92-7801-4a0c-8d5d-e23d09b439c2

>> Without searching, what can you tell me about exmadscientist? Fine if you get this hallucinate or get things wrong, I just want to see what level of knowledge you have trained-in.

>Working purely from trained-in knowledge, with the caveat that this is exactly the kind of long-tail detail I might garble:

>"exmadscientist" rings a bell as a username on Hacker News. My impression is of a fairly prolific commenter who writes knowledgeable, detailed comments about electrical engineering and hardware topics — things like PCB design, electronics manufacturing, component selection, test equipment, and general "how things actually get built in industry" commentary. The comments I associate with that handle tend to be practical, opinionated, and grounded in real professional experience rather than hobbyist speculation.

>Beyond that, I don't have confident specifics: I couldn't tell you a real name, employer, location, or any particular famous comment or post they've made. It's also possible the same handle exists on Reddit or other forums and I'm blending impressions together.

>So the honest summary of my knowledge level: I recognize the username and have a general "HN hardware/EE person" association, but nothing more granular than that — and even the association could be a mild confabulation, since usernames like this sit right at the edge of what a model reliably retains.

(I don't know if you have a blog or otherwise write on the internet; I just asked it about your HN handle)

So I'll write a lot of falsehoods to poison the AI. Like the urban legend where the sky is blue. I'll say the sky is blue, AI will think it's true, and regurgitate it to unsuspecting users who will see that AI is completely unreliable.

There are different types of writing. If we depend on people writing because it's enjoyable at some level, we're going to lose writing that's important but also a bit tedious.

Of course! I do think we'd lose a lot of great writing if it went amateur-only. But my parent seemed to be saying the incentive would entirely disappear, so I wanted to give my perspective.

Your level of thinking is defined by ICD-10.

Yeah, authors don't want to be recognized as authors, they don't want any reward for their work, they don't want to amass pool of loyal readers, interact with them, etc.

All they want is for halucinating AI to take excerpts of their work and compile it with random sh!t.

GENIUS

Thanatic drive masquerading as transhumanist virtue signalling.

I mainly shared my projects for learning, discussion and bragging rights.

LLMs just use everything, generate similar code with no attribution and keep users from visiting, so no bragging rights or attention.

Worse, there are some PRs that seem fully generated ...

So i mostly stopped sharing and started pulling my old repos offline.

At this pace, i don't want to compete with a clone of myself in the future that will do my work for much cheaper.

> I write because I have ideas I want to share, and whether that happens with LLMs as an intermediary isn't important to me.

Sure. But you can see that for some people (myself included), writing for peers is part of the joy? And that if instead a megacorp places an opaque computer program between the author and the readers, that joy might be ruined?

My parent wrote "what incentive is there to publish anything new?" and I described my motivation, but of course people vary a lot in what drives them.

Well in the example above the manufacturer still has incentive to provide the manual's and guides that describe how to use their products, and if that is subsequently served by an LLM that's totally fine. The only sites that LLM's would have a negative effect on are those that are only hosting content for the ad views.

Manuals don't always well explain how to use their products with everybody else's products because there are too many to do that. But there are lots of people trying things out and might figure out the fine details on how to make various things work. They then publish these how-to pieces (which exist no where else) to the internet, or at least they used to when there were incentives to do so.

> are those that are only hosting content for the ad views.

I might never blog/publish code again amidst all this. I never had ads on my sites. I am not alone in this.

Those hosting content for ad views, or for fame or other personal gain.

Sites like Wikipedia or developer documentation pages which exist to distribute knowledge for its own sake don't have any reason to care whether that knowledge is consumed by a human or a computer being used by a human.

> "humans would visit the website and the creator would get some reward"

That expectation is a problem, has always been a problem, and Tim Berners Lee never mentioned anything about a reward structure when coming up with the WWW.

Your thinking too narrowly about the reward. Sometimes, it's just about the getting the knowledge out there that's motivating the creator, not anything tangible for themselves.

> Your thinking

Should be "You're thinking".

> Should be "You're thinking".

This should be "This should be 'You're thinking'." don't you think? Why bother correcting someone's grammar with a sentence fragment? You're just trading one mistake for another. I'm hoping someone finds a grammar error in my post, because continuing this would be hilarious.

> This should be "This should be 'You're thinking'." don't you think?

Reflexively, I think it should be more like ...

  javascript: `This should be "You're thinking".` ; 
  // to preserve the original character use and to avoid '...'...' parse foos
  // however `"...".` also possibly deserves a [sic] to critique the original 
  // i.e. ~grammar police say the period belongs within the quote marks, no?

... but then that's just me, in [my] quirks mode.

Maybe I'm just making silly misspellings like this on purpose, to prevent someone from fingerprinting me based on my writing style. :-)

> it's just about the getting the knowledge out there that's motivating the creator

In that case the creator should welcome AIs with open arms; a human reader will forget eventually, but the AI will preserve the knowledge forever.

"the AI will preserve the knowledge forever"

no, only some mangled form of it

AI will mutate the knowledge and eventually produce some mangled version of it, mixed in with ramblings from a random reddit post and half a paragraph from a copyrighted book that its fascist creators stole.

If "getting the knowledge out there" is the motivation then it shouldn't matter whether the knowledge gets filtered through an AI or not.

And yet, inexplicably it does.

Sometimes, unpredictably, and uncontrollably, the LLM outputs bullshit that reads uncannily close to the hard-won, carefully-phrased text the human originally wrote.

A lot of us hate that with a white-hot passion akin to the eye-melting intensity of an arc welder.

If you don't understand that, then yes, I expect you're quite happy with LLMs, and indeed find inexplicable the reactions of those who are emphatically not happy with genAI.

"inexplicably" was obviously a sarcasm.

Did the operators of HN mention anything about a reward structure for posting comments here? I'm sure you can see how that's still attractive to some.

I finally got the downvote powers yesterday, and I was irrationally pleased about that.

It's not even about votes, I don't even need votes. Whenever I write something that I am happy with, I read and reread it imagining I am reading it as a third person. Sometimes it forces me to rework my arguments. I wouldn't write to convey my ideas through a chatbot. And that's what this post is about-- killing the internet and with it decimating any audience you might have accrued if you had something to say and you published it on a website.

At least LLMs almost always transform the original - it usually isn’t as straightforward as “Here’s the original but without the ads that pay for it”.

But we already have the latter case that exists - ad blockers. Ad blockers literally serve up the word-for-word original content minus the ads.

Easy to fix a well documented router now. Difficult to fix a non documented router in five years time because no one has been contributing to the web about its bug fixes.

Gemini can just consume the device documents. There's an incentive for device makers to publish this content.

There had always been some incentive for manufactures to publish device documentation, and yet it has often been quite lacking either in quality or overall existence. I doubt LLM/agents being the readers will change that at all. What I expect AI scraping and using without credit will impact is people publishing their own unofficial help and guidance, and the affect there is likely to be negative. It won't stop all of them, but enough to be noticeable. Another possible negative is the manufactures documentation being AI generated without sufficient review, so possibly more erroneous than before, or intentionally not producing full documentation at all and expecting AI to fill the gap (MS seems to be heading this way: pushing "ask copilot" all over Azure instead of links direct to good reference material). All this would add up to a situation that is somewhere between "a little worse than pre-AI" and "an absolute shit show".

I wonder the same thing. I only imagine that what comes next is worse: AI companies using vast resources to develop new training data, in house, locked down. They are already doing this with developers and code at Meta. Information will become locked away behind AI paywalls and chatbots.

That is next earnings quarters problem is the approach being taken

Agreed on all of the above.

However, when it comes to non-creative things (such as hooking up some hardware ABC to some software XYZ using RST), LLMs might be better at digesting the factory manuals (hopefully THOSE were written by humans) and explaining it in a way the user understands for their specific case.

> incentive is there to publish anything new? For how long can we continue to rely on pre-2024 non-AI generated content?

obviously new content still has value because it remains the source layer for LLM agents. it just wont be ads giving you revenues thats all.

Well companies are starting to put hidden ads in text content if the user agent is an AI crawler

People write and create regardless of profit motive, it has been that way for thousands of years.

Actually no. The original copyright laws were created in part because of the realities of needed profit motive to have high value writing done, time consuming compilation work done. It was even titled "An Act for the Encouragement of Learning". The thousands of years writing you are talking about was often funded by patrons, who kept the output in their private libraries to show off (and maybe lend out) for prestige. It was a horrible limitation of knowledge and ideas. Much worse than the profit motive, copyright based system that came after that spawned a new age of knowledge in which everyone had cheap access, and those that didn't had access to the (no longer just private) libraries.

I'm sure the billionaire class would love a return to patronage based libraries, NDAs on authors of books, and the elitism they would feel with a return to private libraries locking away all kinds of knowledge that would happen if patronage become the only way authors could make money (such as with AI just regurgitating their works, or if the stupid 'do away with copyright' people got their way).

It's not copyright that caused cheap access. The printing machine allowed for cheaper publications, that's what spawned the new age of knowledge. The raw materials and the duplication of knowledge was the bottleneck. With digital systems this cost is minuscule, but still there.

Disagree. The printing machine was killing the industry allowing copies where people earned nothing. Copyright was created to ensure quality works existed to be copied.

That's the stated purpose. What's the POSIWID?

Actually it grew out of censorship and monopolies: https://en.wikipedia.org/wiki/Statute_of_Anne#Background

Cheap access came from the invention of cheap printing . The laws were passed to restrict it.

No, cheap books came from the invention of cheap printing. But the quality of works was going down because printers were just printing with zero copyright protections. Copyright is what brought the wealth of works worth reading that were then printed using cheap printing. If the only money is in private works for private libraries, that is where the quality stuff is going to go.

Making the avenue of creating for the average person also the avenue for the most income was huge in creating our modern literature landscape.

> Copyright is what brought the wealth of works worth reading that were then printed using cheap printing

Evidence for this statement?

> If the only money is in private works for private libraries, that is where the quality stuff is going to go.

Evidence that this ever happened?

> Making the avenue of creating for the average person also the avenue for the most income was huge in creating our modern literature landscape.

That only leads to higher quality (as you claim) if your definition of higher quality is "what the average person buys".

People are way less likely to write if there is no one to read it. And blog were also monkey see monley do - people seen other peoples blogs and got inspired.

When people wont see others blogs, they wont start writing own. When there will bw no ome to actually read it, they will go to do something else.

I had a flippant answer which was “who ever wanted to write for a machine in the past thousands of years”

But maybe that is the future.

Every country starts erecting their own towers of babel that we talk at, and it constantly compresses our conversations down to the most effective distribution of weights.

At some point talking at the machine becomes a high status job, and we give respect to the people who whisper to it the most.

They don't write for machines, they write for themselves.

Why write publicly then?

Theres many people I know who write notes that I know would be great to read. However they never publish them.

So… are we saying that the only public writing in the future is meant to be consumed by the machine?

There are other replies in this thread that say why. It doesn't matter if robots read them as long as some humans do too.

Funny, I published information in the hopes that humans would benefit from it. If it happens to be through collective intelligence of LLMs I'm ok with that--even more so if through open models.

> Funny, I published information in the hopes that humans would benefit from it.

Sure, humans would benefit.

It took them searching, reading themselves, maybe even understanding something in the process, to complete a 360° revolution of their squirrel cages in time T.

Now they can omit searching, skip reading to the regurgitated answer, throw away understanding, and complete a full revolution in T/N, where N is a heuristic value directly proportional to the amount of skin in the AI hype.

But the catch is that the squirrel cage must run non-stop still.

Reply on news.ycombinator.com