Hacker News Reader

Comment by duxup | Hacker News Reader

Everyone wants to close down their corner of the internet because they think AI is going to make them a ton of money. We're getting the first part but I'm not sure we're seeing the latter ... anywhere as far as platforms go.

View on HN

Well, Reddit is getting a ton of money out of licensing deals for using their data to train AI.

Whether you classify that as "AI-related" or not, I don't know.

It's funny/interesting/terrifying to me that developers went from the near-religious mantra of "Garbage In, Garbage Out" when I was learning computers - to now training our supposedly super intelligent AIs off of reddit posts or even worse.

Basically laundering outright wrong information into something the next generation is now going to believe as scientific truths.

I often wonder how many people/organizations are seeding places like Reddit with malinformation/beliefs for it to become canical truth in the AI age once it's too late to tell the difference for most people?

Lord knows I've made trolling-level posts that are only marginally accurate back in the day that are now part of the AI corpus of knowledge. Mix those in with some of my well-researched stuff and you couldn't even really filter it based on "this account is a shitposter" to weight it lower. Nevermind plenty of earnest posts made that were outright wrong simply due to... being wrong in the moment and later learning better.

I think filtering on upvotes/downvotes, comments/views, sub, user and whatever metrics they have on the content can help AI companies train on somewhat reasonable things. Blend it with Wikipedia, scientific papers, reliable newspapers and you're golden?

Metadata is what makes gold out of poo, I assume model developers can "train negatively" too if metadata suggests they should.

So now the people who can buy upvotes get to write history?

True, once you start measuring by something it becomes an useless metric so it'd work until people knew if they do it or not, then they will not be able to rely on that data either. Or they improve their bot detection and mitigation and play the cat and mouse game.

Reddit sucks, but it’s also one of the biggest goldmines of human-curated information out there. Alternatives include blogspam, which is worse than useless these days, and forums with limited scope. Figuring out how to sift through dirt to find the nuggets of gold is important for any AI, whether they train on Reddit or not.

It's always funny to me how Reddit is one of the less garbage sources on random unexpected shit you one day want to find out. If you're really lucky, someone will have written a really good article on some niche topic on their personal website or similar, but that's annoyingly rare. You've also got review sites and such for common consumer products which are usually a decent bet after some filtering and cross checking. But the step down from that in terms of quality really is reddit of all places. You sill need to double check be sane about what you read, but the alternatives, like the ones you've listed, are actually worse than reddit's random internet strangers.

I don't think really good blog posts/articles on a personal site are rare. But they are increasingly harder to find, searches only returning SEO spam sites instead.

Every company who pays is a chump. They ought to get better at scraping and hacking IOT devices as residential proxies.

It's not entirely self enriching. AI crawlers hit servers hard and everyone has their own crawler. So it's partially covering a business expense. Especially with Reddit being a goldmine of content for training data.

Internet Archive has been terrible as capturing full pages on Reddit for a while. So it's not a real loss. Unfortunately right now these AI companies have full freedom to do whatever they want. Taking paid content, artistic works, and your own posts on social media. So Reddit trying to charge them is a good idea as it's some form of quid pro quo put on AI scraping companies.

> So Reddit trying to charge them is a good idea as it's some form of quid pro quo put on AI scraping companies.

Except that what Reddit is really doing is selling content they didn't produce and don't own. I don't think they're walking some kind of high road here like they would be if they were actually fighting against the scraping.

I didn't mean to imply they were. As I said, it's not ENTIRELY for self-enriching reasons. As in self-enrichment is a part of the reasons for this effort to combat AI scrapers.

That being said I can still take some satisfaction in seeing AI scapers get jammed up considering how they face zero consequences right now.

Reply on news.ycombinator.com