Bluesky's open API means anyone can scrape your data for AI training | Hacker News Reader

Bluesky's open API means anyone can scrape your data for AI training

techcrunch.com

57 pointsbookofjoe72 comments

Good? I'd rather it be available to everyone than just a handful of large corporations.

Then what's the point for those moving from X to Bluesky?

AFAIK many are moving due to content might be used to train AI models.

I don't know where you got that idea. People are leaving because they don't like Elon.

Very few people are switching to Bluesky so they don't have their data scraped for AI models. X has become a messy cesspool of spammy ads, toxic comments, and an owner intent on making it his personal echo chamber.

"They'll train an AI on my deeply-valuable thoughts" is the absolute least of my worries about having an account on Twitter.

It's more just about me not wanting to stay in the place I grew up after all my friends moved away, especially after a whole bunch of slightly terrible people moved in next door.

Crazy part is this researcher is a part of the Bluesky community, he thought he was giving back - this stuff is great for moderation purposes, spam rejection, crypto scam detection, and just fun "maps of supercluster" projects [0].

[0] https://nucleardiner.wordpress.com/2023/05/21/bluesky-is-for...

I feel like there's a fundamental disconnect between the hacker ethos of "everything online is fair game, let's scrape a ton of data and see what cool stuff we can do" and the more...let's say European ideas about data and privacy rights. Neither one is inherently right or wrong, but they both cannot coexist on the internet.

Here come the flurry of scary articles looking to find that angle. Can’t call this one a foreign actor so now openness is “bad”.

There is literally no one that can be pleased anymore.

try drawing a straight line through 8+ billion scattered dots.

The article takes that angle because Bluesky users are currently harassing and threatening a researcher. Read the replies to his original thread: https://bsky.app/profile/danielvanstrien.bsky.social/post/3l...

From the article:

>Per a report by 404 Media, Daniel van Strien, a machine learning librarian at AI firm Hugging Face, pulled 1 million public posts from Bluesky via its Firehose API for machine learning research, pushing the dataset to a public repository. Van Strien later removed the data due to the controversy that ensued; however, it serves as a timely reminder that everything you post publicly to Bluesky is, well, public.

So what was the license on the data?

HF certainly have a restrictive license on data they produce.

See also the comments where Daniel apologizes. Bluesky hasnt changed anything about the shape of Twitter and the kind of harassment that occurs. A mob of people have some vague enemy (AI bros) and pile on the first target of opportunity directing all their "kill yourself" energy at one individual at a time.

https://bsky.app/profile/danielvanstrien.bsky.social/post/3l...

> hey man i don't know if anyone's told you yet but nobody likes you, you stink, your ideas are stupid, ai is loser shit, and you should go fuck yourself

> Get off this website, you fucking ghouls.

> Just go back to Twitter you Ai trash. Create your own work and stop trying to make money off of everyone else’s. No one wants you here.

> You're not sorry, you lying sack of shit. Get the fuck off this site. Better yet, get off the internet entirely, go live in the mountains and be a hermit for the next 50 years, you rancid fucking malignance.

> I’m using generative AI to make a rendering of you in a mass grave.

etc.

Keep in mind this the post where he takes it down.

Bluesky themselves have made it abundantly clear they're with the artists on this one. The firehose isn't a license to use someone's art for commercial software development.

The hate doesn't prevent the research taking place, but ensures it won't occur publicly where it's more useful.

Is that worse than Meta, X or Google hoarding up for themselves?

Or is it worse than Reddit selling to anyone with money?

Maybe someone is upset that they can’t get competitive advantage by throwing money.

Maybe it's just me, or I might be missing a relevant implication - but I'm having a hard time understanding why so many people have become alarmist about the fact, that things that they publish on the web, can and will be scraped?

It seems to be mainly a reaction against AI (as opposed to scraping in-general, e.g. for a search engine).

I'm not saying it makes sense, but there is a large and growing idea of: I want my content out in the world, but I don't want companies to use it for training AIs, especially for profit.

Then why use a private company platform with their terms of use agreement?

It's like going to butchery and asking for veagn food.

Might just be one of these: https://en.wikipedia.org/wiki/Availability_cascade

I do find PG's idea of "aggressively conventional-minded" people to be a useful concept: https://paulgraham.com/conformism.html

I think it has to do with the previous article where blue sky promised not to train ai on your data. It’s like they won’t do it but everyone else will.

It's mainly artists that got harmed economically and some/many of them have been pushing this line that scraping and training models on scraped data is illegal or unethical and got neutral left leaning people to agree with them on it.

People got angry on bluesky and say it has to be forbidden; you either want open or closed. If closed, your data is still used to train AI, but the owner of the network is making money with it. Better if anyone can get it; not like it is stoppable anymore anyway.

There's a lot that could be said for the behavior of Bluesky users and moderators, but aside from the practical matter that anyone can scrape your data, there seems to be some confusion on users' part about what it means that they have "ownership" of their data. You license it to Bluesky, but in the absence of other licensing agreements (or as Gen Z likes to say, "communication of consent preferences"), is there no way to prevent what you do in public from being absorbed by the facehuggers and monetized thereafter?

A book's open text interface means anyone can scrape your data for AI training.

An open and accessible API is good actually. "Bad" actors will always find a way around closed/restricted/no API access.

Does Bluesky's decentralised nature make it hard/impossible to apply a bot blocker^ like cloudflare?

^ More accurately, cloudflare is a bot-slower as it (and services like it) make it slower and most costly to scrape data, but not impossible.

Each relay could implement bot rejection as a means of saving bandwidth but the whole point of the architecture is that the firehouse can be mirrored

2 million bluesky posts if you want to use it for something:

https://x.com/AlpinDale/status/1861819574259192082

That's a very depressing and unpleasant-looking dataset. I'm don't know if that reflects more on BlueSky, or on the "uniform random sampling" aspect.

If we sampled HN data randomly, would *we* look that bad?

You can try it out here: https://huggingface.co/datasets/nixiesearch/hackernews-comme...

The first comment I see is Spez saying "winner winner chicken dinner"? Haha.

Associated torrent magnet link: https://hastebin.skyra.pw/raw/iqenuliqaz

As soon as someone figures out how to DCMA the AI industry the whole industry will become enshittified. It all relies on copyright infringement and “generative” AI doesn’t generate as much as it originally promised, it’s more like an extremely advanced search engine with the ability to combine and edit the source data.

An analogy is music producers who sample other tracks, who most definitely have to pay royalties.

If it was as easy to detect trained AI data as it is to detect a music video or movie in a YouTube video, every AI company would be toast.

You’d basically have a category of application whose cost is so high it’s hard to justify. You’ve gotta run the world’s most expensive type of computing to do the AI training and you have to license a massive amount of work from copyright owners for it to have any use.

To swing this back to being more related to the article at hand, I think that being open to the public can be okay if BlueSky or people who publish content on the platform are better able to exert their rights under copyright law. When I post something online I shouldn’t be giving up my copyright rights just because it’s hard to enforce.

If there was a law that was truly progressive about online privacy it would protect individuals’ intellectual property rights more on social networks. A social media company shouldn’t magically get to own my content just because they said so in their EULA.

That's not how the scaling laws work. The number of samples required to reach a given quality level reduces exponentially over time. Most researchers use small datasets.

Interesting, because in The New York Times' lawsuit there is a very large block of text repeated verbatim. Page 30: https://nytco-assets.nytimes.com/2023/12/NYT_Complaint_Dec20...

How much of a copyrighted work do I have to copy and reproduce/redistribute to violate copyright law? Am I allowed to sell my handheld recording of the last two minutes of Gladiator 2 for $1.99 at the flea market?

That has nothing to do with my comment. My point is that generative ai models can achieve good perf without using large datasets

In theory. In practice, it’s a plagiarism machine.

If they didn’t need the large dataset why did they use it?

And honestly I think you need to provide more proof of your assertion since it’s so far off of the status quo lived experience of AI. I ask ChatGPT about product specifications and other specific search engine-like queries like that and I get accurate answers. If there’s no big dataset where is that information coming from? Nothing is being generated, I’m getting regurgitated product specs and reviews when I ask for them.

The default model can search the internet just like Google

And it’s allowed to reproduce the content of copyrighted articles?

Because non-AI Google search doesn’t do that. Search engines produce a list of links with, at most, a short snapshot of content.

You don't have to give your data to social media companies, you know. That's just part of the trade off in using their apps.

A lot of creative professionals work on commission. Social media is basically a requirement if you want to make money in certain artistic spaces. If you can't show your work in public you won't get jobs. Catch 22.

Me making a comment on social media: anyone can can redistribute it, they could even charge money to read my comment if they'd like.

Disney playing Frozen 2 in the airplane in-flight entertainment system: I'm not allowed to copy, reproduce, distribute, or disseminate it.

See the double standard here?

Good. Just like the open web.

That way they can claim it's not just their scabbing up your data but it's a design feature in case you try to sue us to resell. For the betterment of humanity and such.

Does bsky have an actual plan for how they'll make money long term?

The users are all in a honeymoon phase about how it's so different from twitter, but it seems like it's only a matter of time until they're directly selling data for AI training, offering paid corporate accounts, intrusive ads, etc. In the end, I imagine it'll end up just like twitter but with a different CEO. I use it simply because lots of others who I follow migrated, but I'm under no delusion that it'll be any different long term.

Well they recently published the AT protocol that they use.

It’s an open protocol, so they’re going for some kind of community angle maybe?

They could probably sell instances like mastodon, but I don’t think that will scale how they probably want.

I think they will eventually find a way to either monetize the data or add an ad extension to AT proto.

Long long term is anyone's guess, but I think if the team stays mission driven and doesn't get distracted they won't have any problems with money. The company is still owned by employees, and the entire team is just 20 people. They don't seem to have a bloated stack or unnecessary features. They got a sizable initial war chest of $14 million from Twitter. When that runs out I bet adding just a "donate" button somewhere on the site will be enough to cover expenses.

Already raised another 15 million from Blockchain Capital. Not a donation, but a Series A. They could have walked the nonprofit donation route like Signal but they dance with VCs instead.

https://www.blockchaincapital.com/blog/bluesky-13m-users-and...

14 million pays for a couple salaries and a few server bills. That burns away fast when users get higher and higher into the tens of millions.

Being mission driven and not getting distracted is nice, but that doesn't put money in their pockets. Firefox and other products that are/were beloved by similar crowd have been asking for donations for years and it hasn't worked out. Wikipedia seems like the rare exception that pulls it off, and that's because they're aggressive with their campaigning for cash and they're basically an indispensable public service at this point.

The scale is also completely different. In terms of storage Wikipedia is not that big. Around 10TB for the English version. (https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia). That is relatively small potatoes. Hosting costs in 2022-2023 were around 3 million (https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2...).

They say no ads (https://www.thekeyword.co/news/bluesky-says-it-has-no-plans-...). Seems like they will pivot to subscriptions (https://www.theverge.com/2024/10/24/24278666/bluesky-working...). I remember reading somewhere they also wanted to add payments from user to user, so they could take a cut from that. If that is enough to cover their expenses is left to be seen.

There's a statement here (July 2023) with some info,

https://bsky.social/about/blog/7-05-2023-business-plan

but selling domain services doesn't seem it will go very far. I've seen some other rumors about paid accounts for extra features (posting longer or high-quality videos, etc).

Does anyone know if being a "public benefit corporation" is significant? Or will the same monetary pressures build up?

I mean, I think a lot of people would settle for another pre-Musk Twitter, tbh.

What's the point of "enable[ing] users to communicate their consent preferences", when none of the entities who receive those preferences are under any obligation or restraint to respect them?

This is a "close elevator" button. It's a placebo button your users can press that can make them feel—without any basis in reality—more safe, more private, more $whatever. It's deceptive. An ethical company should get rid of those no-op preferences settings altogether.

HF wouldn't exist if it was an ethical company.

I think the goal is to enable users to avoid content they don’t wish to see, not prevent content from going to specific places.

No, in this case they are talking about consent about where their public posts are scraped to. I was quoting and replying to this part:

- "Bluesky said that it’s looking at ways to enable users to communicate their consent preferences externally, though it’s up to those parties whether they respect those preferences. The company posted: “Bluesky won’t be able to enforce this consent outside of our systems. It will be up to outside developers to respect these settings. We’re having ongoing conversations with engineers & lawyers and we hope to have more updates to share on this shortly!”.

Yes, this particular feature isn’t effective at stopping content from going somewhere unwanted, but I think the main design goal of bsky and the AT protocol isn’t that.

says some 60+ npr guy who thinks AI wants his watermarked digital slr pic of a sunset so it can profit from his creative genious.

oh no :scream:

I don't see how Bluesky is sustainable. Who's paying for hosting the main instance of Bluesky? Who's paying for the firehose? How long will this last?

It’s a solid business model just like the Underpants Gnomes from South Park…

1. Build Twitter clone

2. ???

3. Profit

>We believe that there must be better strategies to sustain social networks that don’t require selling user data for ads. Our first step in another direction is paid services, and we’re starting with custom domains...

>...

>We’re partnering with Namecheap, a popular domain registrar, to offer a service for easy domain purchasing and management.

https://bsky.social/about/blog/7-05-2023-business-plan

Blockchain Capital

https://www.blockchaincapital.com/blog/bluesky-13m-users-and...

So that's an investor, but it's not a revenue or profit model. The article notes:

> The future for Bluesky includes expanding their go-to-market efforts, building out their product roadmap with new features like subscription-based profile customizations, and further engaging with their growing developer community, all while maintaining their commitment to a free and accessible platform.

Is "subscription-based profile customizations" a sufficient revenue model? The investor will at some point want a return.

indeed.

Reply on news.ycombinator.com