Hacker News Reader

Comment by xeubie | Hacker News Reader

It seems that Meta didn't mention one aspect of their "next-generation" AI training system: their incompetent web crawler hammers websites while completely ignoring robots.txt. Meta's 57.141.0.0/24 made up about 90% of all requests to my server this past week, and it didn't stop after I blocked it. They continue to make a request every 2 seconds despite me responding with a 403 every time. Truly an engineering marvel that Meta has bestowed upon us.

edit: I just checked and they are back to hammering my server at 105 requests per minute. These are the Meta-owned ranges I'm blocking now, in case it helps anyone: 57.141.0.0/16, 57.142.0.0/15, 57.144.0.0/14, 57.148.0.0/15

View on HN

This sounds suspiciously close to what used to be called a DDoS attack

So they can do something which used to be considered stalking

Every single thing this company does would get an individual thrown in jail

Why do the zuck and his goon squad get a free pass to literally be criminals?

I've blocked Meta, OpenAI and Anthropic and yet they seem to find ways to get back in. It's pretty annoying.

nft add table inet filter

nft add set inet filter meta_crawl '{ type ipv4_addr; flags interval; }'

nft add element inet filter meta_crawl '{ 57.141.0.0/24 }'

nft add chain inet filter input '{ type filter hook input priority 0; }'

nft add rule inet filter input ip saddr @meta_crawl tcp dport {80,443} reject with tcp reset

I did finally block them at the firewall level last night. They are constantly trying to connect anyway.

When you're a celebrity, they let you do it.

Poison the responses with massive ai slop dumps.

Make them block you.

That's probably the best way to be honest.

maybe just drop them rather than reject with tcp reset

> These are the Meta-owned ranges I'm blocking now, in case it helps anyone: 57.141.0.0/16, 57.142.0.0/15, 57.144.0.0/14, 57.148.0.0/15

What's Meta's ASN? May be simpler to just null-route the entire organization.

> incompetent

I think that's not incompetence, I think that's by design.

How? What's the incentive structure for that design? There are no extra ads to be displayed here, so how does Meta get value from that design? Seems way more likely to be incompetence in my world view. Explain why that design is superior!

LLM training data

With Muse, people can enter their bank account information and medical information so that it can work hard for them. For this, they need a lot of training material.

There are no extra ads to be had even by scanning sites with a robots.txt that allows them, because the value of pages could never have been in displaying ads but must have resided in the information on the pages itself.

Someone puts a high number of pages crawled per minute in their PSC. Doesn't matter if they're 403s.

Reverse Hanlon's Razor

Their apps and their hardware might be gorefests but you best believe their tracking software is rock solid with no possible failure modes

Those identified themselves are not the worst ones.

I have seen crawlers fake chrome and edge from a set of Brazil residential IPs crawling at 1.5m req/hour for 3 hours, and it comes back every four weeks at same time.

(Rhetorical question) Who would you trust scraping your website? Google, for who among all their issues I haven't seen complaints of "Google is ddosing the web", both pre and post LLM

Or Meta, run by someone of Zuckerberg's character that wants to dominate everything and it shows by how many requests per minute? He's one of the reasons sites now have to implement bot blocking technologies

I didn't want to have to put everything behind cloudflare, but this sort of nonsense made it the least bad path forward for us.

bet it was written in PHP

Reply on news.ycombinator.com