Hacker News Reader

Comment by bicx | Hacker News Reader

Google scrapes like a maniac. And for profit. Many others do the same.

A website can put up a TOS prohibiting such use, but my understanding is that is essentially unenforceable if the site is publicly accessible.

The recent Meta v Bright Data case highlights how extreme it can get without being technically illegal. https://techcrunch.com/2024/02/26/meta-drops-lawsuit-against...

If you’re trying to prevent scraping of your data, your best option is to not make it public.

View on HN

I have a website that has around 5k Bing visits every single day. Basically my most expensive user. Compared to Google with about 70 visits daily.

I randomly block their IPs, tried some stuff with robots.txt and even completely banned it in the past as I thought this must be something else. It would just show up with new IPs and proceed.

The few times I checked it looked like official IPs. If I knew how I would sue Microsoft. They have no business in scraping my website 3-5 times a day when they send me basically no traffic

Edit:// it's also not my only website where bing goes crazy. And it's not new, this is going on for several years now (so no AI scraping I guess)

They absolutely should be paying you if they're single-handedly abusing your resources

There is a YouTube video about a content creator who got a huge AWS bill out of byte spider (bytedance's scraper) indexing his site incredible numbers of times.

Or maybe just don't host your stuff in such a way that any internet user can bankrupt you just from sending GET requests?

If GET requests increase my AWS bill, does that make them not idempotent any more?

This sounds like a scoping issue

This so much. While it's annoying to waste resources for nothing. All I really had to do to make this a non issue is upgrading my VPS to a slightly larger box.

If I would pay per requests I couldn't sleep anymore

On the other hand the amount of traffic he's mentioning is about 3.4 requests per minute which is effectively background noise. They're probably getting much more hits from actually malicious scanners than Bing's misbehaving spider.

> The few times I checked it looked like official IPs

Considering Microsoft now runs a cloud service, it may very well be their cloud provider users and not official Bing scrapers.

Usually it's robots.txt that 'prohibits' such use, but you're right it's not enforceable.

Reply on news.ycombinator.com