Comment by cortesoft | Hacker News Reader

Comment by cortesoft | Hacker News Reader

Parent Full thread

I find it really interesting when you describe “good bot” traffic.

> During my investigation, a source from the e-commerce data industry provided a crucial piece of the puzzle. He explained that his former company was responsible for scraping 70 million retailer web pages every single day. This is a legitimate and massive source of automated traffic.

> Why do they do this? For vital business intelligence. Major retailers like Amazon do not always notify vendors when they run out of stock. So, brands pay for data scraping services to monitor their own products. These "good bots" check inventory levels, see who is winning the "buy box," ensure product descriptions are correct, and track search result rankings. They even scrape from different locations and mobile device profiles to analyze what banner ads are being shown to different audiences.

I think a lot of the players involved in this would say those are bad bots. Having your competitors scrape your site for data would probably be something most website owners wouldn’t like, but getting data about THEIR competitors would be something they WOULD like.

All bot traffic is good from SOMEONE’S perspective, otherwise it wouldn’t be happening. Someone had to program the bot, someone has to be running it. They obviously think they are good bots.

The people running AI scraping bots think they are good bots. Many content creators think those are bad bots. Price comparison sites scraping retailers think their bots are good. The sites being scraped often think they are bad bots.

I just don’t think we can clearly separate good and bad bot traffic without specifying whose perspective we are talking about.

> brands pay for data scraping services to monitor their own products.

This case seems "good" by most perspectives.

"Good bots" would be ones that fuel a healthy, competitive economy. Whether or not they're good from the perspective of a retailer, data about products and supply chains is important.

"Bad bots" would be ones that primarily support grift. They siphon money from advertisers, or get other people to waste advertising $, or game product listing metrics so that the bots' owners products get better placement, or so that other manufacturers' products get worse placement. They're not creating any new value; they're removing value from the economy by causing bad allocation of resources.

I can see many bots falling into the two categories you describe, but I can also see many other bots that would not fit easily, and whose value is debatable.

For example, bots that scrape content for AI training; is that good or bad for a healthy economy? AI can be productive, but is it ‘stealing’ other’s productivity, which could hurt in the long term if it causes decrease in future human production because of diminished rewards?

"Good" vs "bad" was not intended to be an existential question about whether some bot-powered activity like AI training will be a net good or bad 20 or 100 years from now.

It's not even a question of whether, in that particular case, AI will displace some forms of human labor, even if that causes a macroeconomic crisis where society needs to figure out a better way to operate in light of new AI-driven economic realities.

It's a simple question of whether internet bots are causing economic inefficiencies and mis-allocation of resources in the classical sense.

The bad bots, discussed in the article, are. AI scraping bots are not. All it takes to see this are all the mentions of fraud in this HN thread. Bad bots are, at their core, fundamentally fraudulent in how they interact with sites. That fraud is compounded into how companies realizing the extent of that fraud react: by hiding or lying to investors and boards, because the fraudulent bot activity has significantly altered market perception. Nothing about pure scrapers is fraudulent, regardless of whether their ultimate goals are good or bad for the economy or humanity, if that could even be known.

Reply on news.ycombinator.com