Hacker News Reader

I am curious about this too, from a copyright perspective:

1) Whether it is legal for Common Crawl to collect this data in the first place? Presumably they don't have explicit permission from every site they have crawled?

and

2) Whether it is legal to use this data to train ML algorithms for commercial use (e.g. GPT-3)? They provide a bunch of examples of this on their site but the terms of use are kind of vague about this.

View on HN

It has to be legal to use legally acquired published information for a commercial purpose. That is not nearly the same thing as selling someone else's work. Training a ML algorithm on published work is no different from me reading a tutorial on the web and using the new skills I learned to do something that makes me money. That is very different from reselling the tutorial as if I authored it.

The data is open they don't crawl inside password protected areas of the internet. Why shouldn't they be able to read data put into the open? If the authors didn't want their data openly accessible they should put it in a private place.

Except just because someone published something on the internet doesn't mean you have the legal right to reproduce and redistribute it yourself.

That has nothing to do with reading data that the publisher does distribute.

They're not just reading data, they are redistributing it.

Reply on news.ycombinator.com