Hacker News Reader

Comment by riedel | Hacker News Reader

As a European (German) I am always wondering about the legal basis for a) making a copy of copyrighted material and databases available and b) processing contained personal data.

The Internet archive seems more like a library with exceptions applying, but common crawl seems to advertise also many other purposes that go beyond archiving publically relevant content.

Would this be possible in Europe, too? My feeling is that US legislation different here. Do you have to actively claim copyright in the US or enforce technically e.g. via DRM? Anyone can use anything without a license as long nobody finds out?

View on HN

Well they've got a bunch of different data that would fall under different precedents. Metadata would likely fall under the phone book ruling. The raw warc data could be argued to be transformative, and would certainly fall under fair use - especially because it isn't even close to threatening market replacement. Property abandonment might fit in somewhere as well :)

https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._R....

https://en.wikipedia.org/wiki/Authors_Guild,_Inc._v._Google,....

I am curious about this too, from a copyright perspective:

1) Whether it is legal for Common Crawl to collect this data in the first place? Presumably they don't have explicit permission from every site they have crawled?

and

2) Whether it is legal to use this data to train ML algorithms for commercial use (e.g. GPT-3)? They provide a bunch of examples of this on their site but the terms of use are kind of vague about this.

The data is open they don't crawl inside password protected areas of the internet. Why shouldn't they be able to read data put into the open? If the authors didn't want their data openly accessible they should put it in a private place.

Except just because someone published something on the internet doesn't mean you have the legal right to reproduce and redistribute it yourself.

That has nothing to do with reading data that the publisher does distribute.

They're not just reading data, they are redistributing it.

It has to be legal to use legally acquired published information for a commercial purpose. That is not nearly the same thing as selling someone else's work. Training a ML algorithm on published work is no different from me reading a tutorial on the web and using the new skills I learned to do something that makes me money. That is very different from reselling the tutorial as if I authored it.

Reply on news.ycombinator.com