For those also wondering: https://pile.eleuther.ai
> The Pile is a 825 GiB diverse, open source language modelling data set that consists of 22 smaller, high-quality datasets combined together.
By what's the legal complication with it?
> The Pile is a 825 GiB diverse, open source language modelling data set that consists of 22 smaller, high-quality datasets combined together.
By what's the legal complication with it?
Books3 is the most notable example - nearly 200,000 pirated ebooks - but a lot of the rest of it is (unlicensed) scraped web data.
The legal questions over whether this is a problem are currently still unresolved. Many people are also bothered by the ethical implications, which is a separate issue from the legal questions.
Heck, a large fraction of ethics seem to be so fickle that they’re subject to potential revision by every generation.
In fact, I’d argue that those revisions are a significant portion of how one generation distinguishes itself from their parents.
Yet strangely every generation feels like they have arrived at a set of “universal laws” in their ethics.
Pretty excellent alignment, for once?
> The Books3 component of the dataset contains copyrighted material compiled from Bibliotik, a pirate website. In July 2023, the Rights Alliance took copies of The Pile down through DMCA notices. Users responded by creating copies of The Pile with the offending content removed.