Notably “The Pile” doesn’t seem to be part of the training data. So this might be more sound legally than many other “open” LLMs
> The Pile is a 825 GiB diverse, open source language modelling data set that consists of 22 smaller, high-quality datasets combined together.
By what's the legal complication with it?
> The Books3 component of the dataset contains copyrighted material compiled from Bibliotik, a pirate website. In July 2023, the Rights Alliance took copies of The Pile down through DMCA notices. Users responded by creating copies of The Pile with the offending content removed.
Books3 is the most notable example - nearly 200,000 pirated ebooks - but a lot of the rest of it is (unlicensed) scraped web data.
The legal questions over whether this is a problem are currently still unresolved. Many people are also bothered by the ethical implications, which is a separate issue from the legal questions.
Heck, a large fraction of ethics seem to be so fickle that they’re subject to potential revision by every generation.
In fact, I’d argue that those revisions are a significant portion of how one generation distinguishes itself from their parents.
Yet strangely every generation feels like they have arrived at a set of “universal laws” in their ethics.
Pretty excellent alignment, for once?