It’s not clear to me that it’s any easier to judge this than in human cases of “inspiration vs infringement”. What part of the model weights can a lawyer point to and say “this bit clearly incorporates a substantial portion of my client’s work”.
Practically speaking I feel this is a strong hint that some degree of copyright laundering is taking place.
You are presupposing that the company’s own code would somehow be a massive boon for the model, resulting in lower loss overall.
In reality, it would skew the model towards that company’s “mode” of coding which isn’t what “normal” programmers expect. In fact, they are most likely to expect the coding styles they learned from, and that is most likely to be found in public examples (GitHub, StackOverflow, textbooks, Reddit, etc.)
This argument is so silly to me. Anyone who has worked at a large enterprise, whether it’s Google, Amazon or Target, knows that company code is effectively guaranteed to be extremely hard to work with. This happens for organizational reasons and really the best thing to do about it is admit that it’s happening rather than pretend it’s all perfect.
I'm amazed you called that argument silly while countering with this.
That definitely wasn’t very clear from my comment however.
This does not matter for open source since anyone can see this info already.
Actually, we do see them do exactly that. Microsoft is ahppy to have a shared-source licence that gives away much of the Windows source code to universities.
The fact that they don't want to train their models on it says a lot.
If, as they claim, learning from existing materials does nto devalue those materials in any way, they'd chuck the entirety of the Windows source code at the LLM for training purposes.