Take for example a human. If I studied a bunch of different open-source projects, learned techniques from them, and implemented them in my own projects, is that a derivative work? Probably not. But if I were to reverse engineer Windows and implement the techniques I saw in ReactOS, that's where it seems issues start to arise.
Sounds good in theory, until it starts producing snippets verbatim from uniquely-identifiable sources.
That's pretty unclear actually. If it's quite close to the original work, it is derivative. Even though you have probably been "trained" on quite a few different codebases over the years. Hence the existence of clean-room implementations, wherein the people building a new implementation have never seen the original.
Also, given that code that has been passed through a biological network (ie, brain) can constitute infringement, it seems obvious that code passed through a mechanical one could too. Maybe not in every case, but it certainly seems plausible.
A human being can understand abstract concepts and reason about them based on material they learn from. An ML model is a statistical model that is closer to compilation or lossy encoding or compression.
Often, ML models can encode their training data verbatim in the model itself, which is exactly what happened with Copilot and this example[1].
[1] https://twitter.com/mitsuhiko/status/1410886329924194309
So while it would not be a legal precedent, it would be a social one that would be fought against by other companies.
At the risk of adding another uneducated voice onto this legal discussion - the law is not written like code. Judges can absolutely say "hey asshole, we know exactly what you are doing" even though engineers like to imagine that all situations that use the same technical processes will be handled in precisely the same manner.
The point of my hypothetical wasn't to whitewash stealing Microsoft code in some kind of legal quirk. It was to point out that if learning the structure of code with this kind of ML model is fair use then doing so with Microsoft copyrights is also useful for other purposes. And if Microsoft themselves think that is ok it would actually be a strong argument in this discussion. I suspect they don't and was pointing that out.