It was trained using source code whose licenses don't permit such usage.
The phrase “derived work” is, IIUC, a phrase from copyright law. And you’d have a hard time convincing me that Copilot-generated code is not a derived work from its training data.
> And given that they’re open source code bases I expect licenses to explicitly disallow things, and consider anything not explicitly disallowed as permitted.
That is very much not how copyright and licences work. Copyright law gives the copyright holder the exclusive right to make copies of the work, making derived works, (and to do some other related things, like making a public performance of it, etc.), so to do any of those things, you need explicit permission, i.e. a license from the copyright holder to do it. A license is not a list of things you are forbidden to do; on the contrary, it is a list of things you are permitted to do, which you would not otherwise be legally allowed to do according to copyright law.
This is a novel scenario. It seems unclear how the courts will interpret it? Never mind what we think, will they decide it's a derivative work, or is it a transformative use?
Suppose we create a new AI image generator, and use as training input every image ever made of a Disney character (official images by Disney, that is, no fan art), including every frame of every Disney movie. Could we just use the output images of that AI however we wanted to? (Not withstanding trademarks.)
https://en.m.wikipedia.org/wiki/Copyright_protection_for_fic...
It's not even clear how often training machine learning algorithms on code results in copyright violations. CoPilot does have a setting to detect and disallow direct copying, but how well does it work?
This legal uncertainty is enough that I wouldn't advise using it, but maybe people who use it will be fine?
Europeans will probably preempt that with legislation.