> Out of curiosity, how do we define license violation in that case? I, as a human being, have trained by reading code, much of which is covered by licenses that are somehow not compatible with code I'm writing. Am I violating licenses?

That depends, if you end up writing copies of the code you've studied then yes. You are on thin ice. Plagarization is definitely something that you can do with computer code. There have been several high profile cases around this in arts. As far as I can see it usually ends up being a question about how much of the work is similar, how similar it is, and how unique what was similar is. And added wrinkle in programing is that some things can be done in only one way, or at least any reasonable programmer will do it in only one way. So for example a swap(var1, var2) function can usually only be done in one way, and therefor you would not get in trouble if your and someone else swap function are the same.

I've been following the discussion about Copilot, and one issue that comes up again and again is that people seem to think that since Copilot is new, the law will treat it, and the code it writes differently, than what it would you or a copy machine. I think that is naive, my impression is that courts care more about what you did, not how you did it, and if you think Copilot can be used to do an end run around the law. Prepare to be disappointed.

So if Copilot memorize code and spits out copies of that code, then it is at best skating on thin ice, or at worst doing a license violation. If the code it is copying is unique, then it definitely is heading into problematic territory. I'm fairly sure sure someone in legal at Github is very unhappy about the quake fast inverse square root function.