why? they trained on my code without my consent, why is user data any different?
training is either fair use or it isn't
and high growth Silicon Valley companies aren't known for the adherence to the spirit of the law
It's different. Code and other content that you shared online - no matter what license you shared it under - is still fundamentally different from user data that you never shared anywhere at all.
That difference is really important, to me at least.
I would be absolutely furious if I found that someone had trained an AI model on my private data in Dropbox. I personally have no problem at all with someone training an AI model on content I have posted to my blog.
Even if training on copyrighted, private code _was_ fair use, training on PII without consent is _still_ problematic. It's a huge violation of privacy and adds all kinds of legal/regulatory/ethical risks that potential consumers of said models won't be on board with.
The language and spirit of the law certainly hasn't caught up, but I feel like in some cases (unconsenting use of PII, medical information, etc), the laws and ethics of how such data is used is pretty well established.
You can certainly argue that generative AI systems are different than previous AI systems and should be treated differently. But the current situation is basically that you are allowed to train an AI on any data you have, regardless of copyright or consent. I wouldn't be surprised if that ends up being considered legally and ethically okay, because it's the status quo, and because it's hard to define "what counts as AI".
For example, it's possible to extract training data from an LLM—which could include PII/medical data/etc. Those risks don't exist with spellcheck as far as I'm aware.
To your point about what is "AI", I'd state that AI is a misnomer. What we're really talking about are generative large language models (LLMs). What an _can_ be considered an LLM is definitely up for debate, but if you were to describe one in general terms I think we could reasonably say that most (or all) things we consider LLMs are:
1. A probabilistic model of a natural language
2. Have the ability to interpret and generate natural language text
3. Typically encode a large volume of training data
1. How do you know for sure it was trained on your code?
1.1 if you saw it reproduce you code verbatim how do you know it didn't just hallucinate it ?
2. Was you code publicly available?
3. What licence was it under?“ GitHub Copilot is trained on all languages that appear in public repositories.”
Note it says “public” and not “open source”
https://docs.github.com/en/copilot/overview-of-github-copilo...
But my point is, this (training) isn't the main problem.