Models can easily regurgitate back training data verbatim, so anything private can be in theory accessed by anyone without proper access to that file
IMO the bigger concern is that this data is not just used to train models. It is stored, completely verbatim, in the training set data. They aren’t pulling from PDFs in realtime during training runs, they’re aggregating all of that text and storing it somewhere. And that somewhere is prone to employees viewing, leakage to the internet, etc.
Isn't this like encryption, though?
I'm fairly sure that the cryptography community basically says: if someone has a copy of your encrypted data for a long time, the likelihood over time for them to be able to read it approaches 100%, regardless of the current security standard you're using.
Who could possibly guarantee that whatever LLM is safe now will be safe at all times over the next 5-10-20 years? And if they're guaranteeing, they're lying.
To which the reply was "they'll just make the LLM able to better defend itself".
And my point was "the attackers will learn to build better prompts, too".