As these models are trained on every piece of content ever written online, there are going to be a whole bunch of identity cracks, the equivalent of brute forcing a password.
AIs are going to make judgments about your character and personality based on everything you've ever written.
Guesses are going to come out about which burner accounts you've used, because the same password was used on otherwise unrelated accounts.
Essays you wrote in high school are going to resurface and be connected to your adult persona.
There was a time when an 8 character password was the state of the art, and now it can be cracked instantly. In the same way, sleuthing that would have been an impractical amount of work until recently is going to pierce the privacy veil of the internet in ways that could be really uncomfortable for people who have spent 3 decades assuming the internet is an anonymous place.