BERT is an LLM. I don't see why Google needs to cater to increasingly fine parsing of politically acceptable vs. not ML workflows, because that is really what this is about.
And frankly I see the distinction in only number of model parameters vs number of people in cohort it was trained. So 50k-param model trained on 1b emails from 1 million people is ok, encoding/compression loses anything personal. But huge LLMs are closer to lossless compressors that can just spew all my passwords, prescriptions etc verbatim.
But it's not only that. Model size, for instance, also plays a role. Will modern BERT keep their lossy compression nature? Or could there be a privacy risk as well?
Thus, what are the characteristics of privacy-aware models? So far: discriminative vs. generative, model size, and lossy transforms (related to size). Perhaps looking at pre-2020 models could help to narrow other necessary attributes.
My guess is that the pairing of all of these things is due to the fact that Google has likely consolidated everything under the same style of decoder model.