I think being able to steal a language model that can instruct you on things like the notes of every meeting, employee relationships, back-channel drama, security discussions, etc. and how these things relate to corporate and technical assets, all contained in a single artifact/tarball, really ups the ante for what kind of "security hardening" you'd need for this.

Even exfiltration of API access for a few well-prompted queries could be really bad.

Presumably threat actors can already feed stolen document dumps to LLMs to quickly find interesting data.