It's evident that Mandatory Access Control has become, ahem, mandatory.
Both VM and inference engine processes should be confined, at least. There's a project where it's already done for many applications. [0]
Additionally, `lockdown=confidentiality` in kernel boot flags greatly reduces attack surface. (interferes with unsigned modules like NVIDIA drivers) [1]
Then AppArmor profiles could be written for absent applications with a learning-mode tool [2]
Nothing is bullet proof, but these approach significantly reduces potential consequences [3]
[0] https://github.com/roddhjav/apparmor.d (I'm the contributor)
[1] https://madaidans-insecurities.github.io/guides/linux-harden...
[2] https://github.com/nobody43/apparmor-suggest (I'm the author)