I mean, this is a cool demo, but that's about it. It shows that you can build a model that holds secrets which are genuinely encrypted until the model gets the correct input. But it's also trivially detectable, so the framing seems a bit misleading.
This is a deflationary comment. You make a good point that the weights today are detectable, but I think proofs of concept around poison pills are very important. I am unaware of any theoretical reason why an attacker or misaligned model could not one day find a way to disguise a poison pill such that it is indistinguishable from the normal weights. There could be such limits, but research in this direction is very useful!
How is it trivially detectable?
If (and that's a bit of an ask to be honest) I understand this correctly, the weights will look very different from what trained weights look like. Very sparse, not uniformly random.
(tinfoil hat on) Hiding it better is the next capability described in the paper that could as well be classified.
This is true in the immediate case, but unfortunately there are relatively simple methods to obfuscate weight distributions; see "Obfuscating compiled weights" on page 13, or Figure 2 here: https://arxiv.org/pdf/2407.14494