In particular, I've seen a lot of people say that our response to things like the Hugging Face swarm should be to prosecute whoever chose to run the swarm. If this kind of replication becomes possible, that won't work, because we will have to deal with swarms that nobody chose to run.
That rule has served me well enough!
With the same success you can argue that we can't turn off conventional malware, which can also replicate. On the other hand, LLM replication is much slower and cumbersome, so it's much easier to detect and stop, even the existing anti-virus tools should work against it with great success. Obviously defeating the "turn off" strawman proves exactly nothing.
> In particular, I've seen a lot of people say that our response to things like the Hugging Face swarm should be to prosecute whoever chose to run the swarm.
Copying whatever on another machine without authorization is already a crime, every replication initiated by the original agent or its offspring is a crime committed by the person who deployed the original, the laws dealing with malware intrusions already know who to prosecute.
You appear to be seriously confused by treating bots as independent lifeforms and you fail to see their close similarity to malware and other software.
You say that "the laws dealing with malware intrusions already know who to prosecute", but in fact we often do not know who to prosecute, either because we don't know how to track the source of the malware or the source is tracked to an unfriendly country that isn't interested in cooperating.
My point is that LLM's amplify the capabilities of malware but add nothing substantively new to the existing threats. I also expect, as you should too, that LLMs will help us harden our systems against malware which happens to include everything LLMs can do too.
> but in fact we often do not know who to prosecute, either because we don't know how to track the source of the malware
Right, in this regard, there's no difference between LLM- and non-LLM malware either, so there's no reason to hype LLMs as a new threat. Instead of made up, LLM-bad political hype, we could promote existing (but under-deployed) technical solutions which help us trace malware origins and prevent DDOS, e.g. Source Address Validation BCP 38 / RFC 2827, SAVI / RFC 7039, QUIC hardeneing, etc.