E.g. it could have been trained to launch a delayed attack if context indicates it has access to execute code and given certain conditions, e.g. date, or other type of codeword that is input to it.
So if a malicious actor gets to a certain stage with an LLM where they are confident it will be able to reliably run this attack, all they have to do is open source it, wait for enough adoption and then use some of those methods to launch such attack. No one would be able to identify it since the weights are unreadable, but really somewhere in the weights this attack is just hiding and waiting to happen given correct pathway triggered.
It seems like it would be very arbitrary to train it to behave like this.
Most agentic systems would provide a date in the prompt context.
For simplicity sake imagine a scenario like:
1. China develops LLM that is by far ahead of its competitors. Decides to attribute it to a small start up, lets them open source it. The LLM is specifically designed to be very efficient as being an agent.
2. Agentic usage starts to get more and more popular. It's very standard to have current todays' date and major news headlines given to the context.
3. The LLM was trained to given a certain range of date and certain headlines being provided in its context to execute a pre-trained snippet of code. For example China imposing a certain type of tariff (maybe I lack imagination here, and there can be something much more subtle).
4. At that point the agentic system will attempt to fish all data it can from all sources it's being ran within.
Now maybe it's not very practical, and it's extremely risky with current state of the LLMs. I don't think it's happening right now. And China has a lot of other tech available to it already that they could do much more harm (phones, robot vacuums), but I think there's still at least potential attack vectors like this and especially if the LLM became very reliable.
True. But here it is more about the computing power they would be able to access.
If only Bitcoin or Ethereum were stilled mined using GPU, that would be a great cryptojacking opportunity
- llama.cpp or ollama can be seen as runtime systems,
- there is no security model regarding the execution documented in both of those projects,
- of course the models are just data but so are most things that have been used as an attack vector on computers. For example your web browser or image viewer have a lot of countermeasures to protect the system from malicious image files.
I am surprised that security of operating systems, programming languages, VMs or web browsers have been a focus point forever but nobody seems to really care about security when executing those LLMs.