> What makes an open-weight LLM dangerous?

Same thing that makes any LLM dangerous. The intent and actions of the person using the LLM to do bad things. Any tool can be dangerous in the hands of the wrong person...

As the sibling comment says, user actions are the main driver, but also the fact that the weights can be modified. If a model is released with safeguards, they can be ablated out or the model can be fine tuned for nefarious purposes, and nobody can really stop that from happening.

That’s not to say that the majority of open weight models are being used in nefarious ways or are dangerous, but unfortunately despite (hopefully) most people being responsible, we have to guard against the small number of bad actors. Like with everything, a small number of people ruin it for everyone.

Nothing makes an LLM dangerous. An LLM remixes existing knowledge and outputs text. Any danger comes from what one subsequently does with the text.

For example, when a lab like OpenAI negligently wires up a harness to an LLM and executes arbitrary code in YOLO mode without common-sense isolation measures, the executed code can do bad things.