I think we already are seeing the destructive potential :/ The genie's out of the bottle at this point with open-weight models (and thus can be abliterated in terms of offensive cybersecurity) that can run inferences well on relatively cheap hardware with relative parity to frontier models.
I'm a bit of an outlier in regards to today's HN culture. I'm still holding on to Stallman's idea that information wants to be free. There was a comment on another thread that aptly elucidated what I think is necessary.
Not everyone has the know-how or desire to compile the Linux kernel, or any what-have-you executable in user space. But a good amount of people do, and it's a boon to security/privacy that they do, and cryptography can verify that this work was done. (granted, manufacturing a computer is an international effort with a lot of unknowns, ie proprietary binary blobs in BIOS/UEFI) (and people are lax to validate the crypto themselves if it's not an automated thing; that's a UX problem imo)
I think we'll eventually find sufficiently performant LLM's trained on public domain information[0] that people can validate. Sufficiently performant training algorithms that people can validate[1]. And FOSS plugins to integrate with their FOSS IDE's which they can validate. And like the Linux kernel, volunteers to make sure there's nothing fishy going on end-to-end.
Can this stop the danger of what open models can do? No. And that does scare me. It's Metasploit on steroids. The cat and mouse game of cybersecurity is going to increase in power law levels of speed. I do think verifiable LLM's for consumers to use is a worthwhile pursuit regardless. And for LLM's capable of offensive cybersecurity, I think it might be better for it to be democratized than held only by state-actors and individuals/corporations with an absurd amount of consolidated wealth.
I have no horse in this race though. I'm just happy I can learn how these things work and toy around with getting quantized models to work well enough for my coding side projects. Though I often reach for a frontier model for serious coding issues, local for rubber-ducking, and Kagi's research assistant for initial planning.
[0] Pipe dream, unless international copyright law drastically changes. Since opening the dataset for any competent LLM will violate said laws, whether obtained legally or not themselves.
[1] Knowing the training algorithm and dataset will let it actually be open source models instead of open weights.
edit: These are just shower thoughts; didn't mean to write such a long response. Would love any pushback or any correspondence in general since it'll be on my mind all day.