It's pretty much how abliteration works too; find where the refusals are happening and flip the bit.
I'm a bit of an outlier in regards to today's HN culture. I'm still holding on to Stallman's idea that information wants to be free. There was a comment on another thread that aptly elucidated what I think is necessary.
Not everyone has the know-how or desire to compile the Linux kernel, or any what-have-you executable in user space. But a good amount of people do, and it's a boon to security/privacy that they do, and cryptography can verify that this work was done. (granted, manufacturing a computer is an international effort with a lot of unknowns, ie proprietary binary blobs in BIOS/UEFI) (and people are lax to validate the crypto themselves if it's not an automated thing; that's a UX problem imo)
I think we'll eventually find sufficiently performant LLM's trained on public domain information[0] that people can validate. Sufficiently performant training algorithms that people can validate[1]. And FOSS plugins to integrate with their FOSS IDE's which they can validate. And like the Linux kernel, volunteers to make sure there's nothing fishy going on end-to-end.
Can this stop the danger of what open models can do? No. And that does scare me. It's Metasploit on steroids. The cat and mouse game of cybersecurity is going to increase in power law levels of speed. I do think verifiable LLM's for consumers to use is a worthwhile pursuit regardless. And for LLM's capable of offensive cybersecurity, I think it might be better for it to be democratized than held only by state-actors and individuals/corporations with an absurd amount of consolidated wealth.
I have no horse in this race though. I'm just happy I can learn how these things work and toy around with getting quantized models to work well enough for my coding side projects. Though I often reach for a frontier model for serious coding issues, local for rubber-ducking, and Kagi's research assistant for initial planning.
[0] Pipe dream, unless international copyright law drastically changes. Since opening the dataset for any competent LLM will violate said laws, whether obtained legally or not themselves.
[1] Knowing the training algorithm and dataset will let it actually be open source models instead of open weights.
edit: These are just shower thoughts; didn't mean to write such a long response. Would love any pushback or any correspondence in general since it'll be on my mind all day.
The state of openness in the hardware that enables computation is and has been pretty much 'not open' for a long time. The idea that any modern computer is really secure or private only goes so far [0], and generally that's been working fine for most consumers.
With RISC-V, ARM64, and the rising Chinese semiconductor industry, the x86 duopoly is broken up. Economically and in terms of viable IP. With the cost of creating software being driven to 0, maybe there will be real CPU's that are free. Instead of 'security for me and not for thee'.
Ironically, so long as discrete GPU's are available as neutral accelerators for certain kinds of computation, local+open LLM's are actually a reality. Nvidia of any company producing models is probably the very best with Nemotron, which also has had open training data [1]. If the development and release of 120b models of either Chinese or American origin continues into 2027, the capability compared to the cost of hardware is going to be insane.
I don't think it will be possible for the powers that be to manipulate the market in order to make the means of computation, including inference appliances, out of reach financially for average folks indefinitely.
Obviously, I view this all with a very geopolitical bent, because that's what I think is really happening. I can't make sense of the AI build out otherwise. And whatever is going to happen I think most of it will happen in the next 2 years, since the pace of AI research and the next election seem to be on track for a spectacular moment in history.
Not sure I'm making sense but ramble for ramble. :)
From what I can gather, DDR5 SDRAM are what data centers need (I assume it's involved with actual training, and efficiently running inferences from caches to consumers of their API's [?]).
VRAM, meanwhile, still remains the bottleneck for consumers. I think this means those working for Anthropic/OpenAI/etc won't be making as many meaningful advances to cut down VRAM for inferences. Why do thusly if that only allows the layman to use your open weight competitors in a more efficient manner?
For my critique on dataset and learning algorithm, that seems to only be pursued by European states who are lagging very far behind on LLM's. I'm happy that Nvidia is doing their best to be as open as they can with LLM's that the EU is acceptable to use, but there are still a lot of gaps in letting the layman verify end-to-end the entirety of the dataset[0][1][2][3][4] and the algorithms used to train the model outside of a high-level view of the framework used.
it goes back to international copyright law. Even if it was obtained in a legal manner, it's not okay to release it to the public. But it's fine to provide to states given you can prove the data was obtained legally.
So I'm not too hopeful for a decent open source LLM for the public to manage and improve on in the near future. Even if the datasets are actually open (which they're not, even for nemotron[4] beyond post-training), the architecture of learning algorithms with said limited dataset will still be proprietary. Just enough is open to be performant and compliant for government use.
[0] https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-1...
[1] https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-1...
[2] https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-1...
[3] https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-1...
[4] https://docs.nvidia.com/nemotron/latest/nemotron/super35/REA...