You do regulate cars so they don’t have spinning blades on the front or flamethrowers from the sides.
Such laws attach to the human operators of such cars, if they recklessly endanger others by driving them on shared, public roads.
But they're perfectly legal to own (and operate off-road). And, there's no technical barrier stopping people from illegally driving them on public roads: just fear of laws.
(There's entire subcultures of people who modify cars to do insanely dangerous things, on private race tracks. Perfectly legal).
If we were to apply Amodei's safety standard to cars—do they have the theoretical risk of killing over 50 people—well, cars empirically kill tens of thousands of people in the US annually, so we'd have to categorically ban them. That standard is bad-faith and absurd.
(Would we also need a real ID to operate curl(1)?)
Regulations should attach only at the point humans willfully or recklessly cause their agents to commit crimes.
What Amodei's trying to do is outlaw free, open-weight models. That's two distinctions: (1) non-commercial, and (2) abstract, intangible mathematics. Pure speech. Same as Phil Zimmerman and PGP.
Even the EU's draconian open-source regulations (which HN criticizes endlessly) exempt free software which is purely non-commercial.
AI is much more comparable to that than to a car.
You can still run someone over with a car. The analogy doesn't require absolute lack of capability to do harm.
And, more importantly, we certainly don't try to limit the capabilities of computing that might be used to do those things. Given how AIs are very general purpose tools, the fact that they can be used to do something potentially harmful is too narrow a threat.
Nobody is trying to push for cars that automatically stop themselves from running over someone either, which would make for a more comparable analogy than cars with flamethrowers.
If a person closes his eyes and hits the gas pedal on the car and runs over someone, the person is to blame, not the car. AI companies do the same. Your eyes may be closed but you're still operating/hosting the car/AI.
Keep in mind that those open source/weight models are not as powerful as whatever closed source Claude/GPT version exists behind the scenes that hack companies. As per the Hugging Face hack - open weight GLM saved the day as closed source ChatGPT refused to help Hugging Face fix their systems. An example of my point that "sidestepping any third-party checks that could be present" IS good.
Part of point of open source/weight AI is that it allows what those model providers can't: uncensored models for whatever reason you may need it. You are responsible for being legal with it.
Until then, I propose we use the 'unplug the server' rule. Can [insert bad thing] be prevented by unplugging the server? Then these things are fully in our control and no, we don't need to re-build hundreds of years of legal precedent from the ground up.
Liability in the event of harm is already something we have a pretty good handle on from living through the industrial revolution 150 years ago, a time when much more activity that companies did existed in the real world with real harms, as opposed to digital companies doing imagined "mental" harm to soft desk chair people who work at email jobs.
No, only metaphorical AI spinning blades and flamethrowers should be regulated. Actual AIs with spinning blades and flamethrowers will be subsidized and unleashed in your town.