Hacker News Reader

Comment by radlad | Hacker News Reader

> Near the end of his column, he offers a pretty radical prescription. “One of two things must happen. Humanity needs to accelerate its adaptation to these technologies or a collective, enforceable decision must be made to slow the development of these technologies.

I'm constantly befuddled by the fact that "slowing the rate of technological progress by agreement/treaty" is really ever considered a possibility.

Take Bitcoin, torrents, or encryption for example. Good luck trying to stop people.

"Information wants to be free"

View on HN

We have plenty of technologies whose development has been stopped via collective agreement: nuclear proliferation, biological weapons, space weapons. AI won't be one of those technologies though, because it is relatively easy to develop and it gives advantages with externalized disadvantages.

Project Orion is the most tragic (but probably wisest) stopped technology, was in both the space weapon and nuclear proliferation bucket. But it also could have enabled luxuriously spacious trips to Saturn in the 70s had it been realized. I hope we manage to trust each other enough one day to make it a reality.

These are fantastic counterpoints, thanks. Still, I can't help but wonder if they are now out of the public view, but still being developed. (e.g. North Korea nuclear missile tests.)

I think you make a fantastic point that the barrier to entry in developing an AI is much lower than building a nuclear bomb.

In regard to chemical weapons - no, I think that they are abondonware because they are ineffective. Something similar with bio weapons, they can backfire to your own population rather easily. Nuclear are their special case, but still their proliferation is extremely limited.

You're right that it won't be - the difference is the tools for building those things were not distributed across millions of different people. It was distributed across only a small number of governments.

"Stopped" is a strong word. Nuclear proliferation and chemical weapons are ongoing and increasing threats. I'd bet a lot of money that we'll see the use of space weapons in our lifetime.

In regard to chemical weapons, they are decreasing danger because they are not effective in modern warfare. I'd expect for their usage to decrease. Nuclear is is not something proliferating besides North Korea and even then it is deturance and not an appocaliptic scenario. Space weapons are forbidden by treaty and though some countries are playing close to the read line, I doubt that they will be deployed. They would be political liability while being rather vulnerable in space. Hence hypersonic weapons which work better for the same goals.

Nuclear weapons are proliferating in many places besides North Korea:

https://www.npr.org/2023/02/22/1158529106/nuclear-treaty-new...

https://abcnews.go.com/Politics/iran-make-material-nuclear-d...

https://news.usni.org/2023/01/24/report-to-congress-on-chine...

Chemical weapons are very effective in modern warfare:

https://en.m.wikipedia.org/wiki/Use_of_chemical_weapons_in_t...

Though, it's more commonly used by non-state actors:

https://en.m.wikipedia.org/wiki/Chemical_terrorism

Iran is the exception that confirms the rule. They were so heavily sanctioned that it became feasible for them to say "screw it, we will do nukes anyway". I won't say that nuclear proliferation is completely stopped, but the system works for the most part. China and Russia are old nuclear powers and them having nukes is expected.

Better counter examples are India and Pakistan, but I don't know the history of their nuclear programs well enough to comment with confidence.

The chemical weapons in Syria were old stockpiles and I'd argue that the use there was not an increase of their use, but an opportunity attack. I don't think that there are many groups who currently have any kind of chemical program and can use it systematically.

I hope your comment ages better than I think it will.

> Chemical weapons are very effective in modern warfare:

Not really: https://acoup.blog/2020/03/20/collections-why-dont-we-use-ch... -- all of their uses are in militaries that are not capable of engaging in modern warfare and instead have to resort to WW1-era tactics.

Not so much chemical weapons. They're simply not that useful. Their appeal has been niche for decades, and shrinks by the year, even if you took politics completely out of it.

Nukes, yes. International pressure is the only reason at least a dozen more countries don't have nuke programs, than do today.

It’s only states that can afford nukes and also want them that badly. Terrorists would love them, and wouldn’t care about sanctions, but they can’t afford them. Large corporations might be able to afford them, but they don’t have a use for them.

AI just keeps getting cheaper, and demand is mostly non-government.

I would argue that they didn't proliferate due to high barriers to entry. I'm sure any "rogue" states that at one point would be ruled by extremists, would invest in those if it would give them a competitive advantage.

AI can be, or will be at one point, developed at home using consumer devices.

In regard to nuclear weapons the high barrier to entry is mostly because the international community made artificial difficulties in obtaining and purifying the materials. Do not forget that the first nuclear weapons were created with tech from the forties in just a few years. Since then the challenges and possible approaches have become better known and if given the chance and will, any modern country can create them if not for the international backlash.

North Korea is dirt poor, but still managed to create nuclear weapons. The barrier is not that high.

If they could do it then countries like Poland or Finland could do that too. They don't, because they have other security guarantees at the moment. If NATO and EU collapsed - the number of nuclear weapons states would definitely increase.

> The barrier is not that high.

That ignores the sixty years it took North Korea to develop nuclear weapons. It took the US and USSR less time to develop the first nuclear weapons after the discovery of radioactivity in the late 19th century!

The Soviet Union first began training DPRK nuclear scientists back in 1956 [1] and North Korea only managed to finally build nuclear weapons because of assistance from Pakistan with the uranium enrichment process in the late 90s and early 2000s, specifically the design for the centrifuges that fell into Libya's hands [2][3].

[1] https://en.wikipedia.org/wiki/Timeline_of_the_North_Korean_n...

[2] https://en.wikipedia.org/wiki/North_Korea%E2%80%93Pakistan_r...

[3] https://en.wikipedia.org/wiki/Abdul_Qadeer_Khan#North_Korea,...

You misrepresent the facts. NK does not attempt developing a nuclear weapon until the end of the cold war and it had access to plutonium at the time of the first nuclear deals in the nineties.

Nuclear and space weapons proliferation have been slowed down but not stopped. The number of nuclear powers is larger now than ever before. While no one has launched kinetic weapons into orbit lately, the superpowers are currently engaged in an anti-satellite weapon arms race.

The anti-satellite weapons are ground-based and are part of the playing close to the read line together with the spy satellites. The space weapons are when you have a nuclear head deployed in space with a few seconds to minutes ability to reach anywhere on Earth.

You seem to have some misunderstandings about orbital dynamics and nuclear weapons. Placing nuclear weapons in orbit wouldn't allow for striking anywhere on Earth within a "few seconds to minutes" unless you had an absolutely enormous constellation that would be unaffordable even for superpowers. You are forgetting to account for orbital parameter changes and then re-entry. In most realistic circumstances, a ground or sea based ICBM would be able to strike most points on Earth faster than a space weapon. Do the math on this, or you can try to simulate it in something like KSP.

Nuclear warheads also can't be stored for more than a few years and still be expected to work reliably. They require periodic intensive inspection and maintenance. Even though space launch costs are coming down there is still no feasible way to conduct such maintenance in orbit.

Check some of the old cold war designs, they definitely thought in that direction. Space weapons were prohibited partly for the reasons you mentioned and partly due to the political liability of militarizing the space.

Another way to put it:

As technology progresses, the amount of stupidity to make an "oopsie" exponentially decreases while the size of the potential "oopsies" exponentially increases.

Eventually, one gets to the point where even super geniuses are very likely to create a civilization ending "oopsie" in any given year.

This seems to be where we are quickly heading with AI.

Stopped is an interesting word here. A lot of the countries that agreed not to further develop nukes and or hold nukes are either a) lying or b) we (the USA) not provide their defense to our detriment meanwhile the real bad guys who we really need to be concerned about still have plenty of nukes.

The supplies to create biological weapons are actually surprisingly cheap! Not as cheap as graphics cards though.

To stop AI development would require pretty extreme restrictions on computing resources. It's really hard to imagine that working without having massive negative knock-on effects on other fields/industries. The economic pressure alone seems to make an effective "anti-AI" policy a non-starter.

In regard to biological weapons, they have the issue of being easy to backfire. I'm not mentioning any lab theories related to events from the recent years as an example of possible developments. There are other factors like not being very effective against people in the prime of their health like those serving in an army and the whole thing where nobody internationally likes the guys spreading illiterate viruses which do not respect boarders and kill outside of war zones.

>We have plenty of technologies whose development has been stopped via collective agreement: nuclear proliferation, biological weapons, space weapons.

Last time I checked, there was no agreement to stop nuclear proliferation at all. What do you think North Korea has been doing all this time? And Russia just decided to tear up one of the arms control treaties it was signatory to.

At this point, putting society under the control of AI is probably safer than letting humans continue to run things.

Oh, I hope that next time you will check this one:

https://en.wikipedia.org/wiki/Treaty_on_the_Non-Proliferatio...

Russia had nuclear weapons since 1949 and no matter the treaties, both it and the US have more than enough nukes to burn the planet surface a few times too many. Those treaties are for posturing and they will sign a new one when China decides to gain some international diplomatic points and join the next one.

Right, that's a bunch of countries agreeing not to develop nuclear weapons, but it doesn't include all countries. So what's the point? A few countries deciding to flout the treaty and develop their own WMDs means they can just threaten everyone else who followed the treaty. This is exactly what North Korea is doing. Now their neighbors are considering abandoning the treaty and joining the nuclear weapons club.

A treaty like this is basically worthless if you can't enforce it.

The point is managing risk. The more nuclear powers there are, the more likely is that something goes wrong. And so far we don't have countries leaving NPT in the dozens so I'd say that it is working rather well. Not perfect, but well.

Are you limiting your argument to software? Because it's been possible to curtail the use of non-software technology. Drugs, guns, cigarettes, drones, etc.

Now even if you look at software, it's not clear to me that it's impossible to stop people from using certain software, as you surmise. If you look at how governments shut down dark markets or how they've taken down certain Bitcoin exchanges or how they've blacklisted certain Bitcoin (and other crypto) public addresses, all these examples show that it's possible to enforce the law even in the software space. Child pornography is another example. Internet infrastructure is very centralized, and governments have over time acquired tools to control how information flows.

Information doesn't want anything. That quote is a freudian way of saying "I want information to be free".

> Because it's been possible to curtail the use of non-software technology.

Curtail but not stop.

There’s an opioid epidemic going on, gun violence is a thing, cigarettes still keep killing people and drones are becoming quite efficient killers in their own right.

People still cheat on their taxes, drive drunk, pirate movies &etc. I could go on for days.

Oh, and the whole “illegal number” thing, how’d that work out?

"Curtail but not stop" sounds like a good, realistic goal.

Alternately, take nuclear proliferation, resource extraction, or pollution.

Of course you can’t reach 100% enforcement but you can make effective guardrails that limit the opportunity for worst case scenarios.

The difference between your examples and mine is basically just how much actual concern there is over the problem and who it impacts.

Is nuclear proliferation not the single most expensive pursuit in the world though?

I've suggested it before: a moratorium on ML/AI publications. That's what's been fuelling these developments. Academic researchers need to publish. They don't make these models for the money (initially). Stopping the publications will slow down research and the money put into commercializing it.

You understand we have this thing called “freedom of the press”?

Maybe get an old fashioned book burning going too?

Academic publications. They are already heavily censored. Oe you can stop counting them towards the researcher's citation index, de-incentivize libraries and schools from buying subscriptions on ML/AI journals, require articles to wait 10 years before publication, whatever. Stopping undesirable publications can be done.

I’m not suggesting ai development should be stopped, but unlike bitcoin, torrents, encryption, etc, ai development—for now—requires prohibitively large computing power and expertise that are usually only accessible to law-abiding institutions. this means that it can be regulated, and relatively easily at that

sure you’d struggle to get China and Russia to play along, but within the EU and US I really don’t think it would be as hard as you think

That's gonna end well. Particularly as Russia's Internet Research unit has direct, immediate and pressing need of a chatGPT-like thing as a weapon of war.

Back in the day, they had to make use of troll farms staffed by humans.

I’ve honestly been sceptical for a while that these “troll farms staffed by humans” weren’t just LLMs in the first place anyway. they’re not that new

Well it's because the initial reports of troll farms came from journalists joining them as employees, and also that doing it manually is not that expensive compared to paying for development of complex software; you can do a LOT of information dissemination with just a handful of cheap full-time employees.

It will require something like IAEA, a UN agency. It will require inspections of code and data centers. We can certainly see what 20 years may look like. There will be "snap inspections", "sanctions", and "rogue nations".

None of the superpowers, specially their militaries, will acquiesce to slowing research, development, and deployment, without equivalent of arms treaties. AI is clearly a dual-use technology with immediate application on battle fields, including cyberspace.

Outside of geopolitical realm, we the little people don't have anything beyond UNHRC to protect human rights in context of mega corporations and governments use of AI. The superpowers may agree to certain things but that does not translate to protections afforded to individuals and societies.

ATM I think it may be unwise to wait for things like GDPR for AI. I very much appreciate, for this very reason, efforts of orgs and hero developers who are working towards making available the necessary for running local, personal, private, and self-directed AI (such as llama.cpp for example).

From a governmental level, thoughtful nations will create programs for the transition. There are precedents from the industrial era as to what approaches worked and what did not work.

Finally, again a reminder that all societal matters including tech must ultimately be decided at the political arena, and purely technical social action (code, services, etc.) to address legitimate concerns are not going to work. We have to mentally and emotionally escape the hype cycle that every new wonder tech brings. You can absolutely love AI, that is fine, but now is the time to call your congress critters and senators. The decisions in this space can not be permitted to be made purely based on the mechanics of the economy.

the expertise and technology for this kind of development reside in law-abiding universities and businesses. if you say it’s illegal to develop, then they will follow that law

you don’t need some kind of Big Brother UN nonsense the same way you don’t need Big Brother UN nonsense to stop nuclear bombs getting into the wrong hands. it’s too specialised, too expensive, and the people involved are mostly scientists and academics

also—and this has begun to annoy me—people have this idea that politics is inaccessible and not representative and there’s no solution: they’re right. politics is inaccessible and unrepresentative. however there is a solution

unionise

band together and create the change you want to see. politicians have never followed the will of the people. they’re too easily bought, or jaded, or blackmailed. real, meaningful change for the common man comes from organising to create a counterweight to the power of corporations and the rich

> ai development—for now—requires prohibitively large computing power

What's crazy is that it may not be that way for long. If people can run LLaMa on a Pixel 6, it seems easy for us to get to a point where all computing resources have to be closely monitored and regulated to prevent unlicensed AI development.

It reminds me of the Butlerian Jihad from Dune.

it’s not so much about running as it is developing. you can run stable diffusion on an iPhone, but that has no bearing on the fact that developing something of its ilk would cost you on the scale of hundreds of thousands of pounds

> all computing resources have to be closely monitored and regulated to prevent unlicensed AI development.

With most chips being made in China?

I agree, but alas tis the motions that are followed. OpenAI went from singing praises of open-sourcing AI to now flagellating themselves for having done so [1]. Won't do much unless like the author says "a collective, enforceable decision must be made to slow the development of these technologies" which like yeah...good luck with that.

1: https://twitter.com/tobyordoxford/status/1636372964001333249

> Good luck trying to stop people. "Information wants to be free"

I agree, we’re struggling with the idea that our technological systems have more agency than us. I think it’s hard for technologists to see this as an actual spiritual reality, even though we borrow it metaphorically for argument.

Torrent use has been for sure slowed down by governments.

The key people behind Torrent websites are likely in jail or fighting lawsuits.

I had a friend who lives in Germany served with a fine of more than 1000 euros because he had forgotten to turn on his VPN while accessing a torrent website.

> "Information wants to be free"

Have you heard of the great firewall of China?

>Have you heard of the great firewall of China?

Yeah, I've heard how it's basically a formality that barely stops the free flow of information:

>During the survey period, it was found that 31 percent of internet users in China had used a VPN in the past month.

https://www.statista.com/statistics/301204/top-markets-vpn-p...

I'm not sure about Germany, but in the US there's been very little legal action taken against torrent users or websites in recent times. There are still plenty of popular torrent sites for anything you can imagine (e.g. The Pirate Bay is still online, BTN for TV, PTP for movies, Redacted for music.)

If anything has curb torrents, I think it was the advent of streaming media services (i.e. market dynamics, not policy.) However, the flood of new services on the market is causing the cost of consumption to increase again and I expect we'll see a revival of torrents... Plus, while torrenting traffic is down, I think more people are sharing downloaded media via services like Plex, which may mask the "actual" distribution of torrents.

> Have you heard of the great firewall of China?

If this is a serious question... yes? What argument are you making exactly?

Torrent use has been slowed down by Netflix, Spotify and Steam. I haven't pirated a song or a game for more than a decade.

Carrot is stronger than a stick.

And why would a German go to a torrent site? They have lots of money.

A treaty isn't even necessary. Civilization will start to degrade in a non-trivial way once we pass the tipping point of people relying more on AI than they do on their own thinking. When people can no longer fix what AI cannot—after all, "AI" is just a model of our past thinking—all hell will break loose and the only choice will be chaos or shutting it off (if the people who can do it even exist).

These people constantly make a mistake by referring to "humanity" and saying "humanity" needs to make a decision, yada yada. Yes, we're all humans. But "humanity" doesn't have much decision making power. Instead the dominant entities with decision making power continue to be national governments. And it sounds a lot dumber to say "America needs to slow the development of these technologies" during a period of intense technological competition with China.

Imo, the problem isn't really the existence of AI but how it's used.

And there's plenty of ways to censor corporations from using AI for various tasks. Corporations automating everything by using unaccountable AI is what I'm most afraid of. No recourse, just talking to unrelenting machines for anything from denied loans, insurance claims, health insurance claims to contesting frivolous Comcast bills.

My currently biggest nightmare would be easy to legislate away, if they don't lobby hard enough...

Before most of our time, but my understanding is that the past restrictions on cryptography research and export were reasonably effective?

Yes they were disliked and probably with good reason, but just mentioning it as a counterpoint that perhaps it is possible. You could have made a similar argument that anyone with pen and paper could do cryptography

Personally I think there's no way to be sure other countries aren't doing it, so perhaps it will continue in some government controlled way

There was never any real legal restriction on cryptography research. There used to be US export controls on cryptography implementations, and those were fairly effective on US based companies and open-source developers. But that was totally pointless because it generally had no power over foreigners. It just put US companies at a competitive disadvantage for no benefit.

For an excellent example of how this can work, check out "Barriers to Bioweapons": https://eukaryotewritesblog.com/2017/06/30/book-review-barri...

Those other technologies do not require huge amounts of computing power, which provides a point of interdiction.

Bitcoin begs to differ.

we’re talking about development, not usage

it doesn’t take masses of processing power to develop a crypto coin. or really any at all. it’s just a bunch of network calls combined with a hash function. any old idiot can develop one on a device with an IDE and access to the internet

yes, mining them can use a load of compute, but that’s not the point

Given LLAMA runs enough to be interesting on a Macbook I'm not sure this is going to be a fundamental limitation, and if it's one today it's certainly within the order of magnitude that models will run locally within a few years.

This is why attempting to put guardrails around it simply won't work tbh.

It is the training of new models that requires huge compute, not running them

Sure, but that isn't like Bitcoin it doesn't have to scale up to every bit of usage. We are already getting interesting scenarios from models that are done and the energy spent. That isn't a fundamental limitation.

the discussion is about whether it’s possible to ban development or not

torrents are pretty dead (due to law enforcement?)

judicially treating bitcoin as asset instead of tender didn't help its cause either (?)

and true encryption without elliptic curve nsa loopholes is not that widespread and needs to be declared in appstores

I'm sure there will be regulations against "DAN"

How about the proliferation of nuclear weapons?

Ultimately we don't want to encourage development of these 'tools' because they stand to wipe us out; is AI actually in a similar class if we take the risk seriously?

It may be that energy prices are ultimately the limiting factor in the advancement of ai.

This is literally the plot of the Matrix.

maybe you could slow down the hardware? like, limit the number of execution units GPUs are allowed to have? or slow them down? extremely heavy handed, but maybe better than Skynet?

You’re going to have problems running React before you have problems running AI engines.

Pretty much just the Mennonites.

Reply on news.ycombinator.com