Hacker News Reader

Comment by politician | Hacker News Reader

After reading the NYT interview, I don’t understand why he still chose to invent, in his words, a dangerous technology and publish the results openly.

Not a criticism of the man, but of the article.

View on HN

I have this same question about the (apparently many) AI researchers who believe it poses significant risks to humanity, yet still push forward developing it as fast as they can.

I recently listened to a journalist who spoke to many AI workers in SV. There is an alarmingly pervasive pseudo-religious attitude that they are ushering in a new form of life, and that it is their destiny to be the people who make it happen. Most of them will candidly tell you humanity may go extinct as a result (with at least a 1 in 10 chance), but they choose to plow ahead regardless.

Others appear to be in common modes of willful denial: hubris or salary-depends-on-not-knowing syndrome.

Just guessing, but I'm sure they get paid very well and receive promises from their companies that everything will be done ethically, nothing rushed, etc. We've seen now that OpenAI and Microsoft and Google care more about the business case rather than doing things ethically and carefully.

If a whistleblower for these companies came out and said "For the last decade advanced research has been conducted on extraordinarily big LLMs and they won't even give the public a clue of what it is and how it works" you would get a combination of people that a) don't care and b) vilify the companies for not being open and having some demonstration of this secret super power.

"why can't joe-schmo get his hands on this technology", "how can we trust something we can't see and use", etc.

A lot of the capabilities of these models are emerging as people discover them. I truly don't believe you can make everyone happy with this tech, but isn't it better than the general public can at least explore it?

Do people think that nobody was ever going to try to improve on transformers with more compute, more data, and more parameters? We knew splitting an atom was going to cause a big boom.... thats not really how this tech emerged.

Here's my theory: if you look at surveys, it does say a 10% chance or so of an extremely bad outcome. BUT it says a ~20% chance of an extremely good outcome, and an 80% chance of at least a neutral one. Simple cost benefit analysis.

I think they’re thinking like this: “it’s dangerous, but it’s better me than anyone else to do it”.

Because they believe the future is uncertain and possible upside exceeds the downside?

"Intelligence, uh, finds a way."

Yeah I don't want to be unfair or unkind, but his responses in this article seem to reflect rather poorly on his character. The thought process seems to be something like:

"There was an opportunity for someone to gain notoriety and money at a profound cost to the human race. Someone was going to do it. I don't actually feel bad about being the one to benefit, but it is fashionable to pretend to have a conscience about such things."

After the war, Robert Oppenheimer remarked that the physicists involved in the Manhattan project had "known sin". Von Neumann's response was that "sometimes someone confesses a sin in order to take credit for it."

- From Norman Macrae's John von Neumann book

Isn't it true of everything though? Explosives, airplanes, electricity, computers - all double edged swords that have both greatly benefited humanity and caused great harm (with the potential for a lot more of both).

It is certainly true that there are many inventions that pose some sort of threat to humanity, and that they are generally pursued by people who have some sort of personal/professional interest in their development. In that respect, this isn't particularly different.

The sentiment of "Oh by the way this stuff is super bad and dangerous so we should be careful" just rings pretty hollow from someone who is at the tail-end of a career spent in pursuit of that exact bad, dangerous thing. If he were 20 years younger or not wealthy it's hard to believe that he would be saying this out loud, even if he believed it.

Also this sentiment rings _extra_ hollow from someone who supposedly left CMU because he didn't want to accept Pentagon funds or work on things that would be used for war. That feels like either an incoherent value system or some pretty substantial half-truths to me.

He's also saying though that his estimate of when it may surpass human intelligence is much less than he estimated just a few years ago.

Its a science fiction trope, perhaps a trope in real life as well. Brilliant scientist gets paid to work on potentially dangerous thing. They know it is potentially dangerous so they warn about it and are reassured over and over again that nothing will be done without their consent, or that things will be done with the utmost care and security. And then scientist finally succeeds in creating the thing and the business owner's greed takes over and releases it in a premature way.

Because it's their best shot at ensuring their kids' well-being? And if they don't have children, maybe they simply don't care.

He partly answers this in the article: “because if I didn’t, someone else would”.

He states himself that it’s not a convincing argument to some.

But it surely carries some weight: in developing nuclear weapons many scientists made the same calculation even though the invention is a wicked one, in and of itself.

So, let someone else do it. It's the laziest excuse.

I imagine there will be a lot of people who agree that AI is dangerous, but continue to use it, because provides something of value to them in the short term. In his case, he might really believe AI is a potential danger, but also wanted to get the notoriety of publishing, and the money and excitement of founding a successful startup. There's not a big difference between our kind of hypocrisy — supporting something we suspect is destructive in the long term because it is neat, convenient, or popular in the short term — and his kind. Both are part of the reason things get worse rather than better. His kind is more lucrative, so it's actually less surprising in a way.

That's assuming something.

Think about it otherwise: how do you know it's dangerous until you've seen it in real life?

You raise a kid, they end up being a murderer, should you have aborted them?

Per the article, he had early misgivings — moving to Canada and refusing money from DoD. It’s not anything like your hypothetical.

Ahh - I see - thank you @politician, that is right.

Untamed nature is far more dangerous to humanity than human technology. As recently as in the 1900s, the average life expetency at birth was 30-40 years.

We're shooting guns, nuking nukes and engineering viruses, and still, on average, we're better off with all that than without it.

Fame and greed, what else.

Reply on news.ycombinator.com