lf88

Well..clearly it isn't safer: alignment is ill defined, and open weights models can be abliterated with limited resources.

I am worried about many things.

I would say that the existential threats are: - biosecurity risks (i.e., AI-assisted development of very effective biological weapons) - a future where humanity becomes a sort of a "pet species" of an artificial superintelligence (humanity would not go extinct in this scenario, but the loss of agency and meaning would be such that I consider it an existential threat)

I'm worried about other risks that I would not define "existential" but are still likely to cause great harm to humans: - large-scale AI-powered disinformation campaigns that could undermine our democracies - the potential of AI for pervasive, automated surveillance, which could lead to an inescapable authoritarian future - autonomous weapons (could become an "existential risk" if at some point we'll allow AI to control also nuclear weapons) - cyberattacks to critical infrastructures - the consequences of widespread unemployment among white-collar workers: social unrest, rampant inequalities... - the consequences of prolonged cognitive offloading on our minds

On a more personal level I would say that AI could automate away some aspects of my job that I deeply enjoy. It's personal, but I think that many people feel the same, and this makes them suffer at various degrees. I'm also saddened by the widespread usage of AI to write texts of any kind, even personal communications, as I feel that the human behind those texts disappears (I acknowledge that, compared to all the other things, this point is minor).

I'm not using them at all. I am too worried about certain aspects of the technology and its current trajectory, which I consider a very serious threat to humanity.

Complete title: Why we must stop talking about artificial general intelligence — and instead build ‘pro-worker’ AI

Sad... another interesting thing broken by AI...

It's refreshing to see serious open-source projects adopting a strict policy on LLM-assisted contributions.

Besides considering frontier LLMs a serious threat to humanity, I find LLM-assisted programming simply...not fun! I tolerate only AI-assisted bug reports, as they are genuinely useful.

I am actively trying not to use it at my day-job, and I am a (mainly) computational physicist. My mother tongue is not English, I live in a country where the language is neither my mother tongue nor English, and I still don't use AI for writing emails or other documents. It is going OK so far. However, if significantly more powerful AI is developed, I think that we'll have much more serious problems than using or not-using AI in our job.

And then an AI will likely read those fancy emails and summarize them into three bullet points.

Chess and StarCraft are games. For many other intellectual activities, part of the satisfaction comes from contributing something meaningful to a collective effort. If a machine can do that orders of magnitude faster, for many people these activities become effectively pointless.

Shameful

Creating this new website seems quite tasteless to me. It's worth reading this post that explains why the original website got rid of the gamified approach that the new website promotes: https://www.erdosproblems.com/forum/thread/blog:9

AI development should be harnessed in such a way that AI augments humans without replacing them. Tools that replace human intelligence altogether are not true progress for humanity.

Building a super-intelligent AI is the stated goal of major AI companies. It may or may not be possible, but due to the recent surge in capabilities of frontier models I would not take any risk.

Frontier models are, indeed, already too capable and too dangerous. I would have been more comfortable if a stop of the development occurred 12 months ago. However, the second best moment to press the "stop" button is now.

This goal seems flawed to me. There's no universal agreement on "morals and ethics". Superficially people may agree on some general principles, but if you go into the details you'll likely find substantial differences.

I agree. I am extremely worried about the development of frontier AI models. I would like to see an immediate stop of further developments of such models and a permanent, global, ban on superintelligence.

On the contrary, I think that less capable models could be used for several, largely positive, applications.

I also think that narrow AI models developed for specific scientific applications (e.g., protein folding, weather predictions...) are in general beneficial for humanity.

According to all the recent polls the vast majority of the population in the western world is worried about AI and wants stricter regulations. I would say it's about time!

Sure. I am aware of this. It's sad that the first "victims" of AI could be people working in some of the most rewarding professions (art, music, math research...). As far as labor is of concern, however, most people will likely suffer more from social unrest and rampant inequality due to widespread unemployment among white-collar workers. And I am also worried by the potential effects of long-term cognitive offloading.

It would be great if AI could take away the soul-crushing part of the work and leave only the rewarding part. It's not heading that way.

As I wrote in (https://news.ycombinator.com/item?id=49984923 ), this announcement feels to me more like an ominous warning about the near future than something to celebrate.

This would be a largely positive solution for humanity, as long as AI remains "just an assistant". I don't think that we are heading that way. It's possible that, at some point, when the time you save becomes comparable to your full working week, your employer (or your clients, if you are self-employed) will discover that you are just a proxy between them and prompting directly the AI. Of course, if your job involves physical labour or requires physical presence or taking legal responsibility for something, you'll be fine for a bit longer (well, comparatively fine in a society plagued by widespread unemployment and social unrest).

Precisely this reason.

I feel this moment is one of the last few warnings before things will get seriously out of hand. We need to stop now. Building a superintelligent AI should be considered a crime against humanity.

It's very unlikely that a superintelligent AI will create unlimited prosperity for everyone on a finite world in a short amount of time. You may not be among the lucky ones.

A global ban on superintelligence is essential for a future in which humanity can thrive. Public opinion on AI is shifting fast: I hope it will shift fast enough to avert the dystopian future we are heading to.

Maybe a golden age of thought for the machines, but possibly (I would even say likely on the current trajectory) a dark age for humanity.

Chess has always been a game. For other intellectual activities, at least for several people, a big part of the pleasure in engaging in such activities is the sense of contributing something that matters to a collective effort. Strip that away (e.g., because a machine can do the same thing more efficiently) and you effectively make such activities pointless for those people.

In some ways, this feels more like an ominous warning about the times to come than something to celebrate.

According to several recent polls, a clear majority of the population in the US is concerned by AI development. A recent poll by Ipsos shows that 55% of the population would be in favour of slowing down its development. Polls in Canada and Europe show similar views.

Solving hunger worldwide, according to several estimations, would cost few hundreds of billions of dollars per year. Addressing extreme poverty (not poverty in general, I agree) may be in the same ballpark.

I would not extrapolate that easily from smart and intelligent people you may know to a machine trained on the whole internet and on the whole corpus of human literature. Also, some of the worst horrors in history have been perpetrated by arguably intelligent people.

I would also say that the "pet species" scenario is a rather optimistic one, as it presumes that developers of superintelligence will manage to instill some sort of love or care for humanity into their creation. An indifferent, powerful AI may be much worse.

Finally, when you suggest that a superintelligent AI may simply have the goals that we will give it, who is "we"? An AI company? a government? the united nations?

I also did, when LLMs could not count the “r”s in "strawberry" . Now that they can solve Millenium prize problems, I think that we should consider at least the possibility that superintelligence can be achieved.