meowface

Correct. He - I do not think unjustly - has long had AGI as the goal. "Solve AGI, then use AGI to solve medicine, and physics, and the rest of science, and create a better world." has been his thing basically forever.

It's not a given. It's about willpower and care. LLMs are the ultimate crutch. I over-rely on the crutch, more and more people will over-rely on the crutch. But it is still inherently a psychological problem.

You can still know things and get force multiplication out of LLMs, if you are disciplined and caring enough. In practice, most people won't be. And you can't force other people to be. But you can force yourself to be.

Their mindset is: the upside is likely so, so, so immense, and the probability of amazing outcomes is so much higher than the probability of catastrophic outcomes, that the gamble is worth it, and it is not in fact a pure gamble because they can (in theory) try really hard to build it really slowly and carefully.

So even with their earnest belief that there is a > 5% chance this technology could be 1) capable of autonomously killing billions of people during this century and 2) may not be sufficiently aligned and might actually act on that capability, the EV basically still sums positive, to them.

Not saying this is necessarily a good or moral calculus. But it is what they think. There is no conspiracy or plot or deception.

The hypothesis is about theoretical future models, not existing public ones. I'm making a prediction based on a pace of improvement that I believe will probably continue unabated and which likely is not going to hit significant bottlenecks for very long.

You were misinformed about his response to Roko's basilisk. He never thought it was plausible. He freaked out for an unrelated meta reason, basically, and later admitted he overreacted. (Basically he did not want forum members putting attempted mindviruses out into the ether as a general principle.)

Critics of Yudkowsky have been trying to misportray that for years to make him seem delusional. You can see an example of that here: https://www.tracingwoodgrains.com/p/reliable-sources-how-wik...

(Roko himself is genuinely crazy, but largely because he became a neo-Nazi.)

If you think ASI is plausible this century, you would have no reason to not fear the end times yourself. If you don't think it's plausible it'll be created this century or perhaps ever, fair enough, but I'd bet against that.

When money's on the line, everyone hedges.

Yes, I think even in some crazy distant future utopia, humans will still often want to interact with humans for things, and even to pay them for some services. Tasks that include speaking one-on-one and producing works of art will always retain some value to people, even when AIs can one day trivially produce things which most humans find aesthetically superior to anything a human could ever produce alone. People will seek other things out of art beyond aesthetics, like expressions of human emotion.

Humans will enjoy and seek out human contact. This is not surprising. Just, for economy-impacting labor, I think it's almost all going to be machines, with very few humans playing any role. Even an oversight or accountability role. I think there's a 95% chance of it by 2100 and an 85% chance of it by 2066, conditional on no laws that forbid replacing jobs with AI.

Sam Altman's Twitter bio a few years ago was something like "#1 eliezer yudkowsky fanfic fan".

(I personally lean towards "Yudkowsky is basically correct about everything, but we should probably try to build ASI anyway, slowly and carefully". I'm pretty much where Amodei's at. I assume Altman has a similar idea, though is a bit less pessimistic.)

2012 - 2015 seems to have been around the time a lot of people first started taking ASI and ASI existential risk very seriously as things that will be relevant in our lifetime, and when they decided to try to build and/or prevent it.

You could be right, but short of an eternal Butlerian jihad or other extremely strict global government action that effectively bans most forms of AI forever, I think the probabilities of that are exceedingly low.

What you think won't happen in 1000 years is what I think is almost certainly happening within 40 years. I'd bet everything I own it'll be way, way, way more impactful than the sewing machine within 40 years. I think some bits of physical labor may still exist in 40 years, but almost none, and only when a human really wants to talk to or deal with a human, such as in some medical and therapeutic professions, some art fields, etc. (Again, assuming no universal outlawing of AI.)

Oh my god, we have these "it's regulatory capture!" populists even on HN now? I thought we were better than this.

They want coordinated action to prevent a multipolar trap and potential catastrophe by future superintelligent AI (which may emerge quite rapidly if RSI ever hits an inflection point). It's not rocket science. The rules would apply to everyone. The rules they're advocating for are not "only OpenAI and Anthropic can make AI".

There exists a wide range of ideological beliefs in between National Socialism and Maoism.

While Musk is now far-right (albeit seemingly not a Hitlerist - yet), Altman is pretty much a liberal.

Rationalism has spawned many offshoot cults which are genuine and in some cases dangerous cults, but rationalism and Effective Altruism are not themselves cults.

Most people among both ideologies have long believed in the arrival of a machine demigod of sorts, but I kind of am skeptical of the notion that there are any tech people who have not always held similar beliefs in the long run. Like, all the people reading this: you think in a thousand years from now humans are going to be doing most labor? Obviously far, far, far smarter machines will have unimaginably changed things by then, and/or we'll all be gone.

The difference is most people in tech have historically thought it's maybe a century or three away, while the rationalists think it's a decade or two away (and also thought it would be 2020s - 2040s even decades prior to now).

Pinker unfortunately has very little clue what he's talking about here. I eagerly await Scott Alexander's inevitable essay response to him.

Or perhaps they've been saying this for literally decades because they earnestly believe it.

The thing is that you're 100% right but the other person is also 100% right.

It is in fact really really really hard to keep things lean and performant over time as you support a more complex multi-purpose web app, but you still have to try. If you try hard enough - especially now in the age of pretty good agents - you can actually thread all of the different needles and keep (perceived and actual) latency low.

While I detest Elon and the fact that Twitter is now filled to the brim with neo-Nazis and tankies and other nutjobs, unfortunately every single other social media site is far inferior for up-to-date tech and AI things. Bluesky doesn't come close, even if you try to curate things to avoid all the anti-AI/anti-tech people. There is no other game in town, and I expect this will be the case even in a decade from now.

There are dual worries here: human negligence and misalignment of capable AI.

Each side wants to focus on only one. It's ridiculous to not focus on both.

Exactly. Comments are basically its scratchpad for whatever it wants. Ask it to change a number in a TOML config and it'll also add 2 new comment lines above it with some Claudeslop about it being changed and what that accomplishes, as a useless note to itself.

I now make sure to do a big decommenting pass before every PR.

Semver or otherwise, I think even decades ago people understood staying on 0. for years or decades is a bit silly for production software tens of thousands of people use every day.

It has an especially noticeable effect if you're a vegetarian. I'm a lifelong vegetarian, and high doses of it for me are a crazy cognitive enhancer.

PuTTY out since 1999 and bumped to 0.82 in 2024 is pretty incredible

Dario never said they would be. Just that more and more code will be produced by LLMs. All his predictions were in fact pretty much right in terms of months and percentages, give or take small margins.

They're getting a bit better at coding. My prediction is that by 2029, they will be noticeably considerably better at coding than they are today. That will include them becoming less myopic.

I experience all the same issues you mention. I am just predicting where the ball is moving. In the scheme of things, LLMs have been useful for coding for, what, like... 1.5 years??? What other technology has ever existed where people expect it to go from "just came out" to "changes everything for everyone" in 2 years?

In the arc of history I see us at the very, very early stages of AI-driven software development.

>People are actually terrible at making predictions. I believe the future will surprise all of us. But, I’m going to make a prediction of my own…

>In the future we will see more and more companies proudly boasting their “NO-AI” policy as a competitive advantage. And they will be right.

I will very happily take the other side of this bet. Maybe if LLMs stayed as September 2026 LLMs for the next 20 years, I'd grant it's possible. But that's not what's going to happen.

I kind of feel "reduce the amount of thinking tokens produced" would fall under degrading model quality.

In any case, I am willing to believe it's possible something degraded, but so far I have not seen any empirical evidence of it since the previous incident with the inference and harness bugs. I lean towards Anthropic probably not intentionally doing anything like this without disclosing it beforehand.

They have repeatedly said they do not ever intentionally reduce model quality and do not degrade in this way, and that a model version number is always the same.

But, of course, OP is an empirical claim to the contrary, and I'd be curious to see if anyone (who's been capturing data over these timeframes) can replicate the same results and if Anthropic has any comment.

Read any blogs or watch any podcasts from high-skill programmers who are now using AI for everything. They themselves will freely say the AI is doing nearly everything, and that they often are not even reviewing the code, and when they do they just leave it as-is. Every few months you should expect humans to have less and less of an active role in software development, beyond initial ideation and UX sensibility.

I like how all the replies to me are either "no it's the humans who did it" or "no it's the AI that did it".

I think one of the significant recent proofs was basically just one person saying "solve this" "keep trying" "try harder" until it was done. I believe most of the rest can be assumed to be the human mathematicians at the very least providing some useful guidance.

I agree that it won't be that long before the AIs can do basically everything autonomously, though. I am essentially a Singularitarian when it comes to my outlook, even if not for September 2026.

But is it Web Scale.