kypro

I do agree with this in some sense. I personally don't think we'll make it to the AI utopia era, but assuming we do, I agree there will be relative resource abundance.

The problem is we're already in an era of relative resource abundance and hunger is still problem, and the super wealthy who have almost unfathomable levels of resource abundance are still doing everything they can to pay as little tax as possible. And this isn't just an issue with the super-rich I myself am no better, and if you're honest, nor are yo. I am relatively very well off, but do relatively very little to help kids starving to death around the world. Somehow I rather take my family on holiday.

But okay, let's ignore this too. Let's just assume we'll be less selfish in the future or something.

Well now, we have another issue – AI has a cure for aging and populations are now exploding. Every beach and tourist attraction has a decade long waiting list. Huge amounts of green land are being built on to accommodate all the unemployed humans who have nothing better to do than have kids. People start to see that the human race resembles something more like a parasite, endlessly spreading and consuming while contributing nothing.

So now what? I'd argue the incentives to provide infinite resources to the economically useless human race doesn't make sense for the AIs or humans in charge. Absence of poverty and disease you simply have to do something to keep the numbers of human parasites under control and/or limit their consumption.

> If it takes 100 years for all jobs to be done by AI we'll have plenty of time to adapt.

How do we adapt an economy in which humans are economically worthless?

The world will just become sub-Saharan Africa. Humans could choose to end world hunger but we much rather selfishly consume. Most people can't even be bothered to look those begging on the streets in the eyes.

I hope you're right, but it would be unusual for humanity to care so much for such economically useless people.

The fact that AI is not a normal technology seems so obvious to me that while reading this I was questioning why someone would even take time out of their day to write this – everyone know's water is wet, what's the point here?

But I there are people who still struggle to see what's happening. When I say that school is a waste of time people are shocked people in my family are shocked and get mad – "how is Timmy going to get a good job if he's not got good grades".

But there will be no jobs, and any jobs that are left won't be dependant on the application of someone's inferior human knowledge or intelligence.

It's getting to the point now where I'm starting to suspect some people are just not wired to accept certain very bad things. When I first started worry about these risks it was reasonable to not see what I did. But today when it's so clear that AI progress isn't going to slow any time soon and there no reasonable moat for human intelligence people can point, I increasingly don't even know what we're arguing about anymore. It's like when someone is diagnosed with stage 3 cancer and the family somehow else think they're going to "fight it". No, they're almost certainly dead and you should accept that reality however awful it is. But if you try to say the obvious they'll find out that it's weirdly hard to convince people that cancer is almost certainly going to kill someone. No amount of evidence or past examples you point to will convince them, they'll just keep saying irrelevent stuff like "but he's strong", "I know he will be okay". We're at this stage with AI now. It's just exhausting trying to reason with people who seem to refuse to things which are so self-evident.

If you want to limit the power of companies you need regulation...

I agree with you that we can't allow a handful of people to hold this power themselves, so what are you arguing for here? That the answer is to just leave AI labs to carry on as they are?

The government should be ensuring that the technology is being developed with caution, but since the government seems to have no interest in that, this is at least better than nothing.

Your solution of giving everyone access to super intelligence is absurd. A good guy with super intelligence isn't going to protect you from the bad guy with super intelligence. If someone uses it to poison your water supply with a bioweapon or to take out power stations, you and your family will be dead long before your super intelligence hacks together plan to stop it.

You're being downvoted because you clearly have the wrong opinion. If you want upvotes here, please share a conspiracy theory about why the CEOs of AI companies might want to slow down.

It doesn't need to based in reality or even reasonable.

> because their improvements are stagnating as they hit financial and technical challenges.

I don't mean to be rude, but how is it even possible for someone to hold this opinion in mid-2026?

Can you explain why you believe this?

Yes. An effective strategy to win at any test isn't to cheat the test by hacking into Hugging Face, etc, but to seek control over those administrating it.

Or in other words – all goals are more likely to be achieved with more resources or influence over those with resources.

Hacking into infra across then web and creating copies of themselves so they can't be easily shut down, then doing stuff like hacking into power stations and threatening to shut them down unless humans give them top marks or do what they want is generally good strategy for any AI system.

This applies to bio-risks too. If I were an ASI and someone prompted me to kill humanity, my first step would be to gain access to the computers of phones or workers at biolabs and find their darkest secrets. When I have a good individual I can manipulate I'd then create a super virus and force then individual to help me produce it or ruin their life.

People massively underweight the risk of AIs having super-human hacking abilities in our modern world. It gives a capable AI almost unlimited safety and leverage.

China isn't as unreasonable as someone people like to make them out. Xi isn't crazy and I think would entertain cooperation if he felt it was neccessary.

The issue right now is that China seems to be far more optimistic about tech and I'm not convinced they see the risks. I think that time will come however, and we should start dialog now.

> OpenAI / Anthropic models have largely stopped advancing - that's not a good look when you're pre-IPO and Chinese models are catching up.

What is going on in this thread?

Are these even real comments?

While I strongly agree on the need to pace the frontier, I do think there are reasonable arguments that could be made against it.

Non of those are being made here though. It's just speculation that the CEOs of the fastest growing companies in the world are somehow obsessed with regulatory capture to maintain their lead, and have been playing the long game by talking AI risk for 10+ years so that when they become the CEOs of leading AI companies they can push for regulatory capture.

Perhaps they're all just genuinely concerned?

What would regulatory capture even give them? Who are they so worried will compete with them? Google, Mistral? Even if it's Chinese AI labs, it seems odd for anyone in the West to be opposed to any regulation that would slow them.

So assuming they are obsessed with regulatory capture – what's the argument for why they would want it?

> For the first time in human history, any member of the working class can just about afford to have a team of expert scientist/physician/lawyer/engineers working directly for them.

It's crazy to me you would be able to say this without acknowledging how dangerous/bad that is.

Do you acknowledge there's is significant risk in concentrating so much power into the hands of individuals?

And what do you want Dario to do? Just keep pushing the limits of AI capability while humanity crosses it's fingers and hopes a swarm of AI agents don't do anything more than hack a few companies?

For all my criticisms of frontier lab leaders, I'm still somehow genuinely thankful that individuals as troubled as Elon Musk are calling the shots and not the average person on HN, who since 2023 have refused to acknowledge even the obvious risks of the technology being built – in some cases even when there was direct evidence of the risks many of us are concerned about.

> The main appeal of RN was being able to leverage your web devs for mobile dev.

> But now that all code is generated, there's little upside to having an RN app... just start native. Your devs are barely going to be writing code anyway.

Are you saying:

- webdevs are now able to write and review native code because of AI (who cares if they don't really understand it); or,

- Because developers are more productive we can cut the number of webdevs and hire native engineers – same no. engineers, same output, but now native apps.

Why not:

Continue using RN but now and just enjoy being more productive? If productivity was the reason to pick RN, then enjoy it. It's not a bug.

I think OpenAI are correct that it's worth noting, but realistically any relevant usage data they have and used to improve their models would be very insignificant unless they were deliberately using logs from other researchers and training specifically on it (which they seem to deny).

The fact the proofs differ suggests that the models were not directed to be particularly focused on that avenue of research nor trained to converge in that direction.

I get the scepticism, but I feel some of the accusations here are bad faith.

> Genuinely grappling with the implication of where we're headed is usually very crushing.

As someone currently prepping for various AI doom scenarios and who has been dealing with AI-related nightmares for years this is very relateable.

Although, I don't personally think it's this. In my experience the opposite is more true – the majority of high probability doomers seem rather laid back about considering what they believe will happen to the people they love in a few years. Equally I don't get the sense those who don't have such extreme predictions are worried at all. If anything there's not enough emotion.

In my opinion people just don't reason well when it comes to exponentials and are ignorant about things they don't have good mental models of. At least I know I struggle with this.

While some neural network architectures can be designed to better fit certain tasks they are at their core general-purpose learning algorithms which can approximate any target function.

> I feel like I have a decent conceptual grasp of what LLMs are doing with written text

Just think of it as input data. In theory it shouldn't matter what each token represents. They could be xbox controller buttons, image pixels, or text.

The model with enough training data will map those inputs to an expected output.

As someone with a background in AI and who has been playing around with neural nets for decades at this point, it's been genuinely amazing watching extremely intelligent people make confident predictions about AI capabilities and progress, then be so completely wrong.

There's a kind of theory of mind for AI (specifically neural nets) which I now realise I seem to have which is very hard to explain to people who haven't felt the magic of these algorithms. In fact, the algorithmic details almost doesn't matter at all. When you have a generalised learning algorithm really the only essential components are – compute, data and time. So long as you can scale these you can be certain you will also scale capabilities. There is never any exception.

That said, the capabilities neural networks tend to progress in step-functions rather than scale in correlation with compute, data and time, because algorithmic improvements tend to come every ~5 years and bring a significant step change in capability (or efficiency depending on what you measure).

I think people like Dario and others working at frontier labs see and understand this very clearly. And I suspect it's also why they worry about AI risk because even if you ignore the significant increases in compute and data these models are being trained with, it's concerning that it only took two real algorithmic improvements to take us from mostly useless predictive language models to AGI-level intelligence – and we're due another step change.

It's not really a commodity when Cloudflare is often cheaper, simpler and provides a better service compared to it's competitors.

For it to be a commodity there would be no good reason to pick Cloudflare over any alternative and this absolutely isn't the case.

This is so it can restore any unsaved changes.

I take no opinion on whether this is good or bad, but I can see how from a UX perspective it's nice not to lose 10 minutes of work because your browser crashes or something.

> We don't have any proof of that, but the approximator thing is proven.

Not in the way suggested... It's not approximating human intellect. They approximate the target function, and the target function frontier labs are trying to approximate is ultimately a super intelligence...

> Synthetic data derive from other linguistic data. Whatever intelligence is in there, it is expanded horizontally, not vertically

I'll assume we're talking purely about language models for a moment, but if you assume that everything can be represented linguistically, then in theory there is no upper-bound on what can be learnt with synthetic data.

It touches on it, but it doesn't address it.

They talk about "value alignment" but fail to define what those values are – is it aligned to the values of the US government, or are they suggesting they want to build an AI with it's own values so it can decide for itself when and how it will intervene in wars and other human affairs? And again, is an AI which disempowers humanity in this way aligned? Many would say no, although as I argue, disempowering humanity is probably better than the alternative if we can assume it's roughly aligned with our interests (which we obviously can't because it's super intelligent, but that's another issue).

Fundamentally the problem here is that humans don't have an aligned set of values you can align an AI to.

The moment you start defining what alignment actually is in practise you simply must accept it will be unaligned with the values of others. There is no getting around this and hand waving around the issue isn't good enough.

They should tell us explicitly what they're trying to build.

I've quite dramatically brought my predictions for humanoid robotics forward over the last few weeks.

For a long time I thought decent humanoid robots were something that wouldn't come until maybe the 2040s, but watching some of the content from the Chinese robot olympics recently and seeing many promising robots projects which are incorporating LLMs has made question that.

It seems quite likely that within a few years we'll have humanoid robots both which have both the dexterity and intelligence layer to start being useful in the real world... Obviously at first they'll just be lifting things and doing other trivial physical tasks like folding clothes, but every year they'll be able to do a little and a little more.

I think it's now is quite likely that within a 10 years we could have extremely competent robots which can do most of human labour. 15-20 years just seems way too long at this point.

No offence, but you clearly haven't studied this and are making some wrong assumptions here.

> Neural networks are approximators being fed human intellect.

They're not "approximators", that's a far too simplistic way to think of them.

Neural nets create models and deep layers of abstraction around the data we feed them in the same way your brain creates layers of abstractions to reason about the world. AIs can use these abstractions to come to come up with novel things no human has ever thought.

> Therefore they can only approximate the intelligence of humans

They're not just being fed human data though... Modern AIs are typically trained on huge amounts of synthetic data. This is why AlphaZero got so much better than humans at chess and Go - they're not just trained on human data but they generate their own data and train on that. Similar techniques are being deployed on SOA language models too.

> Even if the llm speaks an alien language, it should be similar to human intellect

Is AlphaZero similar to a human chess player? There's no reason to assume this.

It's actually worse than this because it assumes alignment as a concept even makes sense. For example:

If the the Chinese government asks their ASI to create a bioweapon against the West, should it? No, presumably not – an aligned AI would be one which disobeys the Chinese government even if they created it.

Okay, so what if the US government asks their ASI to help it in one of their wars instead? Would an aligned AI kill humans on the order of the US government? No, again, presumably not.

So what have we have we even created here? An AI which is more intelligent and powerful than us which also doesn't take orders from us?

Is this what most people thing of as alignment and is this what humanity actually wants?

We should stop using the word alignment. It's a BS term for a concept which simply cannot make sense if alignment is both to mean an AI which we control and an AI which will not harm us.

Also theres not multiple trillionaires in the world, and the one trillionaire we have did that largely by innovating and creating new stuff rather than just leveraging existing OSS for profit...

> The Internet used to be like that.

Not really... Even back in the Web 1.0 days people didn't visit the same websites unless they had the same interests. And even today this exists – I'm seeing the same HN homepage for example, and this is true of most sites.

I think what you're describing was much more of a thing in the 80s and 90s with TV. People genuinely would tune into the same programs as each other every night and have these collective cultural experiences which don't quite exist in the same way they used to.

I don't miss it personally, but I suspect us losing that has led to some of our cultural divides in recent years.

I'm not the global shared experiences you're describing really ever existed.

I care. I've been feeling bad for all the people who have dedicated their lives to that craft. It took years of daily practise to be good at 3D modelling...

When Astra becomes wildly available it seems likely a single 3d modeller (or even someone with no 3d modelling knowledge at all) could do the work of 10 today. Those guys are going to find it very hard to find work.

Yes exactly. I think we all understood at a high-level what it was doing.

The issue I had was that it was doing a lot of weirdness like using `Object.defineProperty` to try to force an existing library to do something it wasn't suppose to do. No human would have written that code.

Although I generate most of my code with AI I still personally try to keep it understandable by a competent developer. Where as most people on my team are not good at FE so they just trust the LLM to get it right. But ultimately it just doesn't matter anymore.

Actually, the world kinda does work like that.

Most intelligence researchers would agree that people seem to have a genetic cap on their intelligence. While someone can underperform their intellectual potential with an upbringing that doesn't adequately enrich their minds, it's near-impossible for humans to become more intelligent through reading, studying, etc.

When humans learn we gain knowledge, not intelligence.

I think the only real difference is that we humans are born lacking a lot of initial knowledge/data which means we have to go through a decade or more of education to reach our potential intelligence. LLMs on the other hand come pre-loaded with that knowledge.

Passed this point, wherever knowledge is passed in as context or stored in the neural net I don't think is that significant personally. I'm of course not suggesting we're exactly the same as LLMs and there is no noteable difference, I just don't think continual learning is as important as some suggest it is – at least assuming a model is deployed with adequate training such that it reaches its potential given it's size + architecture.

As someone who loves FE the slop people are pushing where I work is really quite upsetting to me.

I reviewed some code earlier which clearly wasn't intended for consumed by humans and I know with certainty this dev wouldn't have understood 90% of what that code did. It looked like hours of Claude trying to force square into a round hole tbh, but it worked.

Was I going to reject the PR because of my code preferences? No, of course not. Ultimately it doesn't matter anymore. If Claude can maintain it then who cares... Even if it's broken (which it didn't seem to be) it's just FE anyway so not the end of the world.

I wish it did matter of course, but like most of coding these days, it just doesn't.

> On the other hand, we must be aware that these models are static, and they indeed stop when nobody asks something or requests an action from them

The parent commenter noted:

"if a harness alone can help the LLM saturate the benchmark with a near perfect score then the combination of the two must still be effectively AGI"

Harnesses absolutely can enable models to continue thinking about things. And LLMs do wonder and explore weird ideas like daydreams when you allow them to do this.

4.1 was the first decent 4-series model. It was significantly better than previous generations at tool calling if I'm recalling correctly.