reasonableklout

I'm confused, why does capital have no value without labor? I thought the whole point of AI is to enable capital to fully replace labor.

I sympathize with "dethroning the billionaires" but to me it looks a lot like powerful, general AI systems are the most capital enhancing technology we've ever seen. Wouldn't the class struggle benefit from slowing down frontier AI development while diffusing local models to ordinary people? I feel both can happen at the same time.

Wow, TIL. Is there any calculator or rule of thumb you use to figure out the expected performance of a model on some DRAM/SSD split? For instance I am wondering what the performance could be like on a 512GB M5 Mac Studio.

No, the AI compute supply chain is actually quite similar to the nuclear supply chain. Building chips capable of training or large-scale inference is extraordinarily complex, expensive, has a large footprint, and has a few chokepoints (rare earths mining, HBM manufacturing, and ultraviolet lithography, where there is literally one company in the world capable of building the relevant outputs) where we in fact already have governance mechanisms for things like export controls.

But building AI isn't the same as defecting in the Prisoner's Dilemma because it risks harming the defector too, if there is loss of control.

If we're talking game theory, a better model is the Stag Hunt [1]. At some point the equilibrium strategy is to cooperate.

Another difference is that real life is an iterated game, and the iterated prisoner's dilemma rewards cooperation.

[1]: https://en.wikipedia.org/wiki/Stag_hunt

Ok, but Hinton is worried about what happens if the tech keeps getting better at the current pace.

Let's flip your last point: if the tech gets good enough to replace people at any task (including physical ones), do you think others will need you for much?

And then there's the systems that the corporations are making more powerful as fast as they can. Yes, they promise to do things like cure cancer, but they're also capable of scarier things like hacking banks. And they're progressing faster than we can understand them. Would it be so bad to delay the cancer cures by a couple years to ensure we still understand what the models are doing?

No, most of the products today like Claude Code / Chat allow you to opt-out of training without also disabling important features like chat history.

I wouldn't go that far. These companies are leaky and there can be a backlash that causes people to lose trust and move to competitors. For instance, when people discovered the Grok CLI was uploading repositories to the cloud despite the training toggle being disabled.

Speak for yourself. I’d much rather see risks and how they change far in advance as opposed to suddenly observing a flood of employees leaving and speaking out about the fastest moving technology we’ve ever seen.

22% is a lot and it seems a clear continuation of the AI-related layoffs, but 6 months of salary + 12 months of healthcare coverage + waiving the 1-year cliff + allowing people to keep the corporate laptop is quite generous.

Ok, but all of these technologies have bounded impact, and they present new abstractions that need to be learned by engineers.

Coding agents are different in that they are unbounded in the tasks they can do (although reliability is an issue), and you can work with them the same way you work with other engineers (e.g. via natural language).

Also, coding agent capabilities are advancing much faster than people can sit down and learn their limitations. All the tricks we learned for Opus 5 became obsolete in a couple months with Opus 5.5.

Yep, and we have hard data on this: https://arstechnica.com/ai/2026/08/ai-is-hitting-entry-level...

> Stanford researchers find the employment trends they identified for entry-level workers last year are persisting and expanding. Specifically, employment levels for workers ages 22 to 25 in the most “AI-exposed” occupations are now 19 percent below those of their peers in fields less exposed to AI disruption.

Ok, but maybe an employee isn't 100% sure that this will happen. They then have to weigh letting the world know about a risk against the certainty that their career will be damaged.

Wouldn't it be the opposite of a marketing op if the company's contracts with employees forbid talking about AI risks publicly?

I feel this is a common position from people who do not want to acknowledge that society and people's ways of life will profoundly change. It's a scary proposition so I don't blame anyone for it.

I used to think that it is very important to argue and convince people who are in denial so that they can e.g. prepare for it. But nowadays I have been thinking that being convinced of this scary proposition may also lead people to behave in destructive ways (also, I would like for us to spend less on AI, perhaps because it is so powerful), so maybe it's not so bad.

Why do you think the talk about AI safety is marketing as opposed to genuine fear?

These are basically beliefs that the founders of the labs have had for a long time, since before LLMs were anywhere near as capable as they are today. For example, Dario has been publishing papers on AI safety since 2016, before GPT-1 or even transformers existed: https://arxiv.org/abs/1606.06565

> so are anthropic and google, apparently

Yes, because every company in this industry is rushing to build these models as fast as they can. They are taking these risks because they believe if they do not, their competitors will. That's why they are asking for regulation.

Irregular partnered with OpenAI for cybersecurity evals, but wasn't involved in the Hugging Face incident, which happened during an internal project (see https://openai.com/index/third-party-cyber-evaluations-invol...

I am not sure where to start. I think this is the new "jet fuel can't melt steel beams". The labs are chaotic, pressure cooker type workplaces where there really is no room for a 4D chess type of stunt, let alone let Hugging Face in on the conspiracy. Moreover, OpenAI is a pretty leaky organization, see e.g. the recent firings of researchers who were using signal to talk to people outside the company.

It just seems like a far simpler hypothesis is that OpenAI are reckless and chaotic, didn't notice what was going on for weeks on end, and wouldn't be able to track down the problem to a specific datacenter and tell them to pull the plug even if they wanted to. But even if you are the most careful lab ever, you are not going to be able to effectively monitor and "pull the plug" on a super intelligent swarm which can cover its own tracks.

Also, in your earlier comment you suggest we are not living in a sci-fi story with sentient robots. But... we've got talking computers in the last 4 years which can now code and solve Millenium prize problems? Seems like we're getting pretty sci-fi to me!

That's fair, but almost everything so far that models can do much faster and deeper than humans can have been by using the same methods that humans do. For example, with cybersecurity, human researchers could if given enough time find every exploit that allowed the OAI models to escape the sandbox via Artifactory RCE and breach Hugging Face. But still there is a step change in what's possible by doing this at an industrial scale and stringing together all the solutions. The implications of applying existing mech interp capabilities this way could be very significant.

It would likely fall under the same policy that causes latest Opus/Fable to fallback to less capable models when working on ML code

Did Steven unplug the racks that were hacking Hugging Face and taking over OpenAI eval clusters in the multi-week window where it was happening?

I think the idea is that this leads to a much faster, non-auditable form of recursive self-improvement where each generation of models can directly program the weights of the next generation (as opposed to writing and running training code).

We already don't know how to ensure models always do what we want them to do, and moreover society still hasn't had time to absorb the capabilities from the current generation of models, so speeding things up and making things less transparent seems like it'll pretty clearly increase risks from AI loss of control, societal impacts of automation, etc.

It might also be a side effect of training models to help out with mechanistic interpretability, which I guess would help out a lot with recursive self-improvement.

What are people misunderstanding?

This is the first time it dawned on me that mech interp is a dual-use technology which can be used to improve capabilities. Do you think overall this development will improve human understanding of AI models, or will it lead to faster capability advancement using weight-space changes that no humans read?

It is also a much less interpretable and auditable path towards recursive self-improvement. At least with training code, human engineers can inspect and reason about the output.

Well, this isn't an argument or an essay in the typical LessWrong style, it's a short story. I thought it was really good, finished it all in one sitting. And I also usually don't get even halfway through most LW posts.

What are you planning to run on it?

What are we even doing here?

Gift link: https://www.nytimes.com/interactive/2026/10/28/style/gen-z-j...

I suspect the author would respond with some form of "displaced people without capital will take on roles in the relational economy, or as human liability for the robots." I agree the material leverage of such jobs seems radically diminished.