jackb4040

Oh no, that's terrible! Please send me an example of one of those devices so I know what not to buy.

I would not assume the invisible hand of the market is gonna solve this problem for us. It will be tricky to get this problem to even register at the labs, who are worried about existential risk first and paying (coding) customers 2nd.

Because their leaderships have defunded / fired all ethics researchers looking into their ongoing crimes and failures in favor of a bs "AI safety" narrative based around a creepypasta about an AI monster time traveling backwards to torture their staff for not maximizing shareholder value.

> "get their hands dirty" > "insurmountable wall"

This is gibberish, you may as well tell me you've found a load-bearing seam.

Yes, diminishing returns. Not overall, they've still been able to create more intelligent models even up to today. But the strategy for scaling that intelligence has shifted. From the initial ChatGPT release to GPT-4.1, they were basically scaling up compute training compute / model size. Then 4.5 flopped, while o1 demonstrated that gains could continue by reasoning (scaling up compute at inference time). o1 is now the ancestor of all their flagship models from GPT-5 on.

This is why I'm trying so hard to drill down on the theory of scaling, and not just talk about improvement in general, hand-wavy terms. If the bottleneck of current scaling strategies is training data, or something fundamental about the model architecture, then just throwing more harnessed chatbots at it won't lead to an exponential increase in performance.

Now you could argue that the AI we have now will help us find that change in architecture, and I would agree. But that means we're firmly outside the singularity for the time being, and what people are in fact talking about is a hypothetical.

Boy I could not disagree more

Millenials were in college during the Great Financial Crisis. They're not exactly an example of a generation whose aspirations were met.

You're insulting the intelligence of everyone on this website by even attempting this argument. The handful of upset elections by young socialists who want to decrease the political hegemony of billionaires does not amount to evidence that the political hegemony of billionaires isn't real.

The enshittification model, but with an extra step because they actually have to employ labor.

First they exploit their own gig workers, to make an unsustainably cheap product and corner users. Then they exploit users, to monopolize distribution and corner restaurants. Then they exploit restaurants, threatening to cut off distribution entirely if they don't get an increasingly large share of profits.

Came here to post this. AI rationalists love thinking about paperclip maximizers, but don't seem to care when it turns their own codebase into paperclips. Or to rephrase, turns their whole engineering org into meat proxies, slowing down engineering productivity in the long term because understanding is drained out of the staff and flushed down the drain every time they close a Claude Code tab.

Oh thank you, yes that makes way more sense.

Again, can you lay out your theory for how intelligence scales? You're using a lot of terms like "full autonomy" without definitions. Why do you think that just throwing more harnessed LLMs at (something?) will lead to an increase rate of improvement?

I feel like I laid out several cases where other things were the limiting factor on improvement and more agents wouldn't have helped, and I didn't get a response to those cases.

What "they project" (the labs) is of minor interest to me. Aside from their incentives and track record of lying, in recent months they are laying out a story that is pretty much just the plot of Terminator, and directly referencing rationalist beliefs that were published long before LLMs even existed.

Sorry, yes I agree 100%. I don't agree with their narrative, I was just explaining it. I think it's fraudulent and based on science fiction and will lead to a significant economic crisis.

OpenAI/Anthropic have never pitched themselves as a replacement for farmers. They do explicitly say that they're going to cause significant job loss in knowledge work sectors all the time.

How is this different from arguing that Microsoft Clippy was RSI? An AI tool being involved in the process of work can't be the bar for RSI.

I don't think there can be a coherent definition of RSI unless people lay out their theory for how intelligence scales. LLM-assisted coding is great but respectfully optimizing pytorch features or whatever is not gonna lead to exponential improvements. That approach to scaling diminished years ago, leading all the labs to switch to reasoning.

Now it seems reasoning is also yielding diminishing returns, so all the labs are pivoting to specializing in particular fields like math / infosec / biology. They're improving due to accessing new proprietary training data and doing RL with human experts. Again I don't really see any amount of "AI research interns" leading to an exponential improvement to this strategy, they're not the bottleneck in the first place.

If it requires humans in the loop it throws cold water on the hopes of replacing millions of jobs, which is priced into their current valuation.

A chatbot for cancer researchers to talk to is worth single-digit billions at most. Anthropic is already valued at over a trillion dollars, on the premise that they can replace the majority of jobs in most knowledge industries. All the announcements about hacking / math problems / biological science are meant to create the impression that that strategy works and is repeatable across industries.

Every perpetrator of genocide throughout history has said exactly this. If we don't do it to them, they will do it to us! It is nonsense in every case, and only speaks to the genocidal worldview of those who believe it.

That's theater, the government and more important the defense sector are using Anthropic products like crazy.

> highly entrenched state crony capitalism

If your goal is to rescue the reputation of capitalism, people have found more success just calling Donald Trump a communist.

I think it's a bit reductive to compare China taking inspiration from America in the 20th century to Nazi Germany's explicit admiration for slavery and the genocide of Native Americans.

This is why I don't understand why everyone's freaking out about it. By far the biggest problem with LLM classifiers is that they treat every individual business as the blurry average of all businesses in their training data. Being lighter is fine if you control for everything else, but at least at my company we would actually have room for a significantly more expensive / slower classifier if it were demonstrably better at following instructions.

Jev, ASI, RLHF, Jev, Jev, water usage

The government is sending people to death camps whether you like it or not; there is a set number of non-Jews being sent there regardless of your moral choice.

Would you rather have someone make good software that does not misidentify non-Jews as Jews and send them to the death camps? Or should we outsource this to worse engineers who don't care about morals and don't find this distinction meaningful?

Yeah but if I didn't write the software that killed 123 children, somebody else would've anyways!

At least I can donate some of my salary to progressive politicians! I don't, but I hypothetically could, which makes me a good person.

I don't think that's the example you want. Left to their own devices the Germans would not have held themselves accountable. As we speak the most popular party in Germany is a holocaust revisionist party.

That's because the Hugging Face incident wasn't intended to be an example of the real, present dangers of unchecked AI use in the real world. It was designed to demonstrate the unlimited power (and thus value) of their models, and thankfully it failed to do so.

AI labs don't stand to benefit from the tech being understood as dangerous for its lack of intelligence. They need it to be understood as dangerous like an intelligent alien civilization.

Well in theory the military is deferential to a political class that is elected by and thus accountable to the people. But that is so obviously no longer the case that politicians who pretend it is are perversely less popular than ones who are openly minoritarian.

The logical response to such a self-conscious rejection of democracy is unironically political education and beginning revolutionary political organizing in your neighborhood. But I rarely bother making that point here; HN is probably the least revolutionary-minded grouping in the world outside of tech billionaires themselves.

You do know there are schools INSIDE american military bases right? Would you respond in the same way to Iran annihilating a school in Florida during classes?

It will gradually introduce tumors into every codebase, every academic institution, every board of directors, every intelligence agency. But hey, at least no one will suffer reputational damage!