The gap in shared understanding of LLM capability is widening

Depending on the type of output, the value immediately (?) goes to the electricity cost of the LLMs and the cost of materials. So: more “efficient”, but without a big margin, right? So its cheaper for the workers that remain working, other workers are displaced, and capital owners now have a super slim margin business which can be instantly replicated at a place where material cost are lower (?).

Thus, it almost seems like “value add” by humans would in this simplified view be vastly diminished, and all that remains is resource wealth. Pop the materials in on one end and let the AI manufacture the product. Great news for the third world assuming the USA doesn't invade them.

Also, whose the consumer? Or are we going back to a strange medieval time where consumers are not of concern?

I don’t know what to think

Most peoples jobs do not involve hacking other services in large scale fuzzing/whatnot attacks.

> Whenever someone finds out how to apply this level of agentic sophistication to a problem that actually matters to consumers, then you'll have that monument that everyone can see.

It does not seem to me that tech industry or its investors care about consumers even a little bit. Consumer preferences dont matter once you have enough or raw economic power, I guess.

Isn't this a massive alarm bell? That the capability hasn't done much for consumers yet that isn't basically toys? Although I suppose the pitch was always B2B "you can automate away your expensive white collar workers" and it's not clear how widespread that is actually happening.

so, scamming your customers?

They are proving useful for the parts of those things that have that verification, but in neither case is that the whole of the field. In maths, for example, we have seen LLMs find answers to some interesting problems/conjectures, but haven't yet seen them pose any similarly interesting problems/conjectures themselves because the criteria for a "good conjecture" are unclear whereas the criteria for a resolution of an existing conjecture are usually supplied by the conjecture itself.

How would you quantify this?

Some of the differences were:

- Mythos recommended salvaging damaged labels. I pushed back: data whose structure is broken shouldn’t be trusted

- My simpler more readable test caught a regression that Mythos’s more elaborate test missed

- Mythos allowed invalid values, citing real-world data that I could not reproduce, while I decided to follow the spec

- Mythos kept the old error types. I made every kind of malformed input raise the same error, and accepted a small, documented breaking change for anyone who relied on the old ones

Overall, I much preferred the final version of my PR to the initial draft Mythos provided. However, getting there involved quite a bit of time messing around with the code, as well as some back and forth with Opus (which was valuable).

This is a recent example that matches my overall experience working with current SOTA models (the ones mere mortals like me have access to).

TL;DR: I don’t think AI already has better judgment than (most) senior people. Obviously, that might change with future models. But that’s a hypothetical scenario. It might as well not change anytime soon.

It has better judgement in math and coding, which have infinitely generatable training data; if you ask it a domain-specific question about something like sales or marketing, its answer is often horrendous.

The suggestion from frontier models was to have our service check for this flag and use that to decide whether to trigger this behavior, even though the flag had nothing to do with this behavior semantically. I'd expect this from a junior engineer. Correct now, but almost surely to cause a bug in the future as the codebase evolves.

These tools are incredible and can achieve things I would never have imagined five years ago. I have yet to see their judgement exceed mine.

No offense, but if you noticed it has better judgment than you, consider that you might not be a "senior people in a tech domain"

Today I gave a free tier AI a photo of a burning ship posted on Twitter, and asked it to confirm it was an oil tanker. It said no, it was a bulk grain carrier, and gave a specific ID. I pointed out that the bulk grain carrier it had identified was painted black, not red (like most liquid carriers) like the one in the photo (as well as other obvious differences).

That is just one example. The free tier models continue to make very dumb mistakes all the time. Maybe the top secret models, too dangerous for the public, are able to distinguish a black ship from a red one. Who knows.

So now it's just a question of, do you sit on your advances or release them? Why would sitting on them be the 'right thing'?

My example comes from maintenance in machine shops. My dad's siblings were almost all employed in machining in some fashion. One uncle was maintenance for a local large company. They decided they could move maintenance from in-house specialists and contract that work to a multi-state massive company; anyone can follow a maintenance manual right? All you need are the proper certifications and everything is cool, right? The cost was less, and the number of maintenance staff on site actually went up. Great for the owners.

But what they lost was consistency and understanding in the process that eventually led to closing the company. My uncle was there every day all year long for decades and could hear and feel in his feet when something was going wrong with a machine regardless of the maintenance schedule approved by the company that manufactured the machines. The new outside company didn't pay as well, and didn't have great benefits, which meant that they cycled through staff very regularly. No one was there consistently for more than a year or two. Even though they were technically following the manufacturers recommended maintenance schedule, machines went down. Machines then stayed down for days or weeks longer than they should've, because the expertise to repair either wasn't there, or the person leading the job would quit and they would have to cycle in a new maintenance person. They lost jobs, customers took work other places. 8 years later the company closed.

There are lessons in this kind of thing, but nobody learns them.

I do think they are close enough to the stage where people will be replaced, even though it's not the right or correct thing to do, and that we need to move on from the "denial" phase quickly enough so our societies can adapt / regulate / legislate while not too late.

It kinda reminds me of the story of the Toyota process , while their car product lines were/are heavily automated the process still relies on the smalll teams on the line to spot/solve and iterarte over mistakes/inefficiences then add the refinements back onto the automated tasks.

Similarly althose stories of radilogists going jobless have not panned out , as interpreting slieds is a 'task' that a radiologist does.

Semantics , yeah but it does shape the way of thinkings regarding jobs/economy and not to mention the whole anthropomorphizing by the AI labs and boosters. In essence the wordings can/could introduce a healthy dose of reality/skepticism to the wildest predictions regarding job losses etc.

Sometimes it’s in my favour. LLMs estimate very high prices and long timelines for software projects.

If it’s a legal / regulatory issue it’s usually wrong.

Without additional explanation this is indistinguishable from the role of a technical product manager.

But I assure you it's even "better" (not really but the task is so much easier) at making your spreadsheet look nice and compare actuals vs forecast for your budget than a complicated programming task. It will just one shot it, and it will be objectively correct, all in a very short time. But these non tech people don't know they can use an agent, they have asked one question in the chat interface with 0 context about what they're trying to do, and conclude they are not good enough.

I'm not saying everything is replaceable by AI - and by the way I don't want anything to be replaced by it, but I believe it will happen - just that some people don't realize yet how "trivial" their job is to modern models.

This is the level that many non-technical users are operating on.

There are only a few professions that are intrinsically precise with language.

No surprise we're seeing adoption in software programming and mathematics.

So this, I think, is the crux of it.

When most people, at least in the world of SWE, think of what it means to be skilled at something, they think of knowledge and experience acquired over a much longer timeframe - months to years even. So from that pov, something you can get reasonably good at within just a few hours or days might not seem like much of a "skills moat" at all.

There still seems to be a wide gulf in automation capabilities and things that require human dexterity, such as doing things for the infirm. Even just cooking and laundry seem quite a bit away. I guess the original tweet does mention this:

>Capability peaks in domains that are digital, verifiable and economically valuable.

Driving is the most impressive one so far, in my opinion.

In my case, for some types of work - which sound similar to things you mention - I started slowing AI down intentionally. Software development, if it's something simple/common and thus "in-distribution", LLMs can zero-shot complex projects just fine. But in more R&D work, I found that errors crop up quickly and accumulate, and end result of "go and just make X" is usually bad.

Now, the problem I found with usual workflows for this kind of work is, agents race ahead and very quickly, I can't keep up anymore. So I reworked my workflow to make AI work in small steps optimized strictly for maximizing my ability to validate the outcome, and minimize time I spend doing it. So now, the thing (say diagnosing a performance regression of some complex codebase run on a cluster) may take me a whole day, where "unchained", agentic swarm would do it in 30-60 minutes - but, at the end of the day, my method is near 100% guaranteed to be correct, with every intermediate step vouched and reusable for future work, whereas "agentic swarm" approach, that hour of work, 90% of the times I'd just skim and throw away and reprompt, because I couldn't even begin to guess which part is correct, which isn't, so in the end, it was just a fast way to waste lots of tokens.

I see many people have complex systems involving skills, agent swarms with roles, agent factories, 20 termux panes and whatnot. Maybe that's the way, I don't know. Nobody knows. But it's not necessary to start getting value from LLMs. I therefore recommend my "lazy person's basic approach":

- Throw a task at LLM

- Check result, see what you like or not like about it

- If you just can't keep up, ask it to do smaller steps; if you don't understand, ask it to explain

- Just do that a lot, and don't be afraid to make both absurdly large and absurdly small asks

- You'll quickly get a feel for where AI can help you, and where it's not yet worth the time handholding it

I'd imagine just that could already be pretty useful for your line of work ? That does require to have a claude subscription and using Opus 5.5 for it to be worth it or cost efficient.

Then you can compound this: give it access to more of these docs, have it memorize, create skills on where to find what, then move on to more and more "vague" prompts and you'll notice it knows what to search for, where, and how.

Quickly enough you can ask it to generate an artifact explaining really complex stuff super succinctly and gain a ton of time. You can ask it for links to its sources in your docs for each point it makes in order to easily confirm it doesn't make anything up etc...

But to reach that point, you might need to go step by step so you know you can trust it because it didn't fail you on the smaller, easily verified steps you tried early on.

Compare that to humans. How often do you think humans get it right the first time? Almost never, for sufficiently complex problems. And how often do humans get the overall result wrong, without realizing it? Also, almost always for sufficiently complex problems, or sufficiently ordinary people. And what percent of the general public would be able to get even one question right in a problem set based on Math Olympiad problems? What percentage of mathematicians are capable of solving even one Millenium Problem?

Furthermore, you have to take into account that AIs do not particularly use output to input architectures. I frequently see self-corrections to plans when I'm developing using AI assistants, and plan changes are sometimes major. ("Oh I see. I need to revise step 4 of the plan." -- Claude paraphrased). Clearly current-generation AIs use more than input to output feedback; they also use supervisor/client strategies of AI and collaborative networks as well. And often provide self-correcting infrastructure, varying from "Claude: Oh. That generated a compiler error. Let's fix that.", to "Claude: One of the test cases is failing. Let's see what's going on.". And that self correcting infrastructure allows AIs to try again and again until they succeed. (Not unlike what humans do when solving difficult problems).

As a member of the 0.2% (but not the 0.00006%), I have to imagine that frontier models, when applied to research topics, are using most of those 10,000 instances to investigate alternate hypotheses in parallel. Which is also something that humans do across entire disciplines. Based on my personal experience with difficult technical and commercial and scientific problems, I sincerely believe that most breakthroughs are not made because whoever made them is more brilliant than the other hundred or thousand or dozen qualified people who are working on the same problem, but are made because they happened to be lucky enough to start with a somewhat random hypothesis that luckily turned out to be the right one. In case you were concerned that this is a sour grapes point of view, yes, our team was the one that came in first.

So even 90% may be a very big deal, for problems that human get right about 0.1% of the time. Or even for problems that AIs get right about 0.1% (but have enough instances available to pursue 10,000 hypotheses in parallel).

p.s. Ironically, it really looks like humans score significantly less than 90% on the "is 90% good" problem.

The marketing message being pushed by Anthropic (& also OpenAI) is that these narrow math/hacking/coding skills are representative of the model's intelligence, when that seems to be the exact opposite of the truth - these narrow human/super-human competencies are entirely atypical of the models more general capabilities.

I think what people (sometimes) miss is that building something is one part of the work, but keeping it running, developing and refining it is a whole different story. And I think all those vibe-coded software will not stand the test of time.

Photoshop is 36 years old now and it wouldn't be the most-used, most-established solution for professionals if Adobe didn't do great work over all that time.

Not really rooting for Adobe here, dislike their subscription models just as anybody else, but I think you'll have to recognize that they kept it running and never stopped.

It seems we’re perpetually in this state.

To torture a different analogy, we're all prospecting and yes i can see that the labs have the option to do strip mining which I dont. But I have tried various forms of bulk processing.

The samples i've collected point to the fact that strip mining just adds in more dirt, and i'm asking if there are more positive samples or if you've inferred their existence.

Now even without that, another anti-signal to me is that we have 1 unquestionable result afaict which is the openai math proofs; and they didnt even take the time to wash out the dirt (https://github.com/offline-ant/wiggums-proof-loop) and the trick of scaling really isn't all that mind-bending beyond increasing an integer somewhere to gamble with more money.

I'm sure there are 5000 people experiencing something unique and categorically different with dealing with problems, but unique does not mean they're the place for a glimpse into the future or holders of exclusive insights. So I'm questioning the use of the separation.

Well, the use beyond establishing a flattering hierarchy of course.

I found out due to a live bug that Apache strips out the response body for 5xx HTTP codes for security, and it's not configurable, so you can't use any of those if you want to return the error message to the user. Don't know what else might do the same, so nowadays I only use HTTP error codes for protocol-level issues, and application-level issues all go into a 200 OK with JSON just like that.

Every once in a while, everyone needs to touch grass.

The symbol rate was 1Msymbol/s and there were 66 taps. The circuit (in an FPGA) ran at a clock rate of 66MHz. In each symbol period the error was calculated based on the output symbol, then an iteration was done over all 66 coefficients, using the error to update them.

Edit: I also remember that to make most efficient use of the FPGA I figured out the minimum precision required in the maths whilst preserving convergence. It was a 4-bit mantissa plus an exponent (3-bit or 4-bit exponent, I can't remember). It's interesting that today's AIs seem to have also figured out that things still work if 4-bit floating point is used, though that is 4-bits including the exponent?

I feel like this statement is too anthropomorphizing. We don't teach a computer anything. We write normal computer programs that can process data(training), turning it into what is essentially a configuration file(weights/parameters) for another normal computer program(inference engine). Upon receiving a request, the inference engine is invoked and passed the input, then it processes that input and its configuration(weights), producing an output. "Thinking" is just feeding the output back in as an input, allowing the model to revise it and refine it before it's returned to the user.

There is no teaching, nor learning. There is just data processing. There is nothing happening between requests to the inference engine. It's a relatively simple system altogether, but when the training data set is large enough it produces unbelievably good outputs.

With this in mind I really don't understand how people think these programs can be conscious. They're not doing anything fundamentally different than any other computer program. It's just crunching numbers. You might as well argue that a computer game or 7zip or a chess bot or whatever is conscious.

I know you specifically didn't claim they're conscious, I just thought it was worth mentioning because I think it's ridiculous how many people will seriously argue this as if there is any difference between "ai" and other programs. There isn't. It's just a set of instructions for doing computation, as computers have always done.

3D rendering is actually quite similar. There is a bunch of files (input/configuration) describing 3D meshes, textures, lighting etc which are fed into a rendering engine which processes all that data and turns it into a 2D image. Similarly to AI, once the input files are sophisticated enough, this can produce mindblowingly realistic and beautiful images, and you can run it in a loop to make video or make it interactive, moving the camera around in real time etc. But at the core of it all is just a bunch of files containing a bunch of numbers, and a program that reads those files and does computation with them.

My gut feeling is that the AI field is eventually going to discover how much overlap it has with information and communications theory.

Question to any AI researchers out there: Can a layer in a trained AI be treated as a form of "matched filter"? Is that a concept that is used in the understanding AI: to view it as a cascade of matched filters? The next question is "what is it matching?"

You are right in that the emergent abilities come from humongous amounts of training data and compute which only a handful of companies (and the people working in them) deal with today. I'm sure there's a lot of secret gotchas which only arise at a certain scale so one cannot self-study for them. Companies also guard these wherever possible because of economic incentives. But that applies to most other engineering disciplines. You can take a course in civil engineering and have enough knowledge to reason about why a skyscraper will not fall apart in the wind; doesn't mean you have the ability to build one.

In my experience, it seems to mostly save time on things you don't really know that well, whilst depriving you of the chance to learn new things.

Are AI models incredibly powerful tools? Yes.

Do they have the ability to telepathically sense what the various stakeholders in the development process actually wanted? No.

The divide with developers at the moment seems to be about people who believe in review & are being cautious (who are not seeing major time gains according to all reports I have seen), and those who trust it to be able to fix things forever and will no longer have to worry about understanding the code, so are happy to atrophy those skills.

Yes, my code will have mistakes, and I'll use LLMs for some tasks. I also review and have a mental model, which means I can easily fix them, no matter what happens in future with AI. Reviewing also means I'm seeing the limitations and common mistakes of LLMs.

A no-review approach will be far faster than this, obviously, but if scbench.ai is still true on the frontier models, it may come back to bite people when they have no idea how it works and the context becomes too big to maintain.

I personally have had good success with using agents as virtual pair programmers following some simple rules for how to make that process productive, but that is a very, very different approach. One that recognizes that both humans and agents are prone to errors and attempts to mitigate the problem rather than expecting it will magically work out through the power of AI.

Modern software development practice is built around that humans do make mistakes. Frequently. You don't cross your fingers and hope for the best, you assume it will happen and you put processes in place to prevent errors, catch them early, and make them easy to recover from.

It is a for-real science.

Almost all my code nowadays is AI-written, with me just contributing the concurrent data structures, a good spec, and some review and touch-up. I'm not _that_ much of a Luddite about this stuff. I'm not blind either though. Your "working together" comment does a lot of heavy lifting when arguing against a post saying that leaving the LLM to its own devices is a bad idea.

Are WE the 0.2%?

Because if not, what's the point talking about the merits of AI's competence in of itself? Accomplishes nothing, we're he outsiders. And if we are, then either you're fishing for disagreement and the post you replied to is actually correct, or you get unanimous support to your statements, and establish... Nothing, the in-group could be delusional in a way that does not actually function as an argument against the whole "So where's the software? We're the AI led market disruption instead of the AI-led shovel makers?". The punchline was about the clients being unhappy.

Over time, the errors don't "work themselves out together," they compound. Everything would stall in a mess of unreliability and chaos.

This week it did it a second time on an unrelated project.

"Some mistakes" is putting it unbelievably lightly. It is shitting out complete and utter garbage, and if you think it's doing a reasonable job (or a job that can be fixed later by a more powerful model), you're either not paying much attention, or your own ability has already heavily atrophied.

Karpathy put it well: "gap of understanding of LLM capability," but he's seemingly in one of the less fortunate groups himself.

Not sure that is the position you're arguing with. I would refuse to skip code review for any colleague, no matter how trustworthy they were. This is the behaviour we are expected to accept to get productivity gains, when the colleague is an LLM.

For me, the middle ground is review and keeping a mental model, not never using LLMs. The "Anthropic are our friends and will never fuck us over!" seems pretty naive.

There are ultimately a lot of unknowns at the moment, and it is simply too soon to know if giving up understanding your code base is sensible. People (influencers) have only been saying not to bother reviewing for about a month.

Also, so many vibe coders don't seem to know that you need to keep third party elements patched.

As someone whose world used to revolve around this stuff, I agree completely. For all intents and purposes, what I thought was my career is gone. Sure, it's always been about solving problems but not like this. The problem is that the future hasn't been evenly distributed. This is going to be one long drawn out funeral

That said AI feels a bit alien.

My experience with frontier models is that in many areas of software engineering they outperform the average software developer. In some they blow everyone away. In others they're not as good. But they are getting better. The problem is often not the model's capabilities but rather what you ask it to do and are you able to understand what it does and give it guidance when needed.

Definitely the best comment I’ve read from the demoralization patrol in a good while.

No one has any clue what this thing means....

We have a continuous process of updating skills, specs etc to try and avoid all these bugs they write, give them context about why its the wrong thing to do.

This process is actually a lot of work, continuous, and can likely continue indefinitely.

Meanwhile I'm looking at code that frankly does a bad job of showing why it even exists and nobody involved can tell me why. But it does something vaguely similar to what a prompt said it would I suppose.

Citation?

If you weren't mostly writing code by hand a year ago, you weren't writing very good code.

But nothing else matched that. I was looking up all kinds of resources and that hadn't been the conclusion from any of them. It turned out Google AI was just wrong and put us in a bad spot for even looking at those results.

It very, very routinely points at articles in its RAG "citations" that simply don't support the (modestly likely to be correct) assertions.

It routinely recommends Mac software to iOS users until you ask it to confirm there's an iOS version.

It still claims widely used software has functionality that is non-existent, based on roadmap/wishlist/feature request/unreleased fork type pages.

It is better than it was but it is not adequate yet.

I do wonder what happens when:

"Help me find a tool to do XZY" is responded to (by an LLM) with "Here are a couple of options but I could also build one for you...". But at the end of the day no everyone wants to be a "Builder" or manage it. They are happy to exchange some money for someone else to handle it for them similar to the "everyone can cook but people still go out to eat" concept.

Much less than seeing new software products being shipped, we havn't even seen attempts at them being shipped, or built for that matter, again excluding this sort of meta software around the AI itself like harnesses and tooling.

We could speculate that attempts have been made but not shipped, and the details of why not being revealed, but then we're just back to it being talk with no demonstrable stuff happening in the world that backs it up.

The pitches for Dots, Grok Bot, etc., have been spectacularly uncompelling for me as a civilian. I don't need or want a bot to book my flights or read my inbox. I'm not so busy that these things have a value high enough to be worth the effort.

As a worker or an entrepreneur...maybe. Depends on the task, and how many other people were flinging slop at me on a daily basis. I could see the value of a calendar bot, or an expense reporting bot or a bullshit paperwork bot or a slop-return (slop jai alai?) bot, but there's no way in the world I'd give a bot access to the information necessary to, say, do my personal taxes.

We keep hearing versions of this same drivel since late 2025. This time it's different! Now it's really super duper amazingly awesome powerful and if you don't believe it, just trust me bro.

If you want to go all the way, a fully formalized specification is not less work than writing the software, though the potential promise of LLMs for formal methods is interesting.

But for any informal specs, no matter how detailed, while LLMs can be very capable when it comes to avoiding surface errors when translating them into code (e.g. they're more likely than human programmers to write code that compiles on first try), they still don't have a mental model, and most of the problems I've seen are IMO results of that and they increase with the scale of the project.

That means, it is mostly cheaper to do it with Copilot etc. and not alone, right?

I was talking about omitting mentions why LLMs usefulness is efficient - in coding specifically and STEM in general to a lesser degree.

And this is really a crucial thing, which do need to be brought up when discussing future forecast about multiple industries at once.

And the reason why LLMs are so efficiently useful for coding is because LLMs perform much better if any of the two conditions are met:

1) The problem at hand is formalized and transformed into a specification basically.

2) There is a formalized verification of the output done by external pipeline of different tools.

Any of the two drastically improve LLM performance, and coding activity uniquely has both of them.

Having a formalized task narrows down the scope of generation for LLM, and formal verification allows LLMs to iteratively converge on the verification pass check. Same with those math problems, they have a formal verification and LLMs can be prevented from wandering in a generated forest too much. Same in chemistry or engineering for some degree.

But other human activities usually have rather badly formalized inputs and no formal verification of the results. Karpathy is making a logical mistake (or deliberate misleading) of projecting a very unique industry (coding) on every other human activity without any nuance or analysis. He is biased, but due to his reputation and power he doesn't get called out on that mistake.

this makes it sound like it's ever been proven, when in fact it never has and never will

The only reasonable safety measure is to always keep humans in the loop, and to give them time to understand and consolidate all advances, to actually go slow.

I agree the data trove for these is not nearly as comprehensive as “the entire internet” was for LLMs. But LLMs paved the way - people are building the datasets rapidly now. I suspect this will look a lot like LLM rollout looked over the past 5-ish years. Unusable garbage… less unusable garbage… then suddenly, a fully capable model like Opus 5.5 or the like.

And the “problem” with the way the foundational models seem to work is that they don’t just acquire individual skills one at a time like we do - they go from knowing nothing to knowing everything.

> I'm also continually surprised that people seem to think politics doesn't exist anymore. I find it exceedingly unlikely that massive amounts of people will simply accept this

This is exactly the part I’m afraid of. The political response of the displaced labor (which may or may not be violent / revolutionary), and the subsequent counter-response of capital… potentially using the very robots that displaced the jobs to begin with.

Well supposedly some of the richer folks are busy building doomsday bunkers and whatnot, so I guess they aren't entirely blind to "political consequences"? Not sure what they hope accomplish there, unless they want to be king of some post-apocalyptic wasteland.

I for one am worried about our future, and have neither the money for a doomsday bunker nor the desire to live in the wasteland.

You can test this premise today. I already do, doing repairs around the house; the ability to send thermal cam photos or spectrograms (or just raw recordings) to Claude turns my phone into a tricorder with extra steps.

Again, I’m not saying we’re there, it just feels like the control software is the main bottleneck, and the development of that is now being accelerated by LLMs.

When I was young there were already ideas about 4 day work weeks, it's now been 20 years later and so many things are automated, where are the shorter work weeks? I thought the point as a society was to not work ourselves off and actually enjoy day to day life while automation handles the boring part.

William Miller: Stillwater

Lester Bangs: Stillwater?

HANGS UP

Lester Bangs: Fucking kid’s doing drugs

Meanwhile, back in the real-world, most people I know use Chatgpt a few times a week for totally mundane shit and even more people never use AI at all.

And in my world, the latest frontier models still do moronic shit in the codebase all the time.

Meanwhile, my friend X at Anthropic tells me he fears that AI is going to have a terminator moment within the next year or two...

It's bizarre honestly.

You can work around that to some extent with explicit instructions, but it doesn't scale nearly as well as actual model training.

I don't know if they will replace most humans the way you're imagining. I think the industry has done a very good job of selling this point, but that they are still quite far from this.

The response of many mathematicians to the recent dump is a good example: verifying these proofs amounts to unpaid labour for OpenAI and wastes time that could be spent doing publishable work which ultimately results in money or personal success. The slop factor also compounds the work required to verify the output considerably.

For mathematicians, programmers or anyone, if the work required to deal with slop passes the limit, it is no longer in their own self interest to use LLMs. The expectation that people will use LLMs for the betterment of humanity against their own financial interest is baffling.

ChatGPT also has much stricter guardrails and will avoid going anywhere near large swaths of human knowledge.

I'd strongly implore anyone who uses ChatGPT for domains other than programming to try the same prompts in parallel on Opus 5.5 (Extra) for a few days. The difference is often staggering. ChatGPT is mostly just a coding and automation LLM.

I find 2026 much more repulsive compared to 2016.

Not only from the big companies. Where is the explosion of disruption ?

* Higher load on senior engineers who are about to burn out hard from fighting an endless battle against unreviewed slop people keep trying to yeet into production

* Less junior hiring and suppressed wages

* Culture of uncertainty and fear

* Computers are really expensive now

* Some people now live with diesel generators and gas turbines making their neighborhood hazardous to their health

e.g. Why do Rust and CPython still have over 5,000 open issues each? At some point, will the superintelligence fix all the bugs autonomously? When is that going to happen?

https://github.com/rust-lang/rust/issues

https://github.com/python/cpython/issues

What about new features? In every thread about Rust, people are clamoring for new features and libraries. People wonder why there can't be a batteries-included standard library like Go.

So now that LLMs are so smart, is that a solved problem? Or are we still bottlenecked on the expertise of a few experts?

---

Also, what happened to AI's passing the Turing test? I see tons of slop articles that are obvious to most HN readers. They haven't fooled us apparently -- the longer the interaction, the more obvious it becomes whether it's AI.

(That said, I use AI all the time and find it to be a great tool. IMO is is a tool, not an autonomous intelligence with a will of its own.)

More noise. The world is just so noisy now.

This is far from being proven as a net gain and there are massive lawsuits incoming related to AI recommending suicide to kids and others.

Imagine being in charge of a physical factory, sure you might have a 200 USD Claude subscription but you also have a floor full of physical machines that have, if you're lucky, an undocumented debug interface and a paper manual. Not to mention safety inspections, insurance, etc.

You can't vibe code a factory floor. Not yet, perhaps in 5 - 10 years but there are some very hard, non technical, problems that need to be fixed before we'll see meaningful improvement there.

Likewise when you see other such shortcomings in our world, ask what this incentivizes for otherwise. Then things will become obvious why they are how they are, from housing, to transit, to feeding people, to healthcare.

The priority is profit, not people.

The problem are the power-relationships and laws which allow someone to abuse you and your stuff, block competition, and even put you in jail for trying to remove the software—or helping others to remove it.

Never, ever connect appliances to the Internet. I thought this was common knowledge on HN. Also pull the telematics (cell modem) fuse to prevent your car from snooping on you and selling your location and driving habits to data brokers and insurance companies.

> [Your train] never comes on time. Fuel is more expensive. The roads are still congested with traffic. Your kids are (probably) doing worse at school. Food costs more. Houses cost more. Rent is higher. Buying a computer or a PS5 is more expensive. Politics is still full of mentally unstable people. The environment is getting worse. You will still die of heart disease or cancer. Food quality is worse. Your kids can't find work. Wealth inequality is accelerating.

None of this has to do with AI.

> But hey...on the flip side...a lot of people are rapidly building software (that nobody is using), and AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)

Plain wrong on both counts.

> Imagine what a "country of geniuses in a datacenter" will be able to do Apparently...nothing.

Nonsequitor, probably based on a personal beef. Datacenters, at best, are understood to be for hosting more AI - not for people to somehow live in as an arcology of discovery.

In American history, when previous shocks like this occurred (e.g. with the Great Depression), we needed massive economic programs like the New Deal to restore consumer demand. When such programs are not forthcoming while wealth inequalities keep increasing, one can look to other countries for what ends up happening; either the disenfranchisement of ordinary people over time or unrest and redistribution by force.

Nevertheless, my counterpoint to

> If they actually had access to systems that could do "years of top professional work" across software, cyber, science, and math, they would not mostly be leaving to start more chatbot companies, agent frameworks, eval tools, and "AI for workflow" startups.

> They would be starting normal companies to disrupt the global economies of the world.

> Insurance companies. CAD companies. Compliance companies. Security auditing firms. Logistics software. Tax automation. ERP replacements. Drug discovery tooling. Industrial design. Legal ops. Accounting systems. Boring vertical SaaS with insane output per employee.

is:

Many people who work in such industries already know an insane amount of inefficiencies. Unluckily, they are not in positions and have no political power (with respect to organizational policy) in the company that they are capable of changing the situation.

Additionally, many of these sectors are heavily regulated, so it is very hard to make a difference in this sector as a startup founder, and even if there is an entrpreneur who founds a company to change the status quo, an insane amount of work in the company will be handling the big amount of red tape.

Even if a certain small group of people are also using the hype surrounding this tech to attempt to destroy the home PC market and reverse the personal computer revolution. They will not succeed. There is more of us than them. Even with "thousands of frontier LLMs". A human+llm is going to be more powerful for a long time rather than a bunch of LLMs.

Everything isn't getting worse constantly. The data is quite clearly the opposite. How old are you?

As for things getting worse, I guess it depends on the domain. In hardware, things have gotten a lot better; my desktop computer is orders of magnitude more powerful than I would have expected 10 or 20 years ago. In software, I do see a lot of bloated software being written, but I don't feel that the software that I personally use has gotten worse.

(Those are all examples from just the past 4 days, and just the ones that made it into mainstream media.)

> self driving cars launched in my area

It sounds like the transformation would be a massive collapse of the economy with scores of people unemployed.

> Or it will take control, "optimize humanity survival chance" and realize we don't need as many people on this planet.

Maybe, but more likely it will be the humans who control the AIs and hold the power who will make this decision.

Rather: these problems are not solved because there are not enough geniuses who are capable of convincing the masses to send the responsible politicians to hell.