Diogenesian

  Thariq wrote "personal software was a bit early in 2020 but in 2026, it really can be as personal as a home cooked meal, or a handwritten letter."

Excuse me??? Vibe-coding personal software is not a homecooked meal or a handwritten letter! It is [at best!] more akin to tastefully combining microwaved ramen with microwaved veggies and toaster oven chicken, or using ChatGPT to write a letter then editing it afterwards. It's fine, maybe even very good. But it's not "homecooked."

Ugh. The most depressing thing about the AI boom is watching tech people devalue human experience.

Even in early 2023 people were talking about this as a use case for ChatGPT copy-paste coding. Arvind Naranyan (AI As Normal Technology) used it to build edutainment applets for his kids.

What an asshole:

  On the other side of the AI-reasoning fence, the disdain seems to be mutual. “These ‘scientific’ papers from last summer — I would put this in big, big air quotes,” said Sébastien Bubeck, a member of OpenAI’s technical staff (and a prominent evangelist for the company’s reasoning models among scientists and mathematicians). He called earlier Apple results critiquing AI reasoning “wrong,” claiming that they were due to a training quirk in models that are now obsolete. “Modern models starting with GPT-5.5 do not suffer from this issue,” he said. “It would be interesting to revisit those results.” (Apple did not make its researchers available for interviews.)

Then, later:

  The “think” part is what OpenAI, for one, is doubling down on. When I asked Bubeck if the splashy unit distance proof was produced with methods outside the LRM’s own chain of thought — perhaps with Lean verifying its results — he seemed to find the question almost nonsensical.

  “It’s not like we’re making a mystery of it,” he said. “We have released the chain of thought. You can just go and look at it. The whole point is that the model is reasoning like a human would. And when humans reason, we don’t use Lean.” Technically, OpenAI released a “rewritten summary” of the model’s chain of thought produced by two human experts using Codex, another OpenAI model. Since 2024, the company has not publicly revealed “raw” chains of thought from its reasoning models, a policy also adopted by Google DeepMind and Anthropic.

That "training quirk" thing is obvious (yet unfalsifiable) BS, and who the hell is he to sneer about "science" when his company won't release the raw data for independent scientists to look at?

I strongly suspect it's closer to the latter; CNBC says they had to sell rapidly to meet margin requirements and it couldn't be confirmed if they actually succeeded. Suggests there was a lot more than $250m in collateral on the line.

https://www.cnbc.com/2026/07/30/leopold-aschenbrenners-hedge...

Hmm, fair enough - I checked Wikipedia. My mind completely erased the "sophon" bit or whatever it was called, it just got erased again. Not a fan of that particular fairy dust.

I thought they were referring to the inability of the aliens to make an accurate calendar.

But I think this has to be judged against an adversarial influx of LLM-assisted pseudomathematics slop vendors. A sorry-free proof in Lean (or Rocq!) whose top-level types check out is not good enough. You gotta check for compiler chicanery in all the private methods.

Or, alternatively: refuse to accept a Lean program as a valid proof. I assume LLMs are pretty good at Lean -> mathematical English in LaTeX.

I believe the alien scientists were depressed by the guaranteed collapse of their civilization, not the difficulty of certain differential equations. So it's more like a doctor getting weird results while working on a cure for a pandemic.

Edit: oops, see the reply below... and my grumpy response. I seriously forgot about that entire plot point.

I don't think open-weight models are what motivated these specific comments. It's a way of deflecting OpenAI's failure with HuggingFace/etc as a natural consequence of powerful AI instead of a specific problem with OpenAI's recklessness.

The problem is that this doesn't distinguish LLMs from other forms of software! The whole point of computers is to offload tasks requiring reasoning, and they've always helped us understand complex nonlinear systems is that computers work through the hideous details.

More significantly: many major advances in scientific or mathematical programming were heralded as a thinking computer! 50's numerical scientific programming, 60's Lisp+ symbolic programming, 70's logic and symbolic algebraic programming, 80's unifying all this with encyclopedias and NLP like Mathematica - all of this is clearly useful and cool, all designed to offload intelligence... and, with modern eyes, we see plainly that not a single shred of intelligence is required to execute any of it. The AI researchers of yesteryear consistently made cool computer programs and consistently overestimated their progress at cracking human intelligence. It is plain as day to me that ANN researchers are making precisely the same scientific and philosophical mistakes as yesteryear's symbolics researchers, and even Alan Turing. This does not diminish from the coolness of the computer programs. But pretending these systems are intelligent, even "verbally" intelligent, is bad for society.

It's bad for computer science, too. None of the progress in LLMs or ANNs gets us any closer to making a robot as smart as a cockroach. I doubt any of us will live to see that milestone.

It's plausible quite a few smaller datacenters have modern engineered wood frames. Wood is pretty high-tech these days. Engineered wood is cheap, lightweight, preserves as well as reinforced concrete, and increasingly for new residential buildings under 5 stories. It would be competitive in a bidding process considering the investors clearly have an appetite for risk.

But even for a steel and concrete datacenter you would want a few carpenters for the interior where humans (at least occasionally) work, more to help with scaffolding and other temporary buildings (e..g housing) during construction, and building framing for drainage ditches and other landscaping. I imagine wood-and-nails carpenters are quite in-demand for most datacenter projects.

But bugs like this aren't because natural language is ambiguous, it's because the LLM/etc has inadequate safeguards against unambiguously malicious text. If LLMs were capable of understanding human language and only subject to natural linguistic ambiguities like any other college-educated humans, bugs like this wouldn't be reliably reproducible across different models. People in this thread are trying very hard to argue that humans are subject to this via social engineering but it is not the same. GPT-5.6 is subject to this bug for the same reason it sometimes rm-rfs stuff it "knows" it shouldn't: these machines are still stochastic parrots. It is borderline magical how powerful stochastic parroting is as a means of computation, but in the same way that a minimal Lisp system can magically be extended to a powerful theorem-proving algebra-cruncher. But parroting is simply not how humans actually understand language, and it is clearly an inadequate way of implementing language on a computer.

Philosophically no, but that shouldn't be a distraction from the issue with LLMs. This really is closer to "Outlook runs an untrusted VBA macro" than "intelligent entity gets confused by inherent ambiguity in human language."

No, it's an honest thing to say. If you're not able to read a graduate-level textbook and teach yourself category theory then you don't have the mathematical maturity to understand category theory. You need to start smaller: ideally abstract algebra and point-set topology, or set theory and mathematical logic if you just want the CS applications. Likewise with Lean. Of course there is a place for a good human instructor. But mathematical maturity must be developed the hard way.

To be fair I think Cherny is high on his own supply, not cynically trying to sell us something. He is absolutely stoked that Claude Code is an overengineered piece of junk he doesn't understand.

I believe if you need a human tutor to understand category theory or Lean then you don't have the requiste math/CS background, and a human/LLM tutor won't be able to meaningfully teach you anything.

Note I said "meaningfully" - something with LLMs that deeply concerns me is that they provide edutainment, and people actually think they're learning something. One of the comments on this thread mentioned giving a "dense linear algebra pdf" and having an LLM use the Socratic method. I guarantee they would have learned more if they read the PDF. I strongly doubt they learned anything at all with a bunch of silly "Socratic" questions about linear algebra. But it felt like they learned something!

This seems like a bit of an overstatement:

  Despite HAWK having survived two rounds of expert human review over a period of two years, Mythos was able to improve the best-known attack on it in just 60 hours of work—effectively cutting its key strength in half.

since, later:

  Mythos’s attack works by finding a specific, previously unexploited symmetry called a nontrivial automorphism in the lattice used by HAWK. Prior work proved that efficiently finding such an automorphism would permit an attack, but did not answer if such an automorphism was accessible in the lattice used by HAWK. The automorphism discovered by Mythos allows a faster enumeration attack that, while still exponential, means that one needs to double the size of HAWK keys to achieve the same level of security.

Not downplaying Mythos's contribution here[1], but that first paragraph strongly hinted (at least to me) that there were no known weaknesses. "Discovering a weakness that had previously been only theoretical" is vastly different from "discovering an unknown weakness." Again: very cool Mythos was able to do this. It just seems like another case of "LLMs are good at finding concrete mathematical (counter)examples" - which is also cool! But the PR here is cynical.

...and it is kind of incredible to think that they spent $100,000 over 3 days looking for an automorphism. Not the possibility of an automorphism, that was already known. Man.

[1] ... or focusing too hard on the strange use of mathematical language...

I don't think this is AI-generated. It is focused and direct. It reads like anodyne albeit totally human academic manager writing.

I don't think it makes sense to say anything except "computable real" - the computable / uncomputable distinction seems totally immaterial for the purposes of most real analysis, or even "pathological" topology and set theory involving R (except for puzzles directly involving computability). And the "interesting" transcendental computable reals are a bit of a grabbag.

The mean value theorem isn't true for the computable reals, differentiation of computable function isn't always computable, sequences tend to behave poorly, etc. There's still a lot you can say: https://en.wikipedia.org/wiki/Computable_analysis but in general calculus doesn't care about computability, that's a human problem.

It has always been ridiculous to suppose that some organization spent $500m on a microbiology / genetics lab, but ran out of money for scientists, so they have to ask Claude what to do. Or OTOH to suppose a guy in his garage set up a weapons-grade CRISPR lab without anybody noticing.

I think footnote 16 on pg 12 clarifies his view.

It is too late for the current crop of university students. Most of them should have never graduated middle school. The failure is very deep.

I think there are a lot of serious problems with this article but don't necessarily disagree with the conclusion, as such. I do think robots and whatnot might be conscious some day. (I am a bit outraged that LLMs, despite totally lacking free will, seem to gain moral patienthood based on their fluency with ethical philosophy. It's childish and misanthropic.)

However, the real issue, as always completely unaddressed, isn't attempting to make a gague-invariant theory of qualia, or even defining what "qualia" is (I am totally unconvinced it is an EM field, there must be an epigenetic component). The first step is defining what "consciousness" is! Is it awareness of qualia or awareness of reality? These are not the same thing. It seems significant that any animal I would call a little conscious, even ants, are capable of being startled or curious when their sensations mismatch or incompletely describe reality (and animals that lack any sense of curiosity are at most borderline conscious). Curiosity and surprise[1] seems hard to convey in an EM field. Qualia strikes me as a somewhat technical problem of neuroscience, and the "hard problem of consciousness" is unscientific BS. The simplest basics aren't understood, which leads to totally unhinged statements like "in some ways a thermostat is conscious" because in some ways it seems to fit the hard problem. But statements like that are ideologically motivated by a sort of tech futurism: rationally, we should be deeply skeptical of an account of consciousness which insists that clocks perceive the flow of time.

[1] There's an information-theoretic version of surprise which intuitively describes "surprising" data. A sudden bright light has high surprise and is certainly surprising. A sudden realization that the vine you're leaning on is a large snake might involve very low information-theoretic surprise... and is far more alarming. So how do you systematically quantify this second form of surprise?

Are they? They are smarter than web-spinning spiders but I've never read anything suggesting they are smarter than bees.

https://www.nature.com/articles/nrn3357

Right, the whole point of bees as insect pollinators (reflecting that they are recent evolutionary line) is that they are by a healthy margin the most intelligent arthropod. They will inspect then successfully pollinate anything floral over a wide geographic range. Most insect pollinators are instinctually tied to certain species/families.

The first part of your comment seems to be replying to a pure strawman, and missing the point of my argument. My point was even small natural reactors leave very obvious evidence, so it seems like a city's worth of metallurgy could be detected 200m years later, probably by a surprised geologist in the 1800s. The fact that we haven't seen anything is conclusive evidence that modern humans were the first metallurgists.

  Yes, that's why I'm careful to qualify it with "technological", which is by implication the whole thing we do with metals and minerals and oil and all that stuff. 

"And minerals" is doing a lot of work, and it is just wrong to exclude plant/animal domestication from technology. Likewise earthworks, irrigation, trading networks and currency, etc. You're being very dismissive and closed-minded about what technology is. ("lived like hippies")

In particular, there don't seem to have been any dinosaurs that had anything like primate hands, so they would have never built real tools or weapons, or manipulated fires. That doesn't close off the possibility of an "advanced civilization" unless you mean "uses tools as fancy as ours." We don't actually know what 200,000 years of natural domestication would look like.

The exception that proves the rule is a nuclear reactor from Gabon that dates to 1.7bn years ago... and which was clearly natural, a uranium-rich cave got flooded and went supercritical: https://en.wikipedia.org/wiki/Oklo

Of course we haven't discovered everything. But it seems very likely that if there were a civilization's worth of slag from 500m years ago, we would have discovered it.

A more interesting question is the civilizational technology of domestication, especially since that is short-term technology coupled with long-term evolution. It does seem possible (but purely fantastical) that Earth had talking dinosaurs with sophisticated ideologies and mythologies, feeding dumber dinosaurs and curating the growth of nutritious plants, the descendants of which all died 65m years ago.

I actually do read them. Not in severe detail, but not casually either. 150 pages is really not very long and there doesn't seem to be too much bloat. (I would cut out the moral personhood stuff but that's a political/ideological thing).

This is snarky but I am grumpy: I wonder if there's a correlation between me refusing to use LLMs and me being happy to read a novella-sized PDF about them.

Responding to you again to tell you to stop making personal attacks. I flagged your other comment. Stuff like this is why people leave HN.

It seems like me insisting that LLMs are stochastic parrots [presumably that's what you're mad about] is beginning to draw personal attacks on totally unrelated threads. Happens a lot. I used to have an account here with my name on it. HN is simply not a safe place to disagree with prevailing tech ideology.

"narcissistic rando" half of these replies are making personal attacks. Knock it off.

I answered this several times in the thread: almost every book is out of print, including some of my favorites. I would be much more stupid and boring if I only read new books. And of course indicates why scribbling in a book is so terrible: if it's Tolkein or Susan Rice or Chem 101 7th edition then sure, whatever. But a lot of the worst scribbling comes from graduate students who don't value the text they're bloodlessly working through.