_dwt

I've never read Howey's books but have them mentally filed next to, like, Dungeon Crawler Carl, the Bobiverse, whatever that long thing is with all the superheroes, thousand-chapter cultivation LitRPG blah blah blah as Reddit-beloved high-wordcount fan-fiction-adjacent genreslop.

I hate being a snob, but when we read "artificial Intelligence is writing about as well as humans", maybe take that with a grain of salt? It has not been my experience that AI writing is remotely near a "good" human writer.

HN consistently, IMO, underrates the degree to which technology adoption is “political” rather than inevitable. I was going to write a long jokey bit here but instead; counter examples - nuclear power, eugenics, the AC vs DC wars during electrification, the great firewall of China…

I find that some of my friends and acquaintances have gotten obsessed with prompting style, "prompt engineering", which skills to use, which skills to build, "context engineering", and a billion other variations on "how to write smart things so the model does good".

Friends, look at the prompts that Anthropic's own people are putting into the machine:

> A few hours after the first message, we found that Claude was still searching for simple attacks and sent a message: “no again the goal is that we have highly inteligent [sic] model as good top researcher, we want to find new attacks”;

> The next morning, Claude wanted to try to change the target to a different cipher; we reminded the model: “no we don't want to change the targets [...] agian [sic] we need to find something that worth [sic] publishing”;

> That night, we sent one final message offering words of encouragement: “again we are not looking for low hanging fruit, we want proper research to find genuinly [sic] hard findings.”

All of that RLHF and fine-tuning effort is going toward making prompts like this, or worse, work with no fuss.

There is a lot of rehashing of LLM arguments in the comments here, but I'd love to read an actual distributed systems expert's take on the DNS architecture the author and/or Claude ended up with. One of my personal fears about LLM coding is that some of those implicitly outsourced decisions turn out to matter more than expected. Does the "timeline" stamp make sense? (I know there's not much detail to work from.)

No, in my opinion as a parent this is lunacy. By the time my little kids "need" AI or the Internet in a meaningful way the AI labs will have either delivered the dream of "just talk to the computer" in which case there's nothing to learn to use, or they will be smoldering Pets.com-esque craters. Either way, for us right now it's books and a little bit of "old fashioned" movies and video games; no algorithmic feeds, all human-created content, and a focus on enjoying narratives and experiences together. We can look things up together on the Internet if we need to, and if that routes to an LLM Dad may groan a little but it's OK. Mostly we learn by trying things out, making guesses and talking about them, or looking in books.

I understand the calculus may change with middle school and up, but I still think that despite the "rat race" dynamic of grades and homework, kids who learn to think the pre-2023 way will come out ahead in the long run, even if it's only in life satisfaction.

I don't buy the superintelligence package, but I think uncritical LLM adoption poses plenty of threats to things I care about, in a mundane human-scale way.

Anyhow, I think you're (absolutely! ugh) right about the politics and I try to make the same point to people: whether you love or hate LLMs, accepting the "inevitabilism" framing is just ceding control of the Overton window. For better or worse, technology adoption can be and has been slowed by politics. We don't have nuclear plants everywhere. We don't have Project Orion starships colonizing Mars. We still have very strong social stigmas against genetic selection for human embryos, etc. This all can change in a heartbeat, and I'm not sure that policing the hardware rather than holding specific humans accountable for bad LLM outcomes is productive, but fundamentally: yes, we can stop it.

I think at this point the containment board is the entire Internet. I have no idea how, but we all need to "Atlas Shrugged" this shit and start over somewhere else with something else.

I think you may have missed a subtle point: there is an especial risk from automation which almost always works correctly. The aviation industry calls the phenomenon "automation fatigue". It's very difficult for humans to stay alert and monitor systems like these, and the use of the systems tends to lead to de-skilling over time in the very skills required to monitor them and fix the (rare but fatal - at least in aviation) error cases when they occur.

"Yes, Socrates, you can easily invent tales of Egypt, or of any other country."

I have a question for all the "humans make those mistakes too" people in this thread, and elsewhere: have you ever read, or at least skimmed a summary of, "The Origin of Consciousness in the Breakdown of the Bicameral Mind"? Did you say "yeah, that sounds right"? Do you feel that your consciousness is primarily a linguistic phenomenon?

I am not trying to be snarky; I used to think that intelligence was intrinsically tied to or perhaps identical with language, and found deep and esoteric meaning in religious texts related to this (i.e. "in the beginning was the Word"; logos as soul as language-virus riding on meat substrate).

The last ~three years of LLM deployment have disabused me of this notion almost entirely, and I don't mean in a "God of the gaps" last-resort sort of way. I mean: I see the output of a purely-language-based "intelligence", and while I agree humans can make similar mistakes/confabulations, I overwhelmingly feel that there is no "there" there. Even the dumbest human has a continuity, a theory of the world, an "object permanence"... I'm struggling to find the right description, but I believe there is more than language manipulation to intelligence.

(I know this is tangential to the article, which is excellent as the author's usually are; I admire his restraint. However, I see exemplars of this take all over the thread so: why not here?)

It's getting bad here; I've seen at least three obviously AI-written "anti-AI" or "AI critical" pieces hit and remain on the front page in the last week. I can't help but think about Bill Hicks on marketing: "Everyone here who’s in marketing is now thinkin' the same thing: 'Oh, cool. Bill's going for that anti-marketing dollar. That's a huge market.'"

It’s ok, Harlan, you save up for that baby and don’t let the cult browbeat you over it.

(People are fuzzy and while game theory is fun and even sometimes useful, this kind of stuff skeeves me out and I think it lures “smart” people in and gets them to empower the SBFs of the world. My altruism is ineffective and I’m happy that way.)

> literally

You were either a very talented baby or we’re justified in questioning your ability to assess the correctness of nitpicky formalisms.

The formal methods people may yet have the last laugh. I did not have Lean becoming a hyped programming language / proof assistant on my bingo card for 2025-26 and yet here we are, because these tools help us close the validation loop for LLM agents. That is not dead which can eternal lie...

But yes, I think the best rebuttal to Dijkstra-style griping is Perlis' "one can't proceed from the informal to the formal by formal means". That said I also believe kind of like Chesterton's quote about Christianity, they've also mostly not been tried and found wanting but rather found hard and left untried. By myself included, although I do enjoy a spot of the old dependent types (or at least their approximations). There's an economic argument lurking there about how robust most software really needs to be.

> A number of these phenomena have been bundled under the name "Software Engineering". As economics is known as "The Miserable Science", software engineering should be known as "The Doomed Discipline", doomed because it cannot even approach its goal since its goal is self-contradictory. Software engineering, of course, presents itself as another worthy cause, but that is eyewash: if you carefully read its literature and analyse what its devotees actually do, you will discover that software engineering has accepted as its charter "How to program if you cannot.".

- Edsger Dijkstra, 1988

I think, unfortunately, he may have had us all dead to rights on this one.

For me, “interestingly wrong” becomes just “wrong” without human thinking behind it. I wasn’t bowled over by the prose, I just thought it was an uncommon take and didn’t twig the signs it was Claude product.

As a couple sibling comments said, I took it for an insight into the way an optimistic writer might see AI software development becoming a new form of "end-user programming" or "citizen developer" tooling. I'm personally too deep in the weeds to ever see it becoming empowering in that way (if nothing else, this will be an incredibly centralizing technology and whoever wins the "arms race" [assuming we we're not in a bubble destined to pop soon] will absolutely have the possible Toms and Megans of such a future by the short hairs). But I love end-user programming, or whatever we're calling it now! (I was partial to "shadow IT" - made it sound really cool.) So I enjoyed the idea that somebody saw AI as a "bicycle for the mind" in that sense, even if I feared they'd end up disappointed.

But there was nobody there, and I'm only disappointed in myself for not noticing.

It's a major bummer. When I first read the story (a few days ago, maybe?) I thought it was an interesting metaphor that didn't quite line up with the observed details of software development with AI. I assumed the writer was a journalist or author with a non-technical background trying to explore a more "utopian" vision of where trends could go.

Without the inferred writer, it's much less interesting to me, except as a reminder that models change and I can't rely on the old tics to spot LLM prose consistently any more.

I'm sorry you've had that experience, and I agree there are a good share of "skeptics" who have latched on to anecdata or outdated experience or theorycrafting. I know it must feel like the goalposts are moving, too, when someone who was against AI on technical grounds last year has now discovered ethical qualms previously unevidenced. I spend a lot of time wondering if I've driven myself to my particular views exclusively out of motivated reasoning. (For what it's worth, I also think "motivated reasoning" is underrated - I am not obligated to kick my own ass out of obligation to "The Truth"!)

That said, I _did_ read your comments history (only because you asked!) and - well, I don't know, you seem very reasonable, but I notice you're upset with people talking about "hallucinations" in code generation from Opus 4.6. Now, I have actually spent some time trying to understand these models (as tool or threat) and that means using them in realistic circumstances. I don't like the "H word" very much, because I am an orthodox Dijkstraist and I hold that anthropomorphizing computers and algorithms is always a mistake. But I will say that like you, I have found that in appropriate context (types, tests) I don't get calls to non-existent functions, etc. However, I have seen: incorrect descriptions of numerical algorithms or their parameters, gaslighting and "failed fix loops" due to missing a "copy the compiled artifact to the testing directory" step, and other things which I consider at least "hallucination-adjacent". I am personally much more concerned about "hallucinations" and bad assumptions smuggled in the explanations provided, choice of algorithms and modeling strategies, etc. because I deal with some fairly subtle domain-specific calculations and (mathematical) models. The should-be domain experts a) aren't always and b) tend to be "enthusiasts" who will implicitly trust the talking genius computer.

For what it's worth, my personal concerns don't entirely overlap the questions I raised way above. I think there are a whole host of reasons people might be reluctant or skeptical, especially given the level of vitriol and FUD being thrown around and the fairly explicit push to automate jobs away. I have a lot of aesthetic objections to the entire LLM-generated corpus, but de gustibus...

Well, no, not with that attitude there won’t! I am not trying to insinuate that there is a conspiracy, or that posts like yours are part of it, but there has been a huge wave of posts and comments since February which narrow the Overton window to the distance between “it’s here and it’s great” and “I’m sad but it’s inevitable”.

Humanity has possessed nuclear weapons for 80 years and has used them exactly twice in anger, at the very beginning of that span. We can in fact just NOT do things! Not every world-beating technology takes off, for one reason or another. Supersonic airliners. Eugenics. Betamax.

The best time to air concerns was yesterday. The next best time is today. I think we technologists wildly overestimate public understand and underestimate public distrust of our work and of “AI” specifically. We’ve got CEOs stating that LLMs are a bigger deal than nuclear weapons or fire(!) and yet getting upset that the government wants control of their use. We’ve got giddy thinkpieces from people (real example from LinkedIn!) who believe we’ll hit 100% white collar unemployment in 5 years and wrap up by saying they’re “5% nervous and 95% excited”. If that’s what they really think, and how they really feel, it’s psychopathic! Those numbers get you a social scene that’ll make the French Revolution look like a tea party. (“And honestly? I’m here for it.”)

So no, while I _think_ you’re correct, I don’t accept the inevitability of it all. There are possibilities I don’t want to see closed off (maybe data finally really is the new oil, and that’s the basis for a planetary sovereign wealth fund. Maybe every man, woman, and child who ever wrote a book or a program or an internet comment deserves a royalty check in the mail each month!) just yet.

I am going to try to put this kindly: it is very glib, and people will find it offensive and obnoxious, to implicitly round off all resistance or skepticism to incuriosity. Perhaps to alienate AI critics even further is the goal, in which case - carry on.

But if you are genuinely confused by the attitudes of your peers, try asking not "what do I have that they lack" ("curiosity"?) but "what do they see that I don't" or "what do they care about that I don't"? Is it possible that they are not enthusiastic for the change in the nature of the work? Is it possible they are concerned about "automation complacency" setting in, precisely _because_ of the ratio of "hundreds of times" writing decent code to the one time writing "something stupid", and fear that every once in a while that "something stupid" will slip past them in a way that wipes the entire net gain of AI use? Is it possible that they _don't_ feel that the typical code is "better than most engineers can write"? Is it possible they feel that the "learning" is mostly ephemera - how much "prompt engineering" advice from a year ago still holds today?

You have a choice, and it's easy to label them (us?) as Luddites clinging to the old ways out of fear, stupidity, or "incuriosity". If you really want to understand, or even change some minds, though, please try to ask these people what they're really thinking, and listen.

I think this is coming, alongside professional licensure for "software engineers". Every public-facing project will need someone to put a literal stamp of approval on the code, and regardless whether Claude or Codex wrote the bulk of it, it'll be that person's head on a pike when something goes wrong.

This isn't what many of us probably would have wanted, but I think the public blowback when "AI-coded" systems start failing is going to drive us there. (Note to passing hype-men: I did not say they will fail at higher rates than human-coded systems! I happen to believe this, but it is not germane to the argument - only the public perception matters here.)

I hope we get a follow-up in six months or a year as to how this all went.

Yes, this is the reason I've completely stopped releasing any open-source projects. I'm discovering that newer models are somewhat capable of reverse-engineering even compiled WebAssembly, etc. too, so I can feel a sort of "dark forest theory" taking hold. Why publish anything - open or closed - to be ripped off at negligible marginal cost?

Not the original commenter, but you may have noticed a wee kerfluffle between a large nation-state's "Secretary of War" and a frontier model provider over whether the model's licensing would permit autonomous lethal weapon systems operated by said - and I cannot emphasize the middle word enough - large _language_ model.

Gary Marcus is living in Claude's head rent-free?

This one IIRC: https://samkriss.substack.com/p/the-law-that-can-be-named-is... He writes about it here, a little: https://samkriss.substack.com/p/against-truth

In general long meandering semi-factual pieces like this, with odd historical excursions, are one of his things and I don't know anyone else that does it quite the same. (Hmm... oddly enough Scott Alexander, who he cites here, also does some similarly Borgesian stuff, but with a different bent.) One of my favorite writers and I recommend pretty much everything he's done since the early 2010s.

Descent was a huge part of my childhood (and surprisingly my little kids are now big fans as well)! Unfortunately this seems to stutter pretty badly with audio issues as well for me on Firefox on Linux. As a huge fan of three.js and other past work... I guess I'll blame Claude?

Sorry for coming off combative - I'm mostly fatigued from "criti-hype" pieces we've been deluged with the last week. For what it's worth I think you're right about the inevitability but I also think it's worth pushing a bit against the pre-emptive shaping of the Overton window. I appreciate the comment.

I don't know how to encourage the kind of review that AI code generation seems to require. Historically we've been able to rely on the fact that (bluntly) programming is "g-loaded": smart programmers probably wrote better code, with clearer comments, formatted better, and documented better. Now, results that look great are a prompt away in each category, which breaks some subconscious indicators reviewers pick up on.

I also think that there is probably a sweet spot for automation that does one or two simple things and fails noisily outside the confidence zone (aviation metaphor: an autopilot that holds heading and barometric altitude and beeps loudly and shakes the stick when it can't maintain those conditions), and a sweet spot for "perfect" automation (aviation metaphor: uh, a drone that autonomously flies from point A to point B using GPS, radar, LIDAR, etc...?). In between I'm afraid there be dragons.

This (LOC is an anti-metric, Goodhart's Law, etc.) is true, but I'm reaching the point of "fuck nuance" when I see so many articles superficially critical of AI which contain things like this:

> If AI-generated code introduces defects at a higher rate, you need more review, not less AI.

I think that is very much up for debate despite being so frequently asserted without evidence! This strikes me as the same argument as we see about self-driving cars: they don't have to be perfect, because there is (or we can regulate that there must be) a human in the loop. However, we have research and (sometimes fatal) experience from other fields (aviation comes to mind) about "automation complacency" - the human mind just seems to resist thoroughly scrutinizing automation which is usually right.