This is a terrible article, both poorly argued and ideologically biased. The other story presented, about a peasant named Peter Stubbe who turns into a werewolf undermines its central premise, tying the phenomenon of imagined werewolfs to certain feminist ideas, or at least shows them to not be gender-specific in any meaningful way. Peter was said to have especially preyed on women. He was a "bad guy" in the same way that the she-werewolves were "bad girls". Plus it's his story which includes, as presented, accusations of sexual (shall we say) misbehaviour, so whence the conclusion at the end?...
But those other ones at least have the room to potentially do that, even if most won't. So what GP is saying is that this is a magical stock indeed, but the other way around, in that it's magically capped.
I had in mind living systems, as most software systems of interest are: a perpetually moving target, continuously modified, expanded and refactored. Many of these systems are apparently now being polluted via careless and massive vibe-coding.
One thing to keep in mind is the logic of technological evolution and market competition will ensure that we always push the complexity of our systems to the edge of what is possible using current tools.
I don't deny that coding agents can be (and are) very useful for software engineering, as you say they have a lot going for them, and I'm pretty sure that once the dust has settled, they will be seen as an intrinsic part of the process, but I also am convinced that skilled humans will still very much be needed to direct and supervise them.
> you're ignoring the distinct possibility that [...] LLMs could simply stop writing bugs
That just doesn't seem like a reasonable possibility to me, given what we're seeing from and learning about these models. They have no capacity to be aware and adhere to an overarching vision, which is why they inevitably corrode architectural coherence over time [1]. The only reason these agents are able to still create usable code is because harnesses take advantage of model's ability to find bugs, giving us the famous loop, adding duct tape on top of duct tape, ratcheting up the complexity, until the adversarial model isn't able to find any more issues. Thus, the bugs are pushed out into higher complexity dimensions, temporarily invisible until a stronger model comes along which can detect them.
> There are plenty of examples of bugs in formally verified software.
Of course. Which is why humans will always need to be involved in the software creation process. We have to specify what we want. It's true that we ourselves often don't know what we want, in fact this is a big part of what software development is about: requirements gathering and continuously recondisering and reworking them. But it would be beyond naive to imagine that software might one day know better than us humans what it is we actually want, no?
The bugs will only disappear until the model that was used to security test the generated code becomes obsoleted. So, yes, this is the new cyber normal. Unless we replace the tech stack with a formally verified one. Which will need to happen, one way or another, seems to me. But cheap & sloppy generated code won't be feasible in this paradigm, so I believe we'll need a much more thoughtful approach.
I'm not surprised your understanding of Gödel's findings and their implications is just as superficial as that of maths and philosophy of mathematics in general. Gödel's work has implications way beyond self-referential logic games, and cannot be swept in a corner where it can be safely ignored. For example, there are certain diophantine equations that have no general solution due to Gödel. Also, worth discussing are the Continuum Hypothesis, Whitehead's Problem and stuff like Paris–Harrington Theorem, Goodstein’s Theorem, etc. Not to mention the huge impact on computer science with the Halting Problem, Rice's Theorem and others. In fact Gödel's work actually spawned CS as a field of study.
The idea that maths is just the study of arbitrarily thought-up formal systems is just a very naive idea that doesn't actually reflect reality, once investigated in depth. Anyway, I have feeling you will let nothing rob you of that sweet feeling of having "magical predictive capacities", so I'll be on my way. I've already dropped some bread crumbs if you're interested in actually expanding your horizons and getting in the weeds, where things are not as straightforward anymore. But beware, "here there be monsters". Godspeed!
"Incoherent" has a static connotation, "decoherent" OTOH is dynamic and has an associated verb, which is very handy when describing a gradual process which (almost) inevitably occurs as features are added to a system. Since confusion with the usage from quantum physics is unlikely, I too think we can safely and fittingly overload the term like the article does.
I think we can safely overload the term since the domains are separated enough. No one will think you're referring to quantum states when complaining that the system has been decohering too much lately. I can see why the author choose the term: "decohere" is a verb, which is fitting since it denotes a process, something that happens to systems gradually as change accumulates. There's no such connotation and associated verb to "incoherence".
But math is not formalizable (ask Goedel why). And no, computers have not decisively beat humans at maths, that's a laughable assertion. Sure, laymen always imagined math as being about proving theorems, but there's a saying among mathematicians: good mathematicians prove theorems, great mathematicians make conjunctures, the greatest mathematicians write definitions. But maybe you already thought they've solved math when they were able to calculate a zillion digits of pi, lol.
It sounds like you're a bit like that AI stuck in the datacenter, confirming their hypothesis without needing to actually check whether it's true or not in the real world. Then proceeding to imagine how the rest of the world is wondering at your imagined magical capacity of prediction.
So what you're saying is that since there are so many examples of what humans have preferred in the past for these models to be trained on, surely there will come a time when they will know even better than us what we want and what we like.
There's a third possibility, which to me is also the most plausible. Models will keep getting better and better within certain domains, while never catching up with humans when it comes to other type of tasks. Like that elusive quality-without-a-name some have called "taste". Or knowing what UX is most intuitive to humans. Really, anything where we have to tap into our common humanity and our human experience in order to be of help to other humans and generally to life on earth. Which is an all-important skill, since the flowering of life on the earth is (or should be) the whole point of anything we do, after all.
I agree that that's a useful distinction to make. Which is why code generated by the model should be marked thus in the repository, for example. Just like someone shouldn't check in assembly code as source code when it's actually a build artifact. So it all depends on the abstraction level one is discussing from, again, just like it is for a compiler. Ultimately, I would say automation is a type of tool. A human OTOH, is not.
It's the same for software companies too, they do much more than just producing code, it wouldn't be feasible for Anthropic to do all that other stuff too.
When coding the old-fashioned way, you were using automation as well, one example being the compiler. Still, if one is not even scanning the diffs anymore, I'm OK with not calling it "coding" anymore. We could call it "vibing" instead. But that's just words, the underlying activity is the same: creating and evolving software. That still takes skill, it's just at another level, and it's all about knowing how to use these... tools. This is what they are. A naive vibe coder is just (still) unskilled at using them and the results will be correspondingly limited.
Falling is very different from orbiting. Orbiting is accelerating, so requires GR, falling means inertial and only requires SR. Of course, in the larger scheme, all of the frames we care about are orbiting at the very least around the center of the galaxy. But that doesn't make all frames equally useful. It depends on the region of space one is interested in modeling. If it's one wants to model the movement of the planets in the solar system, a heliocentric (or more exactly barycentric) frame is the better one, because it allows for the simplest calculations.
Did you even RTFA? His argument absolutely doesn't make any assumptions about hallucinations, implicit or not. It's you who assumes Tao must have surely been complaining about hallucinations or some such. You've not addressed any of his arguments and moreover ask questions his post answers.
These systems don't have a genuine capacity of introspection, beyond just mining the conversation trace. When you're asking them why they did this or that, they are basically guessing anew from the outside, and are just as likely to hallucinate as they are to hit the right answer. These are mechanical systems which brute-force chains of various (re)combinations of techniques acquired from the training data. The true authors are all those who have contributed those techniques in the past.
I recommend RTFA, it explains exactly that: why proofs should not be considered the goalpost, and why dumping all these AI-generated proofs might be an overall negative for mathematics as a whole.
If you're asking why should we value something simply because humans do, the answer is simple: because we are human. It should be obvious, but it tends to get forgotten, that the ultimate goal cannot be anything other than actual human (and generally life) flourishing. Even the search for truth has to bend to this goal.
What you're saying, if I read you correctly (see note at the bottom) it that you find it unacceptable that something can be valuable simply because we find it worthwhile, without having a reason for why it is so. I don't see it like that. The capacity to reason didn't come along with a guarantee that all ignorance will evaporate instantly under the action of the scientific method. We can't just go by reason, because in many areas of our lives, reason alone doesn't go far.
PS: @calf, I've re-read your message above a few times and I must agree with others elsewhere in this thread that incoherence is afoot. Please review your own comments before sending, from the point of view of a reader who doesn't yet know what you want to say.