Loquebantur

No, you're incorrect. A mathematical proof actually is a program by the Curry-Howard correspondence.

Mathematicians are trained to act as the compiler, verifying the code, doing the type checks, and to a degree the state machine "running" those programs.

Human mathematicians use introspection to tell you whether a proof/program does what it is supposed to do. They keep those programs small enough to be run on human brains by modularization, abstracting away complexity behind defined interfaces. They mostly rely on other mathematicians to have checked those sub-modules properly.

By defining that state machine, compiler and the tree of "all" known proofs as programs explicitly on a computer, you get Lean and the according libraries of verified proofs.

A proof-program there is verified when it compiles and passes the deterministic structural type-checks built into the language while fulfilling the requirements stated for its interface.

It all reduces to a few thousand lines of trusted code, the Lean kernel, being correct. It's mostly what humans do too, but actually considerably more thorough.

The physical world is actually part of mathematics. Look into John Archibald Wheeler and his "It from bit". In particular, the physical world is finite. Therefore, mathematical proofs very much apply to physics.

You're right about approximations: we gather data and improve theoretical models in an attempt to exhaust that complexity reality poses.

You're wrong about exhaustive knowledge about codebases being impossible. LLMs enable exactly that.

The trick then is to transfer that knowledge in a usable way to what we know about the codebase.

I agree with the necessary cultural shift.

Yes, that's where the "engineering" part comes in.

Engineering builds models that approximate the real world under practical considerations. You e.g. have to weigh costs vs risks and apply good judgement.

In essence, you try to achieve the best possible with the resources you've got.

Anthropic here is solving one of their own problems, how to make sure you don't combine other company's models with their stuff for example, and make it sound as if they did it for you.

It makes the inference more expensive and helps controlling the users.

Yes, you can also easily prove the wrong thing.

That's why it doesn't suffice for only LLMs to know the relevant stuff.

You can't have a zoo run by animals.

LLMs can, in principle, do everything humans can. And arguably more. "Right now" is how long? People are slow.

"Provable" essentially means, you have explicit operators acting on explicit states in an exhaustively defined way that preserves "truth", explicitly defined invariants.

A finite thing you can argue over deterministically.

Obviously, "you" there is an LLM, the machine your code is run on has to be defined in that very same exacting manner and so on.

That still isn't enough, you need abstraction, a hierarchy of categories that preserve your intent and accurately describe the problem space.

"Programming" then is solving an inverse problem over that hierarchy, filling in the blanks sensibly on your way down through the model hierarchy, all the way to atomic instructions your (proven) hardware understands.

Those layers of proper abstraction is what's largely missing currently. Or rather, is merely vaguely approximated.

All of which is to say: yes, the current state is a mess. Resolving that is far from easy.

But you really have no choice, that mess is transparent to large-enough LLMs. Security by obscurity only goes so far.

It's not me who's "redefining knowledge to my convenience". You weirdly succumb to "them" doing that and come to wrong conclusions.

Merely claimed "descriptions of reality" lack the crucial property of truth. The description has to actually fit reality, or it's mere fantasy. It has to be exhaustive, or it remains a facade, leaving you blind to what lies beneath.

When you don't actually know whether it does any of that, you're merely a believer, entirely dependent on your "oracle" and its slave in effect.

Not caring about nuance leaves you open to exploitation, among other things.

The problem is the "interface" of software engineering being oblique to outsiders. It's alchemy in essence.

The logical extrapolation of AI capabilities stress-testing human software is all bugs being discoverable. Hence, you have to aim for actually bug-free software.

The only way to do that is to make your code provable. That again is only realistically doable using AI.

In the process, people will have to turn the current mere pretense of software engineering into reality, making the engineering part explicit.

That in turn only works when humans actually understand what they're doing and can communicate it to outsiders.

The "world" isn't just what humans currently make of it. You deny the possibility of alternatives for society based on what amounts to ignorance.

Transitioning from the current "order of things" to one compatible with the existence of things like AI is a necessity, not a "fantasy".

When you imagine people as savages to be outdone in their savagery, you become one yourself. Spend more effort on finding ways not to.

The logical extrapolation of AI capabilities stress-testing human software is all bugs being discoverable. Hence, you have to aim for actually bug-free software.

The only way to do that is to make your code provable. That again is only realistically doable using AI.

In the process, people will have to turn the pretense of "software engineering" into reality, making the engineering part explicit.

That only works when humans actually understand what they're doing.

Well, what to do and why it will accomplish what you (and everybody else) want and nothing else is the point. The facts being actually true is a necessity to pull that off.

Think of AI as a magic wand or a genie that makes your wishes come true.

While that sounds great at first, actually thinking it through reverses that mirage completely.

Whom would you want to have one? Who should get to decide? How do you control the whole matter and prevent it from going horribly wrong?

Being irresponsible doesn't work already when you're a grown-up human with only your bare hands.

It's not about merely knowing how to build a "badthing". It's about knowing why not to.

With AI, the point is who uses it for what.

Think about it like a magic wand that makes your wishes come true. Would you really want everybody to have one? Who should and who shouldn't? Who gets to decide?

You operate within the "logic" of capitalism, where no other value exists but the (short term) economic one.

AI indeed goes far beyond "autocomplete" and other nonsense. But the question is, whether it's a good idea to just let the "market" run wild with it.

Short-term capital interests don't align with the public good. AI greatly scales capability, but without anything to restrain it, that implies those non-aligned interests overwhelming society's defenses, outright destroying it in the process.

The idea of rich people being some kind of benevolent caretakers, who independently and dependably foresee and manage the consequences of their actions in everybody's best interest, is laughably wrong already.

With AI, that reality gap will swallow whatever is left.

You're talking about academia, society's attempt to gain economic utility from science. You assume, I didn't know about it based on what?

Exploiting science for economic utility alone misses the incurred second order effects, as I posited above. You completely ignore that crucial part, despite of it being the whole (very much practical) point here.

Perhaps be more thoughtful regarding the "dictionaries" you use? The words you use become the borders of your world.

How does random data differ from "information you gained by study"? Let alone from "what you're being told"? How do you possibly notice?

Other people cannot force you to gain true knowledge. It has to be earned by putting in the necessary effort. Its value is manifold, starting with not gullibly falling prey to other's interests.

You conflate a host of different issues and ultimately miss the point.

Science is an ideal, the pursuit for knowledge for its own sake. Science is a boon for humanity because it provides an unparalleled degree of prediction and control of the natural world.

But it doesn't tell you what to do with that.

A "capitalist" will seek personal, short-sighted gain from it, socializing repercussions, no matter how devastating. They will gladly poison society and the planet and sell it to you as "your own fault". Their life-span is limited still, so what do they care?

Technological capability exceeding human wisdom is the core issue here.

You need scientific understanding on your own side in order to know and judge the alternatives beyond what vested interests feel obliged to tell you.

No, you're wrong.

You might want to look up the actual definition of knowledge. It's a pretty interesting topic in epistemology really, far from your simplistic take.

You already live in coexistence with minds more powerful than yours. Society works in a way that you never need to notice that, those minds taking care of things beyond your grasp for you.

The downside of that "convenience" is you developing an over-simplified bubble-view of the world, rendering you prone to arbitrarily hazardous misjudgements.

Here, you align foolishly with interests that cannot possibly be your own. Namely, exploiting scientific knowledge for personalized monetary gain on an unsustainable scale.

When a tiny group of people hogs essentially all scientific knowledge, that renders you what?

Your idea of being graciously provided with some "basic income" or whatever is blind copium. You would live as livestock at best, to be discarded as useless whenever your masters see fit.

An LLM trained on the entire internet incorporates essentially all (human-known) aspects of things.

Like a superposition of all kinds of writing styles, all perspectives on every topic and all kinds of ways to do things right and wrong.

In post-training (and of course already by sensibly curating the input data in the first place), people try to select useful submanifolds from that.

Ideally, you get a combination of all the extant strengths of people combined into one model. Self-improvement then leaves the sphere of human capability for good.

Yes, true scientists do care about epistemology.

The problem is the systemic disincentivation of true science, the pursuit of truth and knowledge unencumbered by greed.

When you speak favorably of "factories for research", you adopt the position of a mindless consumer, relishing the prospect of "abundant" products of science.

You fail to notice, livestock lives in abundance as well.

You point at the exact problem here: people are so used to arguing from ignorance, they don't even notice.

Verification isn't knowledge. Far from it. You can verify statements without knowing why they're true. Variants of that concept are e.g. used in cryptology.

You need humans to truly understand (to know the context) in order to make correct judgements. Without actually knowing the context, you have no way to compare.

Who will know to ask the right questions?

You imply "worth" being equal to some known economic value.

You even imply, economic value was determined correctly in advance for arbitrarily long future intervals. And the applicability was known in advance to begin with.

Obviously, you're completely wrong.

Applying simplistic models to real life incurs a penalty. The affected scale multiplies with that penalty. When dealing with things affecting humanity as a whole, that makes careless over-generalizations a serious hazard.

Would any PhD guarantee your understanding? You appear to fall for an argument from authority there.

A PhD is merely intended to show your ability to do research-level science. It doesn't show you know independently what to research, let alone why.

You confuse your ability to solve riddles competition-style with the answer to why you should do that in the first place.

You clearly live in a world you don't understand.

Those competitions get set up because some people understand the importance of mathematical knowledge and want their names attached to advances in it, even if they can't do the work themselves.

Clearly, it would be far better to make the public understand how the world, and society in particular, actually works. Only, those with the money profit from that not happening.

You're entirely wrong.

Competitions are a way to motivate people. What do you believe, why they get set up in the first place? Because rich people have fun seeing you run in circles?

Competition for tenure and publishing metrics has been detrimental to science in general.

Your idea of life being about grabbing whatever comes your way and consuming it first says a lot about you, but not about life.

What do you suppose happens in a society where the constituents of that society "have nothing to contribute"? Your habit of over-simplification will come to bite you.

No, it's not.

The point isn't some people not understanding whatever stuff with limited effects, it's no people understanding anything important.

Mathematics is about describing fundamental reality, so you can change it. When corporations own it exclusively, they can do whatever they want with it.

Just like with mathematicians, is it all really just about people losing a random, replaceable job and what they enjoyed about it?

Being able to program means in particular, you can read those programs and understand what they do.

What happens when you don't understand them anymore?

Living in a reality where everything is a black box, doing its thing as if by magic, leaves you in a position devoid of control. The objects upholding your life are incomprehensible to you, you cannot change them or even only reliably exchange them with alternatives.

Some costs are too large to be absorbed even by society as a whole. Can you privatize rational thought?

Mathematics goes into Physics, into Chemistry, molecular Biology and so on. It "goes" there by mathematicians talking with people there and exchanging ideas about what and how to model it.

Imagine all that getting privatized, walled into a corporate garden, accessible via payment only, similar to academic journals. Thought on demand.

How long does a society like that last?

Mathematics isn't about "winning competitions".

Mathematics is humanity's current best approximation to a universal language, capable of expressing conscious thought precisely.

It's a "programming language for the human brain" that lets you describe reality exactly and thereby make predictions about it.

Without people (humans) knowing and understanding it, society is incapable of grasping reality in any meaningful way, unable to shape it in order to achieve a better tomorrow.

Most never realize that, because society is compartmentalized to the point of people never witnessing how it actually works under the hood. "Experts" do things for you that you never heard of, but that your very life depends upon.

While the majority of people thus lives a kind of "simulated" reality, the mechanisms underpinning that cushioned fantasy land aren't infallible. You cannot uphold an economy nor even a social structure without understanding and modeling it to a sufficient degree.

What the vast majority employs as "rational thought" isn't sufficient.

What would such a world look like?

Mathematics represents the rational skeleton that upholds and stabilizes our vague, formless but far more common belief-based thinking.

Without it, losing it from neglect, we would collapse back into a pre-rational state of fantastical, wishful thinking.

Pretty much exemplified by large parts of US society already, but there is no bottom otherwise keeping society from collapse.