Hacker News Reader

Comment by A_D_E_P_T | Hacker News Reader

The article's perspective reminds me of math class when I was a kid. The teacher didn't want the right answer alone, she wanted to see how I got there. In an absolute sense, the right answer is the right answer with or without my understanding, but that's not how the class was graded. We're approaching a point where AI is to science as my old calculator is to elementary school math -- it's capable of leapfrogging to answers faster and more accurately than we can.

Thing is, though, verification is knowledge itself. And if we can get to a point where the important questions have answers, we can surely have the verifiers explain those to us in ways we can understand them.

View on HN

> we can surely have the verifiers explain those

This is not obvious to me at all. LLMs are capable of amazing feats of reverse engineering but are terrible at explaining and abstracting things well.

I think this is an artefact of how we train them, and they could be trained to be better at this if we focused on it, but at the same time, it shows its possible for these things to be able to do things they cant explain to us (granted, they can't explain them well - you can bash your way to understanding with them anyway but they aren't good at it)

> This is not obvious to me at all. LLMs are capable of amazing feats of reverse engineering but are terrible at explaining and abstracting things well.

It's amazing to me that language models are still such poor writers and communicators. The mathematics community wrestled with this recently, when many solutions dropped into their laps but were not explained very well...

Thing is, (1) generic LLMs will improve, and (2) their lack of ability in this area has a lot to do with the rhetorical and structural habits they pick up during RLHF/instruction-tuning, which trained them for general-purpose comms. Zero-training base models can presumably be trained specifically to communicate scientific concepts. Interesting times.

(Edited to add: With respect to (1), compare Opus 5 to 5.5. 5 was a terrible communicator, even of software/coding concepts, whereas 5.5 is excellent and leaves very little to be desired.)

> 5.5 is excellent and leaves very little to be desired.

Hm, that may have to do with (faking) confidence. I've often enough with 5.5 still run into cases where it confidently explained something, at the surface it kinda made sense, but the more I dug and verified, the more it became apparent that it was entirely wrong with the most crucial core concepts. It has gotten to a point where you really need a domain expert though to realize this, which makes it tough for using it in areas where you are not an expert and don't have easy access to one either.

In a world of short videos, shallow reads, and now quick answers from LLM's it is easy to accept truth without verifying before moving on to the next thing. I was running into similar cases where the confidence was there but the answer was wrong, and as the article suggests, math separates correctness and understanding, so I thought it would be a great area to test where hallucinations still exist in the models. I made a test on large prime numbers where the LLM's couldn't know the answer and we can easily check the factors it produces to verify if it faked it. Nearly 75% of GPT-6 Luna were faked with a false factor when given a hard number with no tools.

You point at the exact problem here: people are so used to arguing from ignorance, they don't even notice.

Verification isn't knowledge. Far from it. You can verify statements without knowing why they're true. Variants of that concept are e.g. used in cryptology.

You need humans to truly understand (to know the context) in order to make correct judgements. Without actually knowing the context, you have no way to compare.

An infallible oracle tells Archimedes that the speed of light in vacuum is ~1.08 billion kilometers per hour. Archimedes, with the iron age tools and methods at his disposal, might be able to verify this and make some use of it himself. Maybe. In any case, what he has gained is knowledge itself.

The same goes for our own time. A powerful AI can tell us things at the edge of what we can grasp -- or beyond what we can grasp -- and it would nevertheless be knowledge that we possess. We can work to verify it, we can have it explained to us as though we were schoolchildren, or we can simply accept it as a static fact of the natural world.

In truth, we're going to have to learn to coexist with minds that are more powerful than ours. At least in some respects, if not in all respects. The good scenario is that they share information with us, and we receive it however we can. "Head in the sand" and "mutual disdain" are bad outcomes.

No, you're wrong.

You might want to look up the actual definition of knowledge. It's a pretty interesting topic in epistemology really, far from your simplistic take.

You already live in coexistence with minds more powerful than yours. Society works in a way that you never need to notice that, those minds taking care of things beyond your grasp for you.

The downside of that "convenience" is you developing an over-simplified bubble-view of the world, rendering you prone to arbitrarily hazardous misjudgements.

Here, you align foolishly with interests that cannot possibly be your own. Namely, exploiting scientific knowledge for personalized monetary gain on an unsustainable scale.

When a tiny group of people hogs essentially all scientific knowledge, that renders you what?

Your idea of being graciously provided with some "basic income" or whatever is blind copium. You would live as livestock at best, to be discarded as useless whenever your masters see fit.

I don't think the social problems you listed can be solved by redefining knowledge to your convenience - they can play that game better than you can. The definition of knowledge in this context is simply that which describes reality. And the person you're replying to was correct in saying that the thing gained from the oracle was knowledge itself, even though they've not considered the epistemological nuances.

It's not me who's "redefining knowledge to my convenience". You weirdly succumb to "them" doing that and come to wrong conclusions.

Merely claimed "descriptions of reality" lack the crucial property of truth. The description has to actually fit reality, or it's mere fantasy. It has to be exhaustive, or it remains a facade, leaving you blind to what lies beneath.

When you don't actually know whether it does any of that, you're merely a believer, entirely dependent on your "oracle" and its slave in effect.

Not caring about nuance leaves you open to exploitation, among other things.

The oracle is infalliable in the toy scenario. That was asserted as an axiom. To us who are god's eye observers of the toy universe, we already know that whatever the oracle says necessarily fits reality in that universe.

But I understand what you mean. Suppose I am Archimedes in there. I don't have access to the axiom, so I don't know if what the oracle told me is true until I can verify it myself. So verify I will. This is what is being done with AI-written proofs IRL. Humans may not be able to internalize the proofs, but we can internalize the programs which verify the proofs. So we do know the proofs are correct, or rather, as correct as the verifier programs.

You could have explained why he was wrong before injecting your opinions into the discussion.

He's wrong because the epistemological formula is the justified true belief. The infallible oracle simply gave Archimedes a true belief, the needed justification, sensors that can actually give meaning to the precise number, would take hundreds of years to develop. Archimedes would have to sit there with the oracle and develop modern physics together before the oracle can be said to have given him knowledge.

And AI won't have given us any actual knowledge until it can fit its findings into the mathematical establishment. I forget who it was that said that society doesn't know anything until you can teach it to a room-full of undergrads, but that's about right.

And you will have to actually show your intellectual work before your arguing here will actually accomplish your goal of sounding smart.

The dictionary definition says that knowledge is information gained by study -- i.e., in the very vast majority of cases, by being told. What's your definition? Come on, humor me.

The rest of your post is excessively emotional and off-target.

Perhaps be more thoughtful regarding the "dictionaries" you use? The words you use become the borders of your world.

How does random data differ from "information you gained by study"? Let alone from "what you're being told"? How do you possibly notice?

Other people cannot force you to gain true knowledge. It has to be earned by putting in the necessary effort. Its value is manifold, starting with not gullibly falling prey to other's interests.

To be fair, most people aren't writers, so the "average" writing the statistical model can produce is mediocre writing.

I don't think this is the issue. Most people are mediocre mathematicians, too, yet the companies have selected well enough for math literacy in training data (presumably). I think its a matter of training - they are often post trained by other agents (which read everything), human reviewers not paid enough to care (easily wowed by 10 paragraph length essay with correct punctuation), and deterministic scripts (checking for presence of a specific point, so agents are incentivised to write as much as possible to have more chances of hitting the unknown thing the script is looking for)

As far as I know labs try to train the other way, else we'd have monstrously long and useless outputs, but its hard to overcome these natural biases for length

I don't think that's a suitable mental model of how LLMs work. Their output is not just the statistical average of all the training data. If it was then the output would be quite useless.

An LLM trained on the entire internet incorporates essentially all (human-known) aspects of things.

Like a superposition of all kinds of writing styles, all perspectives on every topic and all kinds of ways to do things right and wrong.

In post-training (and of course already by sensibly curating the input data in the first place), people try to select useful submanifolds from that.

Ideally, you get a combination of all the extant strengths of people combined into one model. Self-improvement then leaves the sphere of human capability for good.

I am curious. What makes you say generic LLMs will improve? I mean, obviously, they will, but to what extent? How can we know what they will or will not be capable of by the time we reach the limit of feasible improvement? Will they ever be able to explain in a way humans can easily digest?

LLMs are excellent writers in the sense that they get the spelling, grammar, syntax, etc... right. They are certainly way better than I am, and they can compete with the best writers considering how AI-generated prose has winned literary prizes.

Many people don't like the style, probably by association with slop and fatigue more than the style being fundamentally wrong.

Being a good communicator is an entirely different thing, you can have terrible spelling and grammar and still be good at getting ideas across. It is a common situation when the communicator in question is not using his native language.

That LLMs are so good at writing and so bad at communicating is not surprizing. The key to good communication is getting into the other people's mind, LLMs can't do that, because they "think" in a way that is completely alien to us. We use words to express our thoughts, thoughts that are based on a physical world model: touch, movement, sound, light, etc... LLMs have no idea about how the physical world work, everything the see is just words, they have no intuitive idea of, say, objects falling down, just that "falling" and "down" go well together. And for explaining things, they will go "someone asks for an explanation, explanations usually use these words, I will be using the same kind of words".

And in many cases it is not really the new solution that is important. It is the new techniques, abstractions, or other sense-making or investigative tools that made the solution possible that matter.

And the abstractions do still matter, since in many cases it is an issue of tractability. Perhaps an LLM can force its way to a solution through 3 million mind numbingly boring and convoluted steps, but if the number of steps increases exponentially without the right abstraction that means it can only handle a slightly larger problem than a human with pen and paper.

>LLMs are capable of amazing feats of reverse engineering but are terrible at explaining and abstracting things well.

I don't think you use them well if you think this.

I have tried dozens of different writing skills and my own prompt ideas, listened to podcasts with engineers from both major american LLM companies, actually read the prompting guides from anthropic for the models I use, and of course, as if it needs to be said, I use the latest and greatest frontier models, for difficult work.

What is your genius setup?

Can you give an example of what you wanted explained, the LLM explanation and why it is bad?

Obviously if you don't understand an idea you aren't really in a position to evaluate how good an explanation of it is. I have a background in Physics research and find models like Astra to be excellent at explaining concepts at a level for an advanced practitioner. It also has done very well, from what I can tell, for areas I am less knowledgable in. So I am confused by statements like yours that directly contradict my experience and the subjective experiences of many others.

> Can you give an example of what you wanted explained

Something I do dozens of times a day: "Please explain what this PR does, I haven't read it"

it will make me 20 pages of gibberish about caches, correctness checking, race conditions, github action runners, tradeoffs between a bunch of invented terminology internal to the PR without defining the terms- you'd have to have already read the code to know what it was referring to

without ya know, explaining it. I ask my colleague and he says something like "oh it deletes records from such-and-such a system when they aren't needed anymore after a merge" and I know exactly what he means. A single well composed sentence was all that was needed.

You can ask the agent for a single sentence explanation and it will write something like "linked candidate records are trimmed when canonical record determination window is passed", none of which is terminology any of us use, some of which is defined in the new PR and some of which is made up. I can't emphasise enough, I use these things EVERY DAY. it knows what I work on, it knows how I talk. There is no earthly reason it should be this bad given all the skills and advance and context it has. As mentioned, I constantly work to correct its shortcomings in this area but its like trying to hold water in a sieve, it just cannot do it.

"What does this PR do?"

LLM: "Oh, it specifies the kernel window over the cache candidate list to classify possible invalidated swap identifiers"

Colleague: "It removes duplicates in the target_uuid_cache table in the db"

The agent has full context of what systems the code talks to, our terminology sheet etc. It just pathologically cannot simplify things down. If you work with them enough you learn that there are some words it reaches for a lot in certain circumstances, like it loves "canonical" in relation to merging things. But this is more like learning to communicate with a deeply autistic person who struggles to adjust their speaking to be relevant to their audience rather than talking with a "good communicator"

The other reason it looking at the approach the strategy the reasoning is so much better than a binary sense of whether something is right or wrong answer is that there's ambiguity in what really constitutes a right answer to begin with. At least in most situations. For example in hard Sciences we may learn that we drop second order details and ignore them for purpose of answering the question but still one could make the argument that in so doing the answers we land on are inherently imperfect. Does this make them less right? Could all be debated. But instead seeing someone who explains I'm going to drop the second order detail cuz I think it's not significant to how I would use this result this is the kind of thing that really is valuable

My teachers tried to explain to me that they were trying to teach me the method, not the answer. I always thought I didn't need the method, if I could find the answer without it. Nowadays, I think that wasn't quite right.

The issue I had in school was I didn't see the point of it. Why should I solve this problem? Due to that I didn't mind only knowing the answer and ignoring the methodology.

I hated classes that didn't have correct answers, for that reason. But in math, there was a 100% correct answer I could use to impress the teacher and the other students. Fueling my ego was enough of a reason for me... Nowadays, I wonder why I needed to do that.

I still don't see the point in knowing a method that takes longer to arrive at the correct answer. For example long division. It's such a convoluted mess.

Long division is just a way to convert a fraction to a decimal. It's useful when you specifically need that and don't have a faster shortcut at hand. In other situations, I agree it's usually overkill.

Same for some methods I've since forgotten for solving polynomials. They were non-iterative algorithms that could work with any constants. They weren't faster than other methods, but you could fall back on them when you didn't have anything better.

That is what they were trying to teach me. General solutions that could work with anything. While, of course, it's valuable to be able to intuit shortcuts on the fly, if that's all you're capable of, there's a class of problems you simply can not solve. If they don't click in your head enough for you to come up with a specific shortcut, you wouldn't have a general one to fall back on.

I've always used computers, and didn't understand why my teacher got so upset when I programmed equations into my calculator. Like you'd think writing a program around a family of equations would develop experience with those equations, and I thought it was a good way to speed up the solving process. Needless to say, teacher did not agree, and quickly learned to ban my calculator on tests.

(technically, it was supposed to be banned anyway because it had a CAS... but I got it because 68k TI-BASIC was better than the Z80 TI-BASIC)

I still don't know the general solutions for most of the problems I work with nowadays, and I think that sucks. I think AI makes this way worse on a scale much greater than mathematics.

Sometimes I wish I could go back to school and catch up on all the basics I've been cheating around. I feel like my apparent intelligence is a mistake and I need to develop a better foundation.

Not sure it would have made a difference. I learned long division, but I've long forgotten how to do it. Or maybe I haven't. There wasn't a need to check whether I can still do it in over 30 years. But I probably forgot.

Looks like I was the opposite to you in regards to calculators. I always preferred to do it the hard way and only used calculators for boring things that I knew by heart, but would take me longer to do by hand and be error prone.

But I didn't pursue a career where this type of stuff was any useful and forgot many of the things I could do easily in school, like integral and differential calculus.

As the saying goes, use it or lose it. No matter how foundational, you'll eventually forget all about it if you do not use it anymore. There are even cases of people that completely forgot their first language.

I'm optimistic about AI because it can have more knowledge than any given human in storage. Eventually we all kinda specialize around something. Even the generalists among mathematicians have little insight into medicine, rocket science or astrophysics. So we all become the hammers of the saying "if you are a hammer, everything looks like a nail". But when including AI we get a broader perspective.

Meaning requires integration. But I assume AI can make meaning of it, now or in the future. I assume the same with people. However I don’t see the point of knowledge without meaning.

I would prefer relation or context instead of integration. Meaning is contextual. Understanding is the process of attaing the meaning or a meaning. Knowledge in this perspective is the final outcome that can be verified in some way or you assume so.

Relation can be a link. I don’t think that’s sufficient for meaning. It needs to have an effect — and arguably a positive one — on the broader coherence.

I sucked at remembering goniometric identities, but I was superfast with testing out a few options on my calculator to make sure I was using the correct one. Without a calculator I was nowhere.

I don't think we're there yet for science. Perhaps for math proofs, but that's a very different domain ultimately since we know the rules of the game in a way we don't with science.

Oh, it's going to move up the old ladder, from bottom to top:

> https://en.wikipedia.org/wiki/Hierarchy_of_the_sciences

First math and CS, where formal logic dominate, and AI can check/verify its own answers very quickly. Then physics, downstream of math. Then materials science, which in some cases is practically a branch of physics. Then chemistry more generally. Then biology. Then various social sciences.

A big difference is that in mathematics, you can do all your experimentation purely in theory and at arbitrary speed, while for example chemistry needs a lab and plenty of time.

Math is really super suitable for quick iterations by a machine.

What makes you believe that it will be such a progression?

Reply on news.ycombinator.com