HNHacker News
TopNewBestAskShowJobs

YeGoblynQueenne

24,940 karma · joined September 26, 2015

This a common question on this board:

What is reasoning?

In computer science and AI when we say "reasoning" we mean that we have a theory and we can derive the consequences of the theory by application of some inference procedure.

A theory is a set of facts and rules about some environment of interest: the real world, mathematics, language, etc. Facts are things we know (or assume) to be true: they can be direct observations, or implied, guesses. Rules are conditionally true and so most easily understood as implications: if we know some facts are true we can conclude that some other facts must also be true. An inference procedure is some system of rules, separate from the theory, that tells us how we can combine the rules and facts of the theory to squeeze out new facts, or new rules.

There are three types of reasoning, what we may call modes of inference: deduction, induction and abduction. Informally, deduction means that we start with a set of rules and derive new unobserved facts, implied by the rules; induction means that we start with a set of rules and some observations and derive new rules that imply the observations; and abduction means that we start with some rules and some observations and derive new unobserved facts that imply the observations.

It's easier to understand all this with examples.

One example of deductive reasoning is planning, or automated planning and scheduling, a field of classical AI research. Planning is the "model-based approach to autonomous behaviour", according to the textbook on planning by Geffner and Bonnet. An autonomous agent starts with a "model" that describes the environment in which the agent is to operate as a set of entities with discrete states, and a set of actions that the agent can take to change those states. The agent is given a goal, an instance of its model, and it must find a sequence of actions, that we call a "plan", to take the entities in the model from their current state to the state in the goal. This is usually achieved by casting the planning problem as pathfinding over a graph with a search algorithm like A*. Here, the agent's model is a theory, the search algorithm is the inference procedure, and the plan is a consequence of the theory. Deductive reasoning can be sound, as long as the facts and rules in the theory are correct: from correct premises we can deduce correct conclusions. We know of sound deductive inference rules, e.g. A*, and Resolution, used in automated theorem proving and SAT-Solving, are sound.

The classic example of inductive reasoning is inferring the colour of swans. Most swans are white (apparently) so if we have only seen white swans we have no reason to believe there are any other colours: we are forced to infer that all swans are white. We may only be disabused of our fallacy if we happen to observe a swan that is not white, e.g. a black swan. But who is to say when such a magnificent creature will grace us with its presence, outside of Tchaikovsky's ballets? Induction is thus revealed to be unsound: even given true premises we can still arrive at the wrong conclusions. Another example is the scientific method: imagine an idealised scientist, perfectly spherical, in a frictionless vacuum. She starts with a scientific theory, then goes out into the world and makes new observations about a phenomenon not described by her theory. She constructs a hypothesis to extend her theory so as to explain the new observations. The hypothesis is a set of rules, where the premises are the consequences of the rules in her initial theory. Then, being an idealised scientist, she goes looking for new observations to refute her hypothesis. Science only gives us the tools to know when we're wrong.

Abductive reasoning is the mode of inference exemplified by Sherlock Holmes. We can imagine Sherlock and Watson standing outside a tavern in London, watching as a gentleman of interest steps out of the tavern with egg on his lapel. "Ah, my dear Watson, what can we conclude from this observation?". "Why my dear Holmes, we can conclude that the man had eggs for breakfast". Holmes and Watson can arrive at this conclusion, about a fact that they have not directly observed, because they have a theory with a rule that says "if one eats eggs, one may get some on one's lapels". Working backwards from this rule, and their observation of egg on the man's lapels, they can guess that he had eggs even if they didn't directly observe him doing so. Abduction is also unsound: the man may have swapped coats with an accomplice, who was the one who had eggs for breakfast instead.

And now you know what "reasoning" means. So the next time someone asks: "what is reasoning?", you can let them know and turn the discussion to more interesting, more productive directions.

submissionscomments
YeGoblynQueenne··on Why I'm still bearish on LLMs after Navier-Stokes
>> I'm not sure why anyone is expecting stochastic systems to be deterministic.

Monte Carlo Tree Search is stochastic.

YeGoblynQueenne··on Why I'm still bearish on LLMs after Navier-Stokes
For me the useful intuition is that LLMs haven't somehow magickally learned to implement any of the algorithms we know that we have used to make strong chess engines: alpha-beta minimax and Monte-Carlo Tree Search on the one hand, and obviously the ability to learn accurate evaluation functions by self-play.

I mean we've done all this before in a task-specific fashion. It's useful to know that LLMs haven't managed to do that in the process of learning to represent the entire text on the web. On the other hand they have gotten say very good at machine translation without being trained exclusively (and I select the preceding word carefully) on machine translation.

Edit: I'm saying this because there is this idea expressed by e.g. Ilya Sutskever, that in order to predict the next token accurately an LLM has to learn something about all of underlying reality. See for example this interview with Dwarkesh:

https://x.com/biobootloader/status/1640512444958396416

Where Sutskever claims that "Predicting the next token well means you understand the underlying reality that led to the creation of that token".

If that were true, we should have seen LLMs play good chess by now. There is a huge amount of data on playing chess floating around on the web in the form of algebraic chess notation and if LLMs were capable of learning the "underlying reality" of chess, they would already have. They haven't. Because they can't. What Sutskever is saying flies in the face of literally hundreds of years of statistical modelling, which is to say, building predictive models that, very explicitly, do not have to understand any "underlying reality" and only have to be good at modelling a dataset.

YeGoblynQueenne··on Why I'm still bearish on LLMs after Navier-Stokes
>> I know HN readers and posters just read numbers and can't be bothered to read, but please read the methodology before making any claims.

This is unfair to HN readers all of whom but one did not post the comment you replied to. You can't just tar everyone with the same brush. There are thousands (hundreds of thousands?) of users on this site.

YeGoblynQueenne··on Why I'm still bearish on LLMs after Navier-Stokes
>> the models generalize well only on tasks within a small neighborhood of the specific tasks they've been trained on, and even then with severe caveats. the frontier labs have developed a general recipe to teach models almost any specific task enjoying clearly defined levels of task performance; many tasks are covered in the training data; but even small perturbations within a covered class of task result in outright failure or reward hacking.

Lots of people make this claim about "specific task[s] enjoying clearly defined levels of task performance" but they forget that generative AI is also extremely good at generating a) art and b) prose in literary style. None of those things has "clearly defined levels of task performance", in fact they are both the complete opposite of well-defined tasks. Who knows what counts for "good" art? [1]

For me the right model for generative AI is "a million monkeys on typewriters" [2]. Holding any other model to heart will at some point fail to predict observations and cause you to be unpleasantly surprised. Not least because AI companies are actively engineering their systems to optimise for this model and they have a lot of people working on that engineering and shedloads of money to throw at it.

Don't underestimate what a million monkeys on typewriters can do. They can do anything and everything, given enough time. Geneartive AI can also do anything and everything given enough resources. The only question is: how much is going to be "enough"?

____________________

[1] Yes yes, AI art tends to be slop. Not denying that. But part of the problem with slop is that it presents as technically very competent except that it lacks a certain je-ne-sais-quoi, which makes it good art; aesthetics. The point is that there is no clear measure of what makes technically competent art, any more than there is for aesthetics.

And yet generative AI is very good at it.

[2] There's even an article on wikipedia except it's about one monkey on one typewriter with infinite time. There's a proof too.

YeGoblynQueenne··on After Math
To be fair, what revolution were you expecting to see, exactly? You can 10x a "dashboard but with AI" app all you want but it's only ever going to be a "dashboard but with AI" app. And I note that nobody seems to be claiming that the AI helped them create a brand new kind of app.

Anyway, Sturgeon's law and all that.

YeGoblynQueenne··on After Math
And he's called Terence, not "Terrance".
YeGoblynQueenne··on After Math
>> If future AI systems will produce genuine proofs, logically correct and intelligible, like those produced by “master” mathematicians, it would still be incorrect to think that mathematics would have been “solved” as some say that chess or Go have been solved.

Chess and Go have most definitely not been solved. In AI research when we say that a game (and usually we mean a traditional board game) has been "solved", what we mean is that we can correctly predict the outcome of any game instance from any board position provided both players play perfectly [1]. This can mean we have an algorithm that chooses the perfect play for each player in any position or that we have mapped the entire game tree for that game: every move in every game, from start to finish.

There are games that have been solved this way: tic-tac-toe; connect four; nim; othello, checkers. But not chess and Go.

Rather, what we have for chess and Go are super-human engines, i.e. computer players that no human has ever beaten in the full game. So for example a chess engine like Stockfish can always choose what move to make in order to win against a human player in any board position. However, that is still not the same as knowing how to play perfectly in any board position. Like the old joke with the two guys trying to outrun a bear, a chess engine doesn't have to play perfect chess to be super-human: it just has to play better than any human.

To the extent that chess and Go are games it could be said that winning is the whole point of AI game-playing [2]. But the article above suggests that there is no point of "winning" in maths. In AI chess it is unfortunate that the development of super-human chess engines has ended all research in using AI chess to understand the way that human players play chess (which is to say: not by alpha-beta minimax, or Monte Carlo Tree Search, or anything like that) which was the original reason that luminaries of the field such as Shannon, Minsky, McCarthy, Michie, and even Turing himself were interested in the question in the first place. The concern then for mathematics is that AI "winning" at maths will kill mathematics research and replace it with ... as McCarthy would have it "very fast fruit flies" [3].

_______________

[1] https://en.wikipedia.org/wiki/Solved_game

[2] Then again see John McCarthy's criticism of that idea written in the wake of DeepBlue vs Kasparov: https://www-formal.stanford.edu/jmc/newborn/newborn.html

[3] ibid.

YeGoblynQueenne··on Navier-Stokes Announcement
... marginalized people ...

... spaz on the floor ...

YeGoblynQueenne··on Don't be the out of touch Kung Fu master
Well. First they came for the testers. Then they came for the developers.
YeGoblynQueenne··on Alan's Random Insult Generator (1999)
>> How does one come up with such a string of words?

By spending all day thinking about them instead of running a country.

YeGoblynQueenne··on A misalignment of AI in mathematics
Amazing. Comment upon comment, thread upon thread, failing to engage with the substance of the Field medalists' letter and instead going off on tangents about the privileges that mathematicians will lose and how they're bitter about it. There is a shocking inferiority complex on this site towards mathematicians and yet users always seem to think of themselves as smart nerds, all about science and reason and maths and whatever.

I've been disappointed by this community before but this is the first place that there is such unanimous rejection of the very ideals the uses here will claim to represent. The reaction is just... small, tiny, petty.

YeGoblynQueenne··on A misalignment of AI in mathematics
The difference is that most software engineers don't care about bringing up a new generation to keep their field alive when they retire. Software engineers usually don't give a flying fuck for other software engineers, young or old, nor for software engineering itself. Mathematicians apparently care deeply for their subject, their field, and its future.
YeGoblynQueenne··on A misalignment of AI in mathematics
Thanks. Regarding plagiarism, I agree that it is possible to check; I don't know how easy it is. But I do note that the mathematicians who complained that their work was plagiarised by AI did not seem to realise that they, themselves, were using a plagiarism machine and were instead quite comfortable admitting that, yeah, we worked our Euler result out together with Sol, Claude and the gang.

To clarify, they do attribute the original ideas to Cordoba and Martinez-Zoroa, but they don't seem to acknowledge that their result which they say was achieved "with a great deal of help from LLMs" is also derivative of others' work.

Independence is about avoiding making errors because of the influence of error-prone models. I guess I didn't explain it well.

YeGoblynQueenne··on A misalignment of AI in mathematics
>> I’m no big AI cheerleader, but why don’t you try these models to see what they offer - they might suggest some things you haven’t thought of, or save you some time in your research.

One reason not to try is to create an air gap between the OP's ideas and the data that future models can train on. We saw that the mathematicians who trusted OpenAI and Anthropic with their preliminary work found the rug pulled out of their feet by those same companies.

Another reason is to avoid inadvertently plagiarising the work of other mathematicians. Any mathematical insight that comes out of an LLM is the result of training on the entire bibliography of mathematical research, but those insights are spat out of the models without attribution. If you use AI in your matematical research you are only using the work of others without even knowing who they are and what they contributed.

And yet another reason is to avoid polluting your mind with the ideas that come out of the AI. Maybe you get a hint that pushes you to one direction, when you would go into an entirely other direction without that hint. And then maybe that becomes a habit and you can't find new directions without asking the all-knowing oracle.

tl;dr: opsec, integrity and independence are the reasons to not use LLMs in your research. I don't.

YeGoblynQueenne··on A misalignment of AI in mathematics
Don't you mean mass investment?
YeGoblynQueenne··on A misalignment of AI in mathematics
Oh, the patient is ill. Let's kill the patient.
YeGoblynQueenne··on A misalignment of AI in mathematics
>> I was apart of the "covid chess cohort" and got really into it for quite a few years. In that time, I also learned about what the chess culture was like pre-engine and post-engine, and I think you could argue that post-engine did kind of make the game more mechanical - and that kinda ruined things for people that grew up in the pre-engine era. it felt like pre-engine chess had a lot of mystique to it because knowing the best move was never truly clear. the only way to know what was best, was by: getting better yourself, socializing at chess club, learning from better players or coach.

I need to be very precise here, but it is not right to say that we know what the best move is in any chess board position. What we know is what move a chess engine would make that would win the game against a human, or sometimes another chess engine. We know a winning move; not the best move.

Winning in chess is not the same as knowing what the best move is. Having chess engines that can beat any human in chess is not the same as knowing how every possible game will develop. The latter is known as "solving chess" and we don't yet have that. We have fully solved games like tic-tac-toe, backgammon and checkers, so that there exist databases of every possible path through those games but this is not done for chess, and there is no hint that it is even possible to do given that chess is such a combinatorially more complex game than those.

More to the point, the upshot of the fact that we now have super-human chess engines that can beat any human in chess (and Go and shoggi) is that there is no longer the will to research the bigger question about solving chess. The solution of the minor problem, how to beat humans in chess, has sidelined and displaced research in the major problem, how to solve chess, and we will now never find the solution to the latter.

And if that reminds you of something, well, yes, exactly.

YeGoblynQueenne··on A misalignment of AI in mathematics
>> This sounds a lot to me like people in the 90's complaining that computers were destroying chess.

Genuine question: who were those people saying that? I don't know that criticism.

The criticism I know is from AI researchers and it is that beating humans at chess using a computer running an algorithm unlike anything that humans do when they play chess, tells us nothing about the way that humans play chess, which is what we are trying to understand when we try to get computers to play chess.

This criticism is exemplified by John McCarthy's article (the real godfather of AI; because he named it) "Ai as Sport" whence I quote:

Ideas about chess algorithms as well as advances in computer hardware were involved. However, it is a measure of our limited understanding of the principles of artificial intelligence (AI) that this level of play requires many millions of times as much computing as a human chess player does. Moreover, the fixation of most computer chess work on success in tournament play has come at scientific cost.

In 1965 the Russian mathematician Alexander Kronrod said, "Chess is the Drosophila of artificial intelligence." However, computer chess has developed much as genetics might have if the geneticists had concentrated their efforts starting in 1910 on breeding racing Drosophila. We would have some science, but mainly we would have very fast fruit flies.

https://www-formal.stanford.edu/jmc/newborn/newborn.html

A more recent criticism that can be made is that getting computers to beat humans at chess (and later Go, and Shoggi and Atari and Stratego) has turned out to only be possible with techniques that are too narrow to have real-world application. Btw, this includes Reinforcement Learning which is still much more capable in virtual environments than in any real-world environment.

YeGoblynQueenne··on More questions about whether researchers can trust OpenAI with unpublished math
You're assuming too much from a single graph with vaguely named axes.

I'll wait until there's more concrete information.

YeGoblynQueenne··on A misalignment of AI in mathematics
So you're not going to do it yourself and you want someone else to do it for you? Some mathematician that dedicated their life to understand mathematics must now toil unpaid and unwillingly to understand the AI slop proofs that you want us to be able to understand?

Do the job yourself. And if you can't, that's maybe a hint that you should listen to the people who can.

YeGoblynQueenne··on A misalignment of AI in mathematics
>> Humans can then work on clarifying why it's true.

Presumably you're a human. Are you going to do that?

YeGoblynQueenne··on More questions about whether researchers can trust OpenAI with unpublished math
>> And you’re seriously going with “don’t ask inconvenient questions” as an argument?

Where did I say that?

The conversation you seem to want to have doesn't seem to have any obvious use and you seem to want to debate a point I never made. I don't seem to be needed here so I will now bow out of the conversation.

YeGoblynQueenne··on More questions about whether researchers can trust OpenAI with unpublished math
>> Those same people are now given tools to produce works that they enjoy and can do pretty damn well at.

It's the tools that produce the works, not the humans.

>> This is a horrible perversion of the quote.

What? The full quote is "Those are brave men knocking at our doors. Let's go kill them!". What's the perversion?

YeGoblynQueenne··on More questions about whether researchers can trust OpenAI with unpublished math
Do I really need to prove that humans have cognition? Or, more to the point, do you want to challenge that assumption?
YeGoblynQueenne··on More questions about whether researchers can trust OpenAI with unpublished math
This is no time to despair. It's the time to take a stand. If you don't want to see your discipline go the way of the dodo, then do something about it.

I don't know what you should do because I'm not a mathematician. But superintelligence schmuperintelligence. We didn't stop running because we have cars or playing chess or Go because there's chess and Go engines. Even more so than chess there's no point in maths unless it's people doing it, for other people. AI maths makes no sense, like AI art makes no sense, because those are things that people enjoy and can do pretty damn well ourselves so there's no point to automate them away. We gotta stop that bullshit, and we can stop it. And if we don't, if we just sit around and wait for OpenAI and Anthropic to destroy society then that's not their fault but ours.

Sorry, I'm not great at pep talks. Those are brave men. Let's go kill them!

YeGoblynQueenne··on More questions about whether researchers can trust OpenAI with unpublished math
Why do I need a source? An LLM prompt does not "move to action", because an LLM does not act. People act, animals act, software doesn't act. Acting implies volition and volition implies cognition and if you think that LLMs have those things then you're the one who should provide a source for your claim.
YeGoblynQueenne··on More questions about whether researchers can trust OpenAI with unpublished math
That's a jingle fallacy.

*Jingle-jangle fallacies are erroneous assumptions that either two different things are the same because they bear the same name (jingle fallacy); or two identical or almost identical things are different because they are labeled differently (jangle fallacy).[1][2][3] The term was coined by Truman Lee Kelley in his 1927 book Interpretation of educational measurements.[4] In research, a jangle fallacy is the inference that two measures (e.g., tests, scales) with different names measure different constructs. By comparison, a jingle fallacy is the assumption that two measures which are called by the same name capture the same construct.[5][6][7]

https://en.wikipedia.org/wiki/Jingle-jangle_fallacies

YeGoblynQueenne··on More questions about whether researchers can trust OpenAI with unpublished math
>> Their final “product” is not just a standalone ML model. They don’t need your data just to “improve their products and services”. They build a whole ecosystem and infrastructure around gathering all the knowledge in the world. Including private and secret knowledge traditionally gathered by “intelligence” agencies. Now artificial intelligence agents can do the same.

And people thought Experts Systems were bad.

YeGoblynQueenne··on More questions about whether researchers can trust OpenAI with unpublished math
Wouldn't they just list the number of problems solved then?
YeGoblynQueenne··on More questions about whether researchers can trust OpenAI with unpublished math
>> If better training data is the reason here, it would still be a case of the models doing something that is in and of itself super useful! The models really can take that data and distill it into solutions for similar problems faster than humans can. This is great!

It's perhaps great in the short term although it's not very clear who it's great for. I'm not sure mathematicians find it all so great, I mean.

In the long term, if this contrives to destroy the tradition of human mathematics the whole endeavour is self-defeating. In time, there will be nobody left with the knowledge and skills to produce mathematics to train AI to do mathematics.

And then we'll be left with no mathematics at all: we'll have no human mathematicians and no AI that can do mathematics, either.

← PreviousPage 4 of 34Next →