24,944 karma · joined September 26, 2015
What is reasoning?
In computer science and AI when we say "reasoning" we mean that we have a theory and we can derive the consequences of the theory by application of some inference procedure.
A theory is a set of facts and rules about some environment of interest: the real world, mathematics, language, etc. Facts are things we know (or assume) to be true: they can be direct observations, or implied, guesses. Rules are conditionally true and so most easily understood as implications: if we know some facts are true we can conclude that some other facts must also be true. An inference procedure is some system of rules, separate from the theory, that tells us how we can combine the rules and facts of the theory to squeeze out new facts, or new rules.
There are three types of reasoning, what we may call modes of inference: deduction, induction and abduction. Informally, deduction means that we start with a set of rules and derive new unobserved facts, implied by the rules; induction means that we start with a set of rules and some observations and derive new rules that imply the observations; and abduction means that we start with some rules and some observations and derive new unobserved facts that imply the observations.
It's easier to understand all this with examples.
One example of deductive reasoning is planning, or automated planning and scheduling, a field of classical AI research. Planning is the "model-based approach to autonomous behaviour", according to the textbook on planning by Geffner and Bonnet. An autonomous agent starts with a "model" that describes the environment in which the agent is to operate as a set of entities with discrete states, and a set of actions that the agent can take to change those states. The agent is given a goal, an instance of its model, and it must find a sequence of actions, that we call a "plan", to take the entities in the model from their current state to the state in the goal. This is usually achieved by casting the planning problem as pathfinding over a graph with a search algorithm like A*. Here, the agent's model is a theory, the search algorithm is the inference procedure, and the plan is a consequence of the theory. Deductive reasoning can be sound, as long as the facts and rules in the theory are correct: from correct premises we can deduce correct conclusions. We know of sound deductive inference rules, e.g. A*, and Resolution, used in automated theorem proving and SAT-Solving, are sound.
The classic example of inductive reasoning is inferring the colour of swans. Most swans are white (apparently) so if we have only seen white swans we have no reason to believe there are any other colours: we are forced to infer that all swans are white. We may only be disabused of our fallacy if we happen to observe a swan that is not white, e.g. a black swan. But who is to say when such a magnificent creature will grace us with its presence, outside of Tchaikovsky's ballets? Induction is thus revealed to be unsound: even given true premises we can still arrive at the wrong conclusions. Another example is the scientific method: imagine an idealised scientist, perfectly spherical, in a frictionless vacuum. She starts with a scientific theory, then goes out into the world and makes new observations about a phenomenon not described by her theory. She constructs a hypothesis to extend her theory so as to explain the new observations. The hypothesis is a set of rules, where the premises are the consequences of the rules in her initial theory. Then, being an idealised scientist, she goes looking for new observations to refute her hypothesis. Science only gives us the tools to know when we're wrong.
Abductive reasoning is the mode of inference exemplified by Sherlock Holmes. We can imagine Sherlock and Watson standing outside a tavern in London, watching as a gentleman of interest steps out of the tavern with egg on his lapel. "Ah, my dear Watson, what can we conclude from this observation?". "Why my dear Holmes, we can conclude that the man had eggs for breakfast". Holmes and Watson can arrive at this conclusion, about a fact that they have not directly observed, because they have a theory with a rule that says "if one eats eggs, one may get some on one's lapels". Working backwards from this rule, and their observation of egg on the man's lapels, they can guess that he had eggs even if they didn't directly observe him doing so. Abduction is also unsound: the man may have swapped coats with an accomplice, who was the one who had eggs for breakfast instead.
And now you know what "reasoning" means. So the next time someone asks: "what is reasoning?", you can let them know and turn the discussion to more interesting, more productive directions.
[ And this was a debug session from hell, enormously helped by an AI
doing much of the grunt-work.
I'd like to call it my tireless helper, but the AI several times
stated flat out that this was impossible and unsolvable and that we
should just write a report about it.
I suspect those things have been trained by people who may not be
quite as stubborn as I am.
But while the AI was ready to give up several times, it did keep
adding debug code and analyzing it faithfully when I pushed. So credit
where credit is due and I let the AI write the commit message above.
This is basically a one-liner fixing a bogus "round_up()" to a
"round_down()", but there were 24 patches adding more and more debug
information to this, and 18 kernel boot to finally narrow it down to
this. - Linus ]So, er, what is the um narrative conveyed by this scene?
So, are you a vet or are you a psychologist? I'm confused.
>> Inside OpenAI’s Reboot
Everyone who treats plant-based diets with any degree of responsibility will stress the need to plan them carefully, considering the needs of the individual and their stage of life, and to accompany them with supplements to make up for the nutrients that not even well-planned diets cannot provide.
For example, "Top Tips" from the British Dietary Association:
* Carefully planned plant-based diets can support healthy living at every age and life stage
* Plant-based diets can help to manage weight and may reduce the risk of type 2 diabetes and other chronic diseases
* You can get all essential nutrients from plant foods but vegans need to ensure a reliable source of vitamin B12
* If excluding dairy, make sure you consume other calcium-rich foods
* Include a wide variety of plant-derived foods to ensure your diet is balanced and sustainable
https://www.bda.uk.com/resource/vegetarian-vegan-plant-based...
Far, far from "just buy some beans/lentils". TBH that's just irresponsible and dangerous advice.
More radically, Singer suggested that meat-eating may be permissible if “farms really give the animals good lives, and then humanely kill them, preferably without transporting them to slaughterhouses or disturbing them. In Animal Liberation, I don’t really say that it’s the killing that makes [meat-eating] wrong, it’s the suffering.”
https://www.newstatesman.com/ideas/2021/05/peter-singer-why-...
Full disclosure: I have profound disagreements with Singer's utilitarian philosophy.
Giving vegan food to cats is animal abuse. If you can't stand the idea of feeding other animals to a cat, don't keep a cat as a pet. Dogs can handle plant-based (but not vegan) diets much better. There are many other animals you could keep as a pet, also. E.g. goats are vegetarian and they have strong, interesting and engaging personalities. If you want a vegan pet, get a goat.
Edit: And if you don't have enough space to keep a pet goat then you probably shouldn't be keeping a pet dog or a pet cat either. One reason people abuse the animals they keep as pets is they assume that an animal is just like a little, furry human and can thrive and be well in any condition a human can, e.g. like living in an apartment etc. This is so far from the truth it's crazy. Most people should really not keep pets.
EDIT:
I don't want to get into a holy flamewar (vim vs emacs? Oh dear) but I've heard that criticism before. I think the reason is that there's a kind of program that is absolutely trivial to write in Prolog: business logic, rules engines, etc, anything where you can write out the program logic as a set of facts and rules. I mean duh. A Prolog program is a set of facts and rules. But at that point you're* not even trying to use the language. Then of course, if you try to do something more complicated it will take more effort and you will need to really understand how Prolog works; which is not trivial.
So I think it's a bit like mathematics. Yeah, if you just want to add 1 and 1, it's easy. If you want to calculate the amount of fuel it will take to fly to the moon and back, it takes considerably more effort. You can see it as maths being "easy for the easy things but hard for the hard things", but it's just that the hard things are hard and take more work, not the fault of maths as such.
__________
* "You" in the abstract, not you personally.
Prolog is one of the two classic languages for symbolic artificial intelligence programming (the other classic language being Lisp). Prolog excels at implementing symbolic rule-based systems in which declarative knowledge is encoded in first-order logic. The language is optimized for expressiveness and efficiency for these types of applications, sometimes at the expense of logical purity. For example, by default Prolog does not use the "occur check" in unification. From a math/logic standpoint, this version of unification is incorrect. However, the occur check is expensive, and in most cases the lack of the occur check is not a problem. This is a very pragmatic design decision, as is Prolog's use of depth-first search, and use of cut (!) to control backtracking. I'm sure these decisions were absolutely necessary when running on the hardware of the 1970s, and today are very useful when working on large problems, and when dealing with huge (often infinite!) search spaces.
Prolog supports many "extra-logical" or "non-logical" features, including cut, assert and retract, projection of variables for arithmetic using is, and so forth. Many of these features make it easier to express complex control flow, and to manipulate Prolog's global database of facts. One very interesting feature of Prolog is that Prolog code is itself stored in the global database of facts, and can be queried against at run time. This makes it trivial to write meta-interpreters that modify the behavior of Prolog code under interpretation. For example, it is possible to encode breadth-first search in Prolog using a meta-interpreter that changes the search order. This is an extremely powerful technique that is not well known outside of the Prolog world. 'The Art of Prolog' describes this technique in detail.
Tremendous effort has gone into improving Prolog implementations, most of which are based on the Warren Abstract Machine (WAM). The WAM uses a side-effecting model in which values are destructively assigned to logic variables, with these side-effects being undone upon backtracking. Many features can be added to Prolog by extending the instructions of the WAM. One disadvantage of this approach is that Prolog implementation papers can be difficult to read without a solid understanding of the WAM. On the other hand, Prolog implementer have a common model for discussing implementation issues. There has been a great deal of research in parallel Prolog, culminating in Andorra Prolog in the 1990s. At least some of these ideas live on in Ciao Prolog. (Ciao Prolog is full of interesting ideas, many of which go far beyond the Prolog standard.)
Prolog has a beautiful unification-based "pattern-matching"-style syntax that results in very succinct programs. Prologers love their syntax, just like Lispers love their s-expressions. Prolog also has a large library of standard predicates. Due to all of the engineering that has gone into making the WAM fast, there are very capable and mature Prolog implementations. As a result, many large knowledge-based systems have been written entirely in Prolog.
So, why do you say that Prolog is a "poor general purpose language"?
Which is to say, I think your criticism is unfair and an attempt to avoid engaging with the arguments in the paper.
I don't agree that it's a useless term. AI, as a research area, is well-defined by its journals and conferences and the grants that support that research. It's not the fault of the AI research community that OpenAI's app went viral and now there's so much confusion in mainstream discourse about the term. I try to set the record straight whenever I can but it does feel like fighting a pointless rearguard action, even in this community that is better informed than most.
What I can tell with some certainty is that, once again, in UK academia, the person blowing the whistle on plagiarism is the one facing consequences whereas the plagiarist has the full support of the academic establishment.
It is an enduring scandal of British academia that misconduct goes unpunished, swept under the rug, ignored and whitewashed. Liverpool John Moores University which had awarded Jason Arday's PhD carried out an investigation that found that copying another academic's work and submitting it under his own name was nothing but an "honest and reasonable error", not deliberate plagiarism, and "well within the accepted range of compliance with the academic standards of the time" [1].
It is also typical that Jason Arday himself did not contest the fact of plagiarism: he only sought to excuse it by shifting the blame to others: "Arday did not dispute the allegations of plagiarism, but said he had not received adequate supervision given his autism and learning disabilities, and described the allegations against him as being part of an "insidious and relentless" strategy to undermine black academics." (ibid).
This, too, is a recurring pattern with plagiarists: because the facts of the matter are often so blatantly obvious for all to see that pretending they didn't happen is too much even for academic integrity panels to ignore, the plagiarists tacitly accept the facts but seek to contest their meaning. And academia, ever fearful of scandal, duly complies.
The first victim of all this is nothing else but the quality of scholarship. If plagiarism is normalised, always treated with impunity, what else can be the result? Second-hand, stolen scholarship is shoddy scholarship.
[1] https://en.wikipedia.org/wiki/Jason_Arday#Initial_concerns
But I also tried it on linux (Fedora, whatever version I was running at the time) and WSL (Ubuntu this time) and I still couldn't get it to work.
Checking my notes, last time (03/25) I was trying to get it to work with an LSP server a colleague had created for an in-house programming language. After pulling my hair out with installation instructions and getting the right version to play ball with the LSP server it seems I hit a snag with neovim not picking up the vimrc_example.vim file from the location it said it should, and then an error when I removed the file at the path of the vimrc_example it wasn't picking up options from (meaning it really was accessing the first file but just ignoring everything in it). It seems I was trying to use that file because it was recommended by the neovim dox online (I think the dox said to create an init.lua file pointing to that vim file).
So I gave up on neovim and went back to using vim.
Now, what's all that puerile invective in the article above about? How is swearing at your users going to make them appreciate your work? Wouldn't it be better to just have something that's easy to use? I've installed vim on all my computers running every major operating system and it's never gave me any trouble with picking up its vimrc or anything else really (except plugins... plugins suck).
Why is neovim so hard to get started with? It's a fork of vim, right? What did the devs change that so drastically changes the experience using vim? I don't understand.
AI research is not over yet. If OpenAI and Anthropic fail to bring on the Singularity, what are we going to call the continuing research on AI? Are we going to call it something else than "AI" because that was taken by LLMs? Does that make any sense at all?
Same as with passports. Even in the Anglosphere people have passports with passport numbers. They could be used to uniquely identify the passport holder, except of course there are many passports, and therefore, many numbers, for each passport holder.
Small problem? Just update whatever database is holding the information for sanctionted persons (or, indeed, persons who should not be sanctioned)?
Turns out, that is not a small problem.
... unless, of course, it's Perl. Right?
I don't agree that LLMs can evaluate an experiment. There's nothing in LLM training that makes them capable of telling what is e.g. a correct hypothesis from an incorrect one. I know that is a common claim particularly encouraged by AI companies but whenever that claim has been studied systematically and carefully the result is that self-verification doesn't work. For example, see:
On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks
https://arxiv.org/abs/2402.08115
Note also that basically all the mathematical results published so far have to be checked by an external verifier, either human mathematicians or a proof assistant like Lean, or both, and some systems explicitly couple an LLM generator to a traditional solver, like e.g. AlphaProof. None of this would be needed if LLMs could really evaluate their own results in any reliably correct manner.
But, learn what? All those ideas where wrong. How does an LLM "learn" from incorrect proofs that it has generated itself? What does it learn? Can you explain how this mechanism works?
It depends what kind of automated theorem prover we're talking about. I'm not an expert on proof assistants like Lean. I am an expert on Resolution-based automated theorem provers and those can be implemented efficiently. For example, the SLD-Resolution based theorem prover used as the interpreter for Prolog (the logic programming language) runs in linear time and the cost of completing a proof is dominated by the cost not of the prover itself but the complexity of the theorem that is being proven. In more plain English when you run a Prolog program the cost that really matters is that of the program, not of the Prolog interpretation.
Now, I know that Resolution-based theorem provers aren't typically used in the same way as proof assistants let alone LLMs; but that is more of a historical accident than a limitation of the technique. There's no reason why one couldn't search for a proof of the Riemann hypothesis expressed as a Prolog program and using a Prolog engine. It's just not where most peoples' heads are these days.
>> Now LLMs have produced multi-thousand line Lean proofs. This is impossible by simply "try everything and see what sticks". LLMs are able to target their efforts to only promising proof strategies. Yes it helps that they work at superhuman speed, so they can try thousands of strategies where a human might try a dozen. But their results cannot be explained only by compute increases; they need genuine mathematical insight.
I understand the argument and I believe it has merit, but that's just to say that LLMs are trained to generate likely text, like I say in another comment. That is enough to explain the much improved ability to search quickly and efficiently (the LLM just has to generate text that looks like a proof; no need to actually carry out the steps of a proof) without recourse to 'genuine mathematical insight'. Which should be easy to believe because it's much harder to explain what 'genuine mathematical insight' is and where it comes from than it is to explain where the ability to predict likely text comes from: it's how LLMs are trained.
My current framing of this is that the advantage of LLMs lies in their ability to generate the text of a proof without having derived each of its steps in order, like a theorem prover (automated or not) would have to. There's nothing forcing an LLM to derive conclusions from premises (or indeed making it at all capable to do that).
They don't have to understand what the proof they generate means, or to be able to tell whether it's true. In fact, they can't do either. But that's fine as long as it's possible to check the proof with an external verifier.
So most LLM-based proofs use the LLM as the generator and an external verifier as the tester, either a solver like Lean or a mathematician. That's the best of both worlds as far as generate-and-test goes. A powerful generator tied to a powerful tester.
EDIT: yeah, like this:
>> An example of a good division of labor is the SAT Attack on Tarski's High School Algebra Problem https://arxiv.org/abs/2608.08421 where they construct a formula with O(n⁴) variables and O(n⁶) clauses and use a SAT solver to show that it is unsatisfiable for n ≤ 11 but satisfiable for n = 12. Then they use an LLM to help them write a Lean proof that the SAT solver input is equivalent to the human-readable description of what they wanted to prove.
I'm not disergarding the fact that LLMs don't generate text completely at random. They generate likely text. I suspect that can make it more likely to generate the text of some correct proofs. But I have no idea how likely that "more likely" is or what proofs are those.
We have just got some very strong evidence about the way in which LLM-based systems solve mathematical problems and this evidence supports what many have already suspected including myself.
Here's what I'm talking about. On 10 August Anthropic released an article [1] claiming that:
An unreleased research version of Claude has improved on a longstanding lower bound for the fraction of zeros of the Riemann zeta function that satisfy the Riemann hypothesis. Drawing on extensive prior research by mathematicians over the past decades, it has increased this bound from 41.6% to 67.2%.
The same article describes the methodology followed by Anthropic's employee, Jarred Sumner, who prompted Claude, as follows:
Jarred Sumner, an Anthropic staff member (and non-mathematician), prompted Claude to “take a real stab” at the hypothesis itself, leaving the mathematical choices from there up to the model. Initially, Claude generated and tried 650 ideas, none of which worked. Jarred prompted Claude to try again, and it spent a day and a half coordinating about 60 Claude subagents, which this time went much deeper: between them, they ran 2,400 shell commands and wrote hundreds of Python scripts.1 The subagents ran thousands of numerical checks against known zeta zeros and refereed one another’s work. Throughout this process, Jarred's input was mostly limited to sending Claude messages of encouragement (mostly variants of “keep going” or “believe in yourself”).2 This seems to have helped Claude overcome some initial skepticism that it could make meaningful progress.
Jarred got Claude to throw stuff at the wall repeatedly (650 initial "ideas" plus unspecified more by "60 Claude subagents" ... running "2400 shell commands" and "hundreds of Python scripts") and then kept whatever happened to stick. In this case, by happy accident, what stuck was an improved bound of the zeroes of the zeta function etc.
This is how every single mathematical result reported by an AI company has ever been generated. They throw stuff at the wall and take whatever happens to stick.
This approach works. Not only it works, it is, in principle, a universal problem solver. "Millions of monkeys on typewriters" will eventually produce a proof of the Riemann hypothesis; or a disproof of it.
The key point being "eventually". Is this a way to do mathematics research? Can that replace mathematicians?
In AI, this method is well-known as the "generate-and-test" method. It is ancient, basal to AI if I may be so bold. It first appeared to my knowledge in the Logic Theorist, the proof-finding program that Simon and Newell presented in the 1956 Dartmouth convention that named "Artificial Intelligence", to such luminaries of AI and CS as John McCarthy (the real "godfather of AI" who named the field), Marvin Minsky, Claude Shannon and others.
We've had the ability to brute-force all of mathematics "eventually", given "enough" compute for nearing a century now. Why haven't we solved all of mathematics? Are LLMs really so special that they can out-brute force search every previous brute force searcher?
Well, you tell me, HN. I say: no.
___________
>> The Bronze Age remains include urns associated with human burial.
Never said anything like that.
In the left half of the video you can see that the robot is (trying to) exactly match the folds of the human in the right half and it's doing so while folding the exact same garments on the exact same surface.
The right half of the video is not a training demonstration, I don't think, since the robot is trained by teleoperation AFAICT (it needs to because it must use its head-mounted camera to control its movements) but that just underlines the degree to which their training regime is exactly copying the movements of a trainer, on the same garment, in the same environment.
This is a limitation of the training approach, by RL. With RL you learn a mapping between sets of pixels (as in the video that comes in through the robot's camera) and robot actions (as in actuator commands). What that means is that once a policy is trained and the robot is deployed, if the input pixels are significantly different than the input pixels at training the robot doesn't have a policy that matches the input pixels and so it can't find the right actions to take. So they have to keep the training and deployment garments and even the environments the same, or as similar as possible.
You can see some more evidence of this in the video right under the paragraph with the title "Hill-Climbing Reliability Through Post-Training". The robot at the front of the video, with the bright red trim, is shown trying to fold a grey t-shirt with white flower decorations and a frilly hem (how adorable <3). But you can tell it fails because the video stops before the robot has completed the fold. If it could complete it, you can rest assured that the video would be showing off the entire folding sequence as it does for the robot with the green cap on the other side of the bed.
That paragraph is making a claim about a "post-training" regime that's supposed to improve generalisation but it leaves more details to a "separate technical post". So I can't tell what it's supposed to be doing, but I don't think it's working.
When I watch videos like that I always remind myself that a) I'm watching a tech demo created to attract investment and b) I've watched way too many of those, going all the way back to the Boston Dynamic videos of Robot Dog or of Atlas doing backflips and yet the state of the art hasn't really budged since. Such videos make it easy to overestimate the state of the art in autonomous robotics and in fact are meant do precisely that: play up robots' true capabilities. It's just impossible to say anything about a robot's general capabilities by watching a few minutes or even a few hours of video. OtoH if you know what to look for you can tell everyone is basically stuck at the same level and trying the same things to escape it. The truth is robotic autonomy is several major breakthroughs away and nobody has any idea how to get there. So we'll be seeing many more of those tech demo videos in the years to come.
Before I put in the effort to reply in good faith I have to know: do you think we're going to have a conversation or are you going to fulminate and scold me like some kind of all-important authority (which I have to say you clearly are not)?
To clarify, I'm happy to have a curious and respectful exchange.