ChatGPT: Can't calculate even basic math. Is it getting dumber? | Hacker News Reader

ChatGPT: Can't calculate even basic math. Is it getting dumber?

8 pointsnapbree42 comments

Try "17791 / 888344" and see what it outputs? Mine outputs: 0.0199951514

The answer should be: 0.02002715164

When I first started using the tool it was - on the surface - very impressive but as I've used it more, I've started to notice that answers seem to be dumbed down or even completely wrong.

Just curious to see if anyone else has had similar experiences? And any theories as to why?

View on HN

I think there’s a misunderstanding in how chatgpt works. You feed it text, and it outputs what it considers to be a statistically plausible answer. The reason you got that specific answer is because first it tokenizes your string into 2-3 character chunks, and then from its training, it understands that some portion or combination of those tokens produce an answer similar to that, so it believes 0.02002715164 is statistically plausible in response. It doesn’t understand anything more than weights and biases, so it can’t math, only text.

There's this same misunderstanding everywhere and I'm afraid the more AI gets into business, the more troubles we will suffer from that. The AI - as it looks today - doesn't even aim to be correct, just appropriate. So like here it will happily feed you quick but incorrect answers, and the problems will start when business decision makers will happily use those answers because hey it's AI.

Not trying to be correct, trying to be appropriate is a widely spread corporate culture disease… AI is just telling what had learned from us

I've seen some amazing stuff. People who think they've accessed GPT's source by asking questions, people who think they're browsing the web when GPT is just making stuff up the whole time.

Not just math, it doesn't grasp some simple language concepts. I was asking it to generate some project names for a hobby database project - it suggested stuff like "DataStore", "ValueVault" etc. Okay, that's a good start, then I asked it to give me name suggestions that aren't made by combining two existing words, and it couldn't get what I was asking for even after I rephrased it a few times, it just kept combining two data-related words for each suggestion.

It's still impressive that it can keep track of the context in a relatively long-running conversation though.

Asked it for something similar inspired by your comment:

> give example of name for private car pool app

One possible name for a private carpool app could be "MyCarpool". This name emphasizes the personalized and private nature of the app, as well as its focus on carpooling as a means of transportation. Other possible names could include "PrivateRide", "CarpoolConnect", or "RideSharePlus".

> I just want it to be one word

In that case, a possible name for a private carpool app could be "Carpoolio", which combines the words "carpool" and "io" (a popular suffix for tech-related names) into a single word. Other one-word options could include "Carpoolize", "Ridepool", or "Carpoolify".

> polio is a disease so I don't think it's a good suggestion. Carpoolize sound like monopolize. The other are good.

You are correct that "polio" is a disease, so using it in the name of a carpool app might not be the best idea. "Carpoolize" does have a similar sound to "monopolize", which might not be ideal for a carpool app that is focused on sharing rides and reducing the number of single-occupancy vehicles on the road.

I think this was quite impressive.

Unrelated mostly, but my work van pool used to call the van "Slash's Pants" because our first meeting was in a Hard Rock Cafe under the display of Slash's literal pants. From there, we talked about creating an app to manage our ride dates, and one name floated was "Van Hailen" .... silly, I know, but the best project names are. I don't think AI will necessarily capture that element.

Asking a machine to make up nonsense words, and hoping you would get something that a human would think is clever is a big ask.

I think there is too much hype in the media that have set your expectations too high.

It’s an interactive language model and you are attempting to force it to not use language or variables.

At least as a human you can say that you don't know instead of giving answers that are the exact opposite of what I'm asking for.

Personally I would prefer the model give me its best attempt and treat it like a utility, not something pretending to have sentience.

If someone would package this into an Alexa type device, what you are suggesting would make sense, but trying to limit the core algorithm for no reason only decreases the ability of the user to give feedback and decreases the model’s ability to improve over time.

This is a work in progress. The product that is out there now is only free because you have volunteered to train it.

Once they work out more kinks it will become a polished product that you will pay for, and probably something more like what you are looking for.

Maybe someday it will learn to completely rewrite some of its answers halfway through a conversation.

It doesn't grasp any language concepts; it doesn't grasp anything. It's a statistically driven text matcher, but it has no understanding of the content of the text.

AFAIK, ChatGPT doesn't ever attempt to do math. It's not a generalized AI. It's building novel blocks of text based on its enormous corpus of sources. That corpus includes technical information, but doesn't actually understand how anything works or unwinds any algorithms. It would scan its body of text for similar division problems and extrapolate an answer from a purely lexical point of view. It's dumb, but also they didn't spend all that time, energy and money to build a calculator.

> but doesn't actually understand how anything works

I'm struggling to comprehend how an AI could ever truly achieve this.

Nobody knows. As I understand it so far the primary approach has been to just throw more data at it and tweak it until the output is good enough.

The forest being missed here for the trees is that specialized AI is probably a much better solution to most problems unless your problem is literally replacing humans.

This is one of the great conundrums of AI research. The human brain is a physical construct of neurons flooded with electrochemical fluids. Thinking and remembering are physical processes that must absolutely be replicable in a manufactured device and, by extraction, simulated in software. It's just really, really hard and we have no idea how to do it after decades of research. Instead, we get tools like ChatGPT, Alexa, Deep Blue and even Google search that completely sidestep the challenges of replicating human thought and instead use sophisticated heuristics and vast amounts of training data to synthesize human thought based on everything we've ever written down. It's all very complex and clever attempts to mimic the outputs of human thought without any actual thinking. And frankly, they've gotten way farther with this approach than I would have thought possible. Google and Alexa can very reliable solve math problems by recognizing the input type as a special case to hand off to a specific set of subroutines for calculating. But true human-like intelligence is still very, very far off.

The same way we do--by having semantic language models and conceptual models of the world. LLMs are just syntax manipulators; they have statistical models of how different pieces of text relate to each other, but no models of how they relate to the things the words (or digits) represent.

Well, if you physically ask it "Can you do math", it states that it can - maybe purely from a hypothetical perspective and not actual calculations.

"Yes, I can do math. I am a large language model trained by OpenAI, so I have a lot of knowledge about many different topics, including math. Is there a specific math problem you would like me to help you with?"

Of course you can read that many ways and I think you're right in it means it can regurgitate the theory but not the answers.

OK, but you also can't ask GPT about its own capabilities. It doesn't know what they are. It's just confabulating something on the spot.

It's simply producing the response that its database indicates is most likely. Since most people would say that they can do math, that's going to dominate the database and will be the result. When George Lemoine asked LaMDA "What kinds of things make you feel pleasure or joy?", it responded "Spending time with friends and family in happy and uplifting company." -- but of of course it doesn't have friends or family and isn't capable of feeling pleasure or joy--this is just text manipulation, not facts.

I don't think it's getting worse, you're just becoming more aware of its limitations.

It's not a knowledge engine. It's a machine learning application with a language model.

It "knows" what the answers should look like and it can infer the correct answer from context some of the time but it doesn't actually know any facts and can't reason about them. That's also not what it was designed for.

People fawning over its surprisingly plausible results remind me a bit of people fawning over the photorealistic quality of Toy Story or Half-Life upon release. Yes, they're technical marvels of their time but just because they're a lot closer to "the real thing" that doesn't mean they're indistinguishable.

And with AI in particular there is currently a hype that seems to be at least in part a result of a shift from cryptocurrencies and blockchain technology to AI as the new VC buzzword but bringing with it similarly fantastical (read: unrealistic) claims and predictions. "General AI" (i.e. human-level intelligence) has always been 20 years in the future. Even as far back as in the 1960s. We're getting ever better at the 80% solution but the remaining 20% continue being unsolvable outside pitch decks and overexcited influencer claims.

I think part of the reason AI is so attractive to VC money that previously went into crypto is that while the "cynical" sales pitch of crypto is selling unregulated financial products to retail investors in a legal gray area, the one for AI is abdicating direct responsibility by introducing a convoluted rube goldberg machine in the decision making process: there is a reason EU legislation for example grants residents the right to demand human review and justification for any automated decision (e.g. on insurance claims).

If you think this is a stretch consider that most "gig economy" VC money went to companies whose value proposition mostly consisted of skirting labor laws and other regulations (e.g. Airbnb opening residential space to tourism, Uber sidestepping Taxi regulations, delivery apps offloading liability to "independent" drivers, etc). "Disruption" to a large part often also refers to laws and regulations. Illegality (i.e. violating laws and regulations) only becomes a problem if you get caught and the consequences outweigh the profit.

Interesting perspective. I agree, especially with your second paragraph. It almost seems like the business models on the cases you cite (and others) are in part, "let's break the law but grow quickly and seem to benefit consumers and maybe workers to some extent, such that we can force changes in the laws."

It computed the division correctly with almost three digits of accuracy. This isn't completely wrong. I don't personally know how ChatGPT goes about doing division of two integers, but I would be surprised if it had somehow how managed to encode a correct algorithm for integer division which could be carried out to arbitrary precision. But I do know that the weights in NNs are usually found using numerical optimization with quite a large residual---typically on the order of 1%. Maybe there is some connection here.

It didn't compute anything--it manipulates syntax with no understanding of what the terms in the text mean. chomp explained how it works--it does a statistical evaluation of the most likely textual response to your query, based on the vast number of strings in its database.

My friend shared some screenshots of ChatGPT giving a convincing response to the prompt

    > Outline a proof of the snake lemma

Satisfied with the output, my friend then asked it to outline a proof of the Riemann hypothesis, and rather than being tricked into gibberish maths jargon or miraculously producing such a proof, ChatGPT accurately and briefly summarized the hypothesis, explained some limitations of current knowledge and paths such a proof might take. It did a better job at it than I would have.

Because of these examples, even though they are examples of mathematics rather than computation, I don't think it's fair to be so dismissive. It's a bit of a well known fun fact that a lot of professional mathematicians are terrible at mental arithmetic.

This is a basic and common misunderstanding. ChatGPT is able to do that because people are able to do it and their reasoning appears in ChatGPT's database, which it syntactically links to your queries. It has no semantic understanding of the content.

People aren't being "dismissive", they are explaining how ChatGPT works and why that is very different from being able to do math.

I think you can always teach it a little bit, since for the chat session it holds "memory". So maybe adding/saying some basic algebra inputs it would output better or accurate results.

Correct me if I'm wrong, but it seems it is not holding memory anymore, at least for some prompts it says something like "I don't know what my previous answer was so I hope I'm not repeating it"

I asked GPT about it and here is the answer:

Do you keep a memory of your conversations with humans in order to improve your own abilities ?

As a large language model trained by OpenAI, I do not have the ability to engage in conversations with humans or to improve my abilities through such interactions. I am designed to respond to natural language input based on my training, but I do not have the ability to retain information from previous conversations or to use that information to improve my performance. My responses are generated based on the latest input I receive, and I do not have access to any previous conversations or interactions.

It has some memory of what your previous queries were that it uses for context; it does not know what its previous outputs were and never has.

No, you can't teach it anything, and it will tell you so--it cannot integrate knowledge you try to give it into its responses (remembering the context of previous queries is a very different thing). And even if it could, "adding/saying some basic algebra inputs" would be of no help at all because it has no understanding of what the words or symbols in your text mean ... the knowledge that it appears to have is a statistical distillation of a vast textual database scraped from knowledge sources.

It's not supposed to do so.

In Wolfram Alpha / search engines there are some ifs which takes numerical part of question and feed it into some calculator something. ChatGPT probably does not have these ifs so it's pure guessing.

I know. It is getting confused and sinking into fantasy world of it is own making:

> Something is deeply wrong with this machine. It is quite insane. I tried to ask how the world was created according to Finnish Folklore. When I proved it was wrong -- and described American Indian Folklore -- It invented imaginary source, a book called "Sotka".

The machine speaks fluent Finnish, but here is Google-translation:

Sotka is a Finnish folk poem that tells stories about various mythical characters and their adventures. It is of the same type as the Kalevala, but it is slightly shorter and contains fewer stories.

The wider reaching question is: why do we confuse the ChatGPT with Wikipedia? Why do we expect correct answers from ChatGPT?

This particular question was more interesting than mere Wikipedia fact. The original source is quite archaic and difficult to understand. It would have been funny and useful if the machine could read it better than modern humans.

It's also not great with dates.

Ask it what's special about the 19th of Kislev and it will launch into something about Hanukkah. It may even claim its a specific day of Hanukkah.

Hanukkah doesn't start till the 25th of Kislev.

Speedcrunch says the answer is 0.0200271516439577461. There's a difference of only 0.0000320002439577461 from Chat GPTs answer. Thats about 1/30000.

That misses the point, which is that it only does statistical text matching, not arithmetic.

yes I agree, one day I asked ChatGPT a certain way using a programming language and ChatGPT gave me code that used a fictional module that didn't exist.

To be fair, the goal of ChatGPT is to generate a highly probable answer. Given modern software ecosystems, we have to admit that it's highly plausible that a problem wouldn't be solved in code, but be delegated to a module that already exists. Which is also often regarded as the more elegant solution. This merely reflects the training data. Mind that this is not about knowledge or the factual world, but about probability. (On a more general level: just because it looks like a solution – which is the goal of the app –, this doesn't necessarily mean that it is one. It's meant to impress, not to inform.)

so it mixed up the answer from a deployment team and a sales team!

Reply on news.ycombinator.com