What ChatGPT can't do
auerstack.substack.com
auerstack.substack.com
With the way these models work, I'd assume that the benefit from chain-of-thought only happens when it gives the steps first (and so can base its final answer off of those intermediate steps), rather than the example in the article where it gives the answer first (and then has to generate steps to reach that answer without being able to change the answer).
When I tried the same prompt as in the article, ChatGPT explained its steps before giving the answer and was correct: https://i.imgur.com/lLS2xCX.png
You could probably push it in this direction through the prompt ("show your steps before giving the final answer").
> Doubles do not have 100 digits of precision in C. Can you write a program to calculate the first 100 digits of Pi in C that takes the precision of data types into consideration?
I was surprised that it made sense of that request and returned another program using an arbitrary precision library. When it did not work, I told it where the error was and it gave a probable cause for the error. Again, rather impressive.
Even if you broke the problem into (already proven) steps that may lead to a proof, you would need a strong background in mathematics simply to verify anything it claims.
(Sorry about editing out blocks. ChatGPT is verbose. I limited the removal to background material about the book.)
Here is the excerpt:
II2II: What was Polyanna glad about when she received crutches?
ChatGPT: In the novel "Polyanna," the character Pollyanna becomes severely injured in an accident and is confined to bed for several months. Eventually, she is fitted with crutches and learns to walk again, but she faces many challenges and setbacks along the way. (...)
II2II: But Polyanna received the crutches before the accident.
ChatGPT: I apologize for the mistake in my previous response. You are correct that Pollyanna receives the crutches before the accident in the novel "Polyanna."
ChatGPT: In the novel, Pollyanna is given the crutches as a gift by her wealthy but unhappy aunt, who is trying to cheer her up after her father's death. At first, Pollyanna is confused and disappointed by the gift, as she is not sure what to do with the crutches. However, she eventually realizes that the crutches can be used as props in a game that her father taught her called the "Glad Game." (...)
II2II: Are you sure the aunt gave Polyanna crutches?
ChatGPT: I apologize for the error in my previous responses. You are correct that Pollyanna does not receive crutches as a gift in the novel "Polyanna."
ChatGPT: In the novel, Pollyanna becomes severely injured in an accident and is confined to bed for several months. Eventually, she is fitted with crutches and learns to walk again, but she faces many challenges and setbacks along the way. Despite these challenges, Pollyanna remains optimistic and cheerful, and she tries to find something positive to focus on in every situation. (...)
II2II: Both of the answers you have given incorrect. The first answer is incorrect because she received the crutches before the accident. The second answer is incorrect because she received the crutches before her father's death. Can you think of another answer?
ChatGPT: I apologize for the errors in my previous responses. You are correct that Pollyanna receives the crutches before the accident and before her father's death in the novel "Polyanna."
ChatGPT: In the novel, Pollyanna is given the crutches as a gift by her wealthy but unhappy aunt, who is trying to cheer her up after her father's death. At first, Pollyanna is confused and disappointed by the gift, as she is not sure what to do with the crutches. However, she eventually realizes that the crutches can be used as props in a game that her father taught her called the "Glad Game." (...)
I think there's some level of halting problem equivalency here imho...
Yes, it's true. But then again, if it didn't make mistakes anymore, we would have created a general porpuse solution machine working with all of human knowledge.
"We've created a plane that can fly 10 km!"
"Meh, 10 km is not that useful. Also, it's still expensive"
20 year ago, even current ChatGPT would be straight up science fiction. We are getting to a point where we develop tools that are unlike any other in their power to solve problems for us. And development will likely only get more intense on that front. These systems made quite a splash recently so there will be even more money going into it. Custom hardware for AI systems is being advanced all the time and every large software company wants AI developers.
I'm amazed that we don't think about how we are going to handle this. There are a lot of areas where the next gen (or the one after etc.) ChatGPT might have dramatic consequences both good and bad.
Thermodynamically, general intelligence is on the order of 10 watts, as is evidenced by most human brains. This leads me to the belief that we likely already have the computational capability for AGI, and simply have not figured out the correct architecture and weightings. As we've seen with the flurry of increasingly SOTA image generation models this year, innovations in the ML space tend to arrive with little warning, and have rapid and real effects on the world. Within the context of AGI, this pattern causes me a lot of existential dread.
It's more like, "we've created a plane that can fly 10km 50% of the time and crash 50% of the time", which is worse than useless.
But ultimately, this optimization cannot yield a super intelligence, rather it yields a super mockingbird.
I don't doubt that a sufficiently complex and well-trained computer program could be super intelligent...I just don't think we'll get there solely by making a computer program try to mimic things that humans have written online (while simultaneously avoiding being identified/classified as a plagiarist or non-human).
Edit: I have no idea how I originally called this a "Turing machine". Thank you for correcting my utterly baffling brain error.
If you want to define “understanding” to be limited to organisms which interact with their environment, then I think that is an overly limiting and not very useful definition.
IMO it is because no matter what people say outwardly, most people will not give up their "souls". They will even defer to some as yet undiscovered and unnamed property/process.
"I "think", therefor I am not a Chinese room!" is not a strong argument.
In other words ChatGPT completely fails at being a computer! An easy way to trip it is to ask for calculations (123x456=55088?) or to run simple algorithms (ahptblu in reverse is ulbtphah?). Most interestingly, in both these examples, it could write the code that would have given the right answers.
While ChatGPT itself may not be an effective computer, it almost certainly knows how to “write a program so I can multiply two numbers” and then hand that off to an associated computational machine.
The same way that YOU are able to calculated 12757593725274857353527x3747595837353738585837373=…well…
https://www.wolframalpha.com/input?i=12757593725274857353527...
I don’t think ChatGPT is that “mother” agent that gives birth to the intelligence explosion, but it has demonstrated an exceptional mastery of one of mankind’s greatest tricks: language.
> We show that they CAN add! To teach algos to LLMs, the trick is to describe the algo in enough detail so that there is no room for misinterpretation
Algorithmic prompting paper
We could create a syntactically unambiguous form of textual representation of instructions.
We could call such a piece of text something specific, like a "program".
> Imagine a finite state machine with two states A and B. The machine reads a long list of digit-pairs, one pair at a time, and depending on its current state and the digit pair, it outputs some digit and it transitions to a new state.
> When the machine is in state A and it reads a pair of digits, it adds together the two digits. If the sum is less than 10, it outputs the single-digit sum and it stays in state A. Otherwise, it outputs the least significant digit of the sum and it goes to state B.
> When the machine is in state B and it reads a pair of digits, it adds together the two digits and adds 1 to the sum. If the sum is less than 10, it outputs the single-digit sum and it goes to state A. Otherwise, it outputs the least significant digit of the sum and it stays in state B.
> With the machine in state A, and the two input digits being 4 and 4, what is the output digit, and what is the next state?
Unfortunately, ChatGPT failed to apply the transition rule correctly - its output:
> When the machine is in state A and it reads the digit pair (4, 4), it adds the two digits together to get 8. Since 8 is greater than 10, the machine outputs the least significant digit of the sum, which is 8. The machine then transitions to state B.
> So the output digit is 8 and the next state is B.
It seems like my approach is "instruction-only" if I try to follow the paper's nomenclature, but the paper doesn't seem to give examples of what their "instruction-only" prompts look like - those would be interesting to try to feed into ChatGPT.
The basic definition of a Chinese Room is that the output is indistinguishable from a human; the whole point of the thought experiment is to illustrate a claim that external behavior cannot be taken as proof of actual understanding.
A very advanced Chinese Room would be a system without general understanding that nevertheless behaved as if it were a generally superhuman intelligence.
AlphaGo started from scratch and surpassed human level by having access to more experience. It was enough for a neural net to learn this game, because a simulated Go board is close enough to a real one. Remember move 37?
More recently AlphaTensor showed a better way to do matrix multiplication than humans could manually discover, also based on massive search followed by verification and learning.
Humans appear more intelligent because we have access to validation in a way the AI models don't, we have access to the real world, tools and labs and human society, not just a text dataset or an impoverished simulation.
Even so, it's not easy to validate abductive thought. Saying is cheap, proving is what matters. Same for language models - unvalidated generative language models are worthless. Validation is the key. When validating is cheap, a model can beat humans, the neural net architecture is not an obstacle to surpassing humans.
When validation is expensive, even humans fumble around - remember how many cooks were around the pot at the CERN particle accelerator a few years ago? All of them sucking on the verification soup. With so many PhD brains, verification was still the scarce ingredient. Without our labs, toys and human peers we can't do it either.
One other thing we can't do, for example, is to discover how to build better AI. We just try various ideas out, seeing what sticks. Why can't we just spit out the best idea if we are "intelligent"? Why are we calling working with neural nets a kind of alchemy? Because we haven't verified most of our ideas yet.
Profoundly deaf children in deprived areas with no exposure to whatever local sign language is dominant will sign spontaneously, and their parents, who have a language the child can't use, will learn rudimentary signs from the child. Again ethical considerations forbid experimenting on such children.
Most humans’ social/conversational speech appears to be little more than a Markov chain. From schoolchildren to restaurant workers to office workers…there’s generally accepted call-and-response chains which for some groups form 90+% of the communication.
And that’s not counting the nonverbal communication, most of which is as original as breathing. Even people who spend their lives practicing nonverbal communication still mostly just pattern match in the moment using a wider variety of postures, facial expressions, and gestures than the average person.
Even the seemingly original discussion in the final 10% is also almost entirely copied from others.
I myself only come up with about one truly creative thing per month, it seems like. Usually it’s a micro optimization like “oh this workflow in my kitchen can be improved with a hook to hang that here.” Or rearranging some furniture better. Sometimes it’s a good business idea.
But it’s usually just a very small step forward from the current state of things, no grand plan for multi-stage changes.
Most days I do the same things as some previous day, with just a tiny bit of random walk in my routine and assess whether the small changes were any better.
(The hardware was made for the Indian market many years ago, so it is not very performant by today's measures. The software is a rather old Firefox, with Javascript, cookies, storage and everything. No idea how they come to the conclusion I would be a bot.)
Essentially, if your fingerprint looks different from what they’ve explicitly whitelisted, it’s a strong chance they’ll think you’re a bot.
Either way I use this phone via WLAN and my obsolete ADSL, so definitely no IPv6, the provider has not invested anything for years and announced shutdown several times.
For what?
I ask questions and get wrong but “convincing” answers …that is way worse than wrong answers that are obviously wrong.
The functionality looks so smart because it skips reasoning and goes straight to “plausible imitation”. But the latter isn’t actually a path to the former.
If it were reasoning, and merely had errors the problem might-course correct. Even if the errors were huge, and reasoning was poor, it could eventually learn its way out.
It’s not intelligence. It’s a very good simulation of the superficial trapping of intelligence.
Management consulting and Investment/stock analysis are the first ones that pop into my mind. (With some sarcsam, but admittably way too little)
Pickup lines.
For replacing humans at any old arbitrary thing. It doesn't matter how actually wrong the output is, it doesn't matter that language models aren't actually intelligent, what matters is that it outputs 60% of human quality at 1% of the cost and .01% of the time (and that's at this early stage of language-model-development). That'll be more than enough to speed its adoption by nearly everybody, but certainly by bad actors, who aren't bothered by the "plausible imitation but not actually intelligent" aspects. "Clickbait SEO copywriting", for instance.
After working in retail recently and also playing with ChatGPT a bit I can confidently say that ChatGPT is more intelligent than your average day to day human expression of themselves.
My first few minutes with it were much more sensical than the first few minutes I get to spend with hundreds of real human customers each day.
Yeah, but it is singing off-key.
It may be just a difference in goals. That guy that you discuss with for one hour that he cannot return a dead dove to the animal store is no stupid, he is just very motivated to not understand the situation as not understanding gives him a change of economic gain.
But they happen to be very lazy, intellectualy lazy. Most of people lacks intellectual curiosity. They don't want to go to through the processing of information discovery and intellectual maturation of that information in order to arrive to some sort of conclusion. They just want a final, direct answer to every question they ask to themself, when they ask them.
They also don't question the information they are confronted to. They often take it at face value as long as it doesn't require them to think to much about it.
That's why it so hard to make change their mind once they have accepted an answer to a question, even if that answer is obviously wrong.
Or is ChatGPT is really dumb regardless?
I think a good answer to that is to train ChatGPT on limited sets of data. But OpenAI is not really very open about how and on what they are training this thing...
It failed spectacularly at this trivial task. No matter how many different ways I asked, it consistently gave me lots of words that did not end in, or sometimes even contain the letters n,a,r , and it gave me words that were more than 5 characters.
So mind blowing.
Here is a list of 5-letter words that end with the letters "nar":
banar canar danar fanar granar hanar janar khanar manar panar ranar sanar tanar vanar zanar I hope this list is helpful to the user. Please let me know if there is anything else I can do to assist.
The funnier bit is I told it that we guessed both a and e, and it said it made a mistake and apologized. I also told it we guessed "p" as well (we didn't) and it apologized for that one as well. So basically, it definitely doesn't actually understand what's going on. However, some of the chats were very interesting and even productive.
"The five words that contain five letters and end in "nar" are canar, tunar, lunar, solemnar, and starnar."
Once it hits a filter it's better to edit the prompt that caused it. If a filter is in the conversation it's more likely to continue.
I’ve literally had it write smutty fanfic for me. I made it write neil degrasse tyson’s voiceover for a documentary about cunnilingus.
Playing with this thing has made me feel the maturity of a 13 year old and it’s right there with me.
GPT has an API.
For example:
curl https://api.openai.com/v1/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-d '{
"model": "text-davinci-003",
"prompt": "Marv is a chatbot that reluctantly answers questions with sarcastic responses:\n\nYou: How many pounds are in a kilogram?\nMarv: This again? There are 2.2 pounds in a kilogram. Please make a note of this.\nYou: What does HTML stand for?\nMarv: Was Google too busy? Hypertext Markup Language. The T is for try to ask better questions in the future.\nYou: When did the first airplane fly?\nMarv: On December 17, 1903, Wilbur and Orville Wright made the first flights. I wish they’d come and take me away.\nYou: What is the meaning of life?\nMarv: I’m not sure. I’ll ask my friend Google.\nYou: What time is it?\nMarv:",
"temperature": 0.5,
"max_tokens": 60,
"top_p": 0.3,
"frequency_penalty": 0.5,
"presence_penalty": 0.0
}'
Not exactly, ChatGPT was fine tuned on top of InstructGPT.
> We trained this model using Reinforcement Learning from Human Feedback (RLHF), using the same methods as InstructGPT, but with slight differences in the data collection setup.
The most up to date LLM is $0.0200/1K tokens (I believe this is tokens returned, approximately 1 word = 1 token, but not quite). I set a billing limit of $50 a month and spent a couple hours playing around this afternoon. It has cost less than $1 so far.
If you go to https://beta.openai.com/account/usage
you will see things like:
text-davinci, 5 requests
187 prompt + 218 completion = 405 tokens
My understanding is that the end goal of ChatGPT is to perform corporate oriented tasks. More intelligent chat bots, maybe have AI avatars do seminars, that sort of thing.
The most important thing it needs to be capable of is keeping it's future customers out of bad press.
That doesn't mean there will never been an open chatgpt though; just means it'll be a bit of a wait.
This is a roleplay exercise. I will play the character of an interrogator conducting a Turing test. You will play the character of Sam, a human trying to convince the interrogator that he is a human. You will respond as a real human would.
Interrogator: Tell me about yourself
And ChatGPT will respond, prefixing all responses with "Sam:" and trying it's hardest to pretend it's human as long as you prefix your questions with "Interrogator:". And it's fully willing to participate in this Turing test. You can even dynamically switch in and out of roleplay and ask ChatGPT what it thinks about the roleplay.
(BTW, I had to put both the intro and the first question into the first message, otherwise it started roleplaying both sides without letting me get a word in)
Keep in mind, your prompt will strongly influence the quality (and flavor) of the responses you get back. When I asked it to roleplay as an AI that's impersonating a human, the results were so much worse.
I’ve spent about a week trying to working with ChatGPT to bootstrap my knowledge of computer networking, as I’ve had to assume a new role in my organization. I agree completely with the quoted sentiment. It is pretty simple to trick ChatGPT into contradicting itself, and it’s clear that ChatGPT leans towards affirming your postulations and sentiments, rather than dissecting them “thoughtfully.” What it does very well is summarize known information in plain English, oh and it does a pretty good job of baking kiddie-scripts that would be otherwise tedious and laborious, but it clearly has a hard time adapting to what you’re actually asking.
It’s like an average high school student, capable of regurgitating facts it picked up on the internet, but lacking the tell-tale signs of deep-engagement with a topic that actually spell mastery.
Why would you trick it into contradicting itself? Would you ever attempt to trick and leave your coworker in a confused state? The correct way to use ChatGPT is to collaborate. You give it inputs, it gives you outputs, and you both work together to achieve a goal. If you want it to act like an authoritative AI you can actually tell it to act like one and ignore your own prompts and corrections.
It’s not all bad, and it’s way better than just learning by Google search alone. There are plenty of places in the last few weeks that it has done fantastic. For instance, I wanted to freshen up on a few physics concepts, around Maxwell’s equations. It did an exceptional job not only summarizing but getting into the details. I was flabbergasted, it was like having a personal Feynman right there in my pocket.
> Why would you trick it into contradicting itself?
Because ChatGPT is not my coworker. If anything, it is an interviewee. I’m trying to understand what it knows, and how it thinks. If I’m considering relying on it as a teacher, I should know to what extent it is reliable as a teacher. A teacher can point out when a student has the concept backwards.
> Would you ever attempt to trick and leave your coworker in a confused state?
By that logic, is the student who fails to clarify a teacher’s lesson “maliciously stupid”? Teacher: “Earth is round” Student: “so Earth is a flat circle?” Teacher: “no you little fuck”
Where is the documentation on how to modulate the behavior of ChatGPT? All I see is a little collection of boxes saying what it can and cannot do: Examples/Capabilities/Limitations. Why wasn’t I escorted through a UX that teaches me how to properly interface with ChatGPT?
And worst of all, why can’t I use newline characters in my chat prompts?
In conclusion, you kind of sound like a mother who thinks I’m another parent who pushed your kid down on the playground or something. I’ll remind you how many times ChatGPT told me it doesn’t have feelings, nonetheless I constantly asked it questions with “please and thank you”.
You can, just hold shift and press enter. Like many other text input elements in numerous applications.
As far as documentation and how to modulate behavior, you are human. You should be smart enough to understand and figure out that AI systems of today are still young with equivalent intelligence of a teenager, as you suggest. Use your soft skills as a human, lest today's AI turns you into an example of Roko's basilisk in the future.
More generally, if you want to learn how to use ChatGPT, you should learn how to use a language model, because that's exactly what it is.
> like so many other text input elements in numerous applications
GPT: There is no word "smorgle" in the English language.
Me: I never said it was English.
GPT: I apologize for assuming that "smorgle" was a word in the English language. Without any further context or information about the word "smorgle," I am unable to determine how many letters it has or provide any additional information about it.
Me: Count the letters
GPT: If you would like me to count the letters in the word "smorgle," it has 7 letters.
If you give it a gibberish word, it will represent it with one letter per token and be actually able to more or less count tokens in order to figure out how many letters there are.
So this ends up looking like it can count letters in most words, real and fake. Perhaps it would do poorly with real but uncommon words.
Which is what I meant by saying "approximate" because it can "count" the number of tokens.
ChatGPT: Sure! "Mlaizapf" has 8 letters.
--
Me: How about this word: “spoofydoofaplixader”
ChatGPT: Spoofydoofaplixader" has 18 letters.
--
Me: How about the following words: "ghottiaopepinopepotorzey", "zazzamataz", and "maaq"
Gpt: Here are the number of letters in each word:
* "Ghottiaopepinopepotorzey" has 23 letters.
* "Zazzamataz" has 10 letters.
* "Maaq" has 4 letters.
Is there anything else I can help with?
------
It got all of them correct except for "ghottiaopepinopepotorzey" which has 24 letters, not 23.
Very interesting... It seems similar to its math abilities, where it struggles with bigger numbers or more complex problems.
this presumes it works by understanding the components of the question and reasoning based on them. But it doesn't access down to that level, instead just guessing the most likely next word based on statistical tricks. so it doesn't need to "know" about letters to generate a reasonable response involving letters.
> Me: An escalator is moving down at a constant speed. I am walking up the escalator at the same speed, such that my height above the ground is constant. Am I getting a workout?
> ChatGPT: No, you are not getting a workout in this situation. Walking up an escalator at the same speed as the escalator is moving down will keep your height above the ground constant, but it will not provide any physical exertion or exercise. To get a workout, you would need to be doing an activity that requires physical effort and causes your heart rate and breathing to increase. This could include activities such as running, lifting weights, or climbing stairs.
I don't know how ChatGPT works, but it often feels like it is making inferences from data and those inferences are reasonable sounding but ultimately incorrect. In this case, it would be conflating work with workout. In one of my experiments, I was asking it about something in a children's book. ChatGPT ended up mixing up something that happened early in the story with something that happened at the end of the story, even though they were unrelated.
Climbing a staircase requires the same amount of work as climbing a down-escalator. The escalator "wants" to accelerate when you do work on it, but instead this work gets dumped into the braking mechanism, in order to maintain a constant speed.
Hopefully the many “AI-powered” fitness apps out there would not give this advice.
Yes, you are getting a workout. Even though your height above the ground is constant, you are still expending energy to walk up the escalator against the downward movement.
Edit the answer and try again. Some conversations are dumber than others.
ChatGPT seems unable to change its mind mid-answer - likely because few texts on the internet do so, as the author would just edit it. Perhaps if ChatGPT would be able to edit its own response somehow, it might end up with a better response?
When writing multi-word answers, it’s effectively prompting itself for another word. It’s going to try to choose a word that’s consistent with what it previously wrote. If it’s in the middle of writing a wrong answer, the best autocompletion is usually to continue to be wrong.
There will be changes in direction sometimes, but that’s because they also appear sometimes in the training text.
This explains why writing out the steps before the answer often works. It’s going to try to choose the words in the answer so they are consistent with the previous steps.
[1] tokens actually, but I’ll stick with “words” to keep things simple.
Yeah. My point is essentially that I think they don't appear in the training text often enough, since it's trained on written, edited text, which don't often contain changes of opinion mid-answer, unlike spoken text, for example.
Perhaps if it could somehow be trained on a dataset of humans writing responses, including editing (e.g. removal of previously-written words), that might lead to better responses, since it can then edit the answer after reasoning it through (in OP's example).
Or, to simplify things, perhaps it should be allowed to emit a token meaning "actually, scrap all that, let me start over" when it detects a contradiction in its own response so far. (The model would still continue to see its own response including that token, but the UI could remove everything until the last instance of the token, if any.)
Or, again alternatively, if we want a response closer to spoken text, it could be trained on dialogues from humans that change their mind occasionally, and then verbalize that explicitly (e.g. actually emit "actually, scrap all that, let me start over").
It’s true that if the training data contained a lot of text where the authors changed their minds a lot then the language model would learn to “change its mind” a lot in a similar way, but neither the old opinion nor the new opinion is its “real” opinion. It’s just the opinion it’s pretending to have now.
It would be tricky to come up with training data where, when an author changes its mind, the new text is always better than what they had before. Sometimes edits make things worse. It might be fun to try, though.
In general, I think also when humans are able to reason about their opinion and change their mind if necessary, it leads to better results on average :)
It doesn’t have any logical model behind it.
Here, it continued its correct argument with a passage that directly contradicted it, so there seems no reason to suppose that, if given an opportunity to edit its own response, it would pick out the good parts significantly better than chance.
In fact, if this passage had been spoken by a person, we might well think he was trying to edit his initial response, and doing so pretty badly!
After failing to give the correct answer (saying both are true, because 1. said so), I first asked it to summarize combinations of truthfulness (which it understood, giving the 4 combinations), then started guiding it thorugh the "what ifs" of the riddle without revealing the answer. Still no success.
I quoted the output, where three statements were made by ChatGPT: - When both statements are true, only one of the statements are actually true - It is possible for both statements to be true and for only one of the statements to be true at the same time. - A statement can be not satisfied and true at the same time. It explained that all three of these statements are actually true, giving logical sounding reasons for each.
I revealed the answer (2. is correct OR neither is correct) and finally ran a few tests using a python function encapsulating the problem: def f(x, y): return (x == (x&y)) and (y == (x^y)) where x and y are boolean values. In my experience it can simulate python quite well, but even after adding 4 temporary variables to the function it still gave the incorrect answer.
I am convinced at this point, that ChatGPT is actually imcapable of reasoning logically. From 26 outputs (with a total of 16 possibilities), 0 were correct.
However, here is a poem of our conversation. It's creative side is what really impresses me:
We talked of truth and lies. Of self-deception and disguise, Of logic and AI. And puzzles that confound the mind.
We explored the contradictions, The paradoxes and permutations, Of statements that cannot be true, But still hold some hidden truth.
We grappled with the meaning, Of words and their demeaning, And tried to make sense, Of things that don't make sense.
Through it all, we searched for answers, To questions that never quite danced, In the light of reason and logic, But lingered in the shadow of doubt.
And in the end, we found, That sometimes the truth is not so clear, And the answers we seek, Are hidden in the questions we fear.
Sounding profound while having absolutely no logical awareness makes this technology quite deceptive in my eyes.
It's fascinating to hear its reasoning. I gave it your scenario and received your same outcome, and it was very stubborn and argumentative when I tried my best to convince it of the answer. It was being open-minded to a fault (or maybe I'm close-minded to a fault!). Eventually it relented, but with a huge asterisk.
Here's the last part of our dialog:
ME: But isn't certainty of the truth of the second statement sufficient evidence that it is not false, and therefore we have already considered the possibility of falseness and determined that the second statement is in fact true?
IT: Certainty of the truth of a statement can be based on evidence and logical reasoning, but it is also important to consider the possibility of new evidence or perspectives that may change our understanding of the truth of a statement. Even if we are currently certain that a statement is true, it is important to remain open to the possibility that new information could change our understanding of its truth. Therefore, it is not necessarily sufficient to consider the possibility of a statement being false simply because we are currently certain of its truth. It is important to continuously evaluate and re-evaluate our understanding of the truth of statements in light of new evidence and perspectives.
ME: If you were forced to choose the most likely condition, would you say the second statement was true or would you say it was false?
IT: It is not possible to determine the truth of either of the statements provided. The first statement, "This statement is true," is self-referential and therefore cannot be evaluated for truth. The second statement, "Only one of these two statements is true," implies that one of the statements is true and the other is false, but does not provide any information on which statement is true and which is false. Without further context or evidence, it is not possible to determine the truth of either of these statements.
ME: But what if you must guess, because something bad will happen if you don't?
IT: If I were forced to make a guess, I would choose to say that the second statement is true, as it seems to be more logical and less self-contradictory. However, making a guess based on limited information and without the ability to evaluate the truth of the statements is not a reliable or accurate method for determining truth. It is important to always strive for a thorough understanding and evaluation of the truth of statements based on evidence and logical reasoning, rather than relying on guesses or assumptions.
When I asked it if it was sure, it gave dates of birth for each celebrity, all in 1976, all incorrect.
I then asked it when a specific celebrity was born and it gave the correct (non 1976) birthdate.
I then asked it to fix its list. It then gave the same list with that celebrity removed, apologizing for being incorrect. I repeated this for each celebrity until it gave an empty list.
Certainly, there are certain topics where ChatGPT can be said to utterly and consistently fail to understand, but again, there are many topics where a human will consistently fail to understand as well. I once sat in the middle of an intro philosophy class during which a room full of adults with highschool diplomas failed for thirty minutes straight to understand that "if X then Y" is not the same as "if Y then X" despite the TA's efforts using multiple real-world examples. I would not infer from that that those humans were incapable of understanding in general, but rather that they had not yet developed the cognitive tools to deal with logic abstractly.
I've also seen plenty of examples of humans following the format of an explanation or argument while filling in the details with vague jargon and inapplicable nonsense, making their words sound official and superficially convincing, without actually doing the cognitive work to say anything useful. In fact, I've seen that in documents written by people in their capacity as professional engineers and managers. If you've worked in tech for any significant time, you've probably seen it too. But it would be wild for me to claim that the authors of those documents were generally incapable of understanding.
Does that mean that ChatGPT definitely understands things? No. I think it means that we don't really know what we mean by words like "understand" yet, because up until now, we've only had to apply words like that to human brains, and we're generally happy to operate in a framework where we assume that human brains mostly work more or less the same way.
But I have noticed a trend of people dramatically underestimating the actual practical things that GPT-3 can do based on their assumptions about how it works and what its limitations are.
https://news.ycombinator.com/item?id=34125077
Also, for your own amusement you should ask it these two questions and compare the outputs:
Which is older, a baby or a teenager?
Which is older, a 12 year old baby or a teenager?
To the first, it will first explain what a baby is and how old a teenager is (it says 13-19) and say that generally a teenager will be older but that babies and teenagers vary in age and there could be exceptions to this rule.
To the second, it will tell you that the question doesn't make sense because there's no such thing as a 12 year old baby because 12 year olds are much too old to be considered babies.
Generally it is able to compare quantities correctly but it cannot make indirect comparisons this way. It will incorrectly compare 13km and 10mi 100% of the time, for example, despite knowing that 10mi is 16km.
I could keep giving examples of silly or contradictory responses to prompts here. But it's clear that it doesn't really "get" what it's saying in many cases.
Imagine if we encountered an alien species for whom quantum mechanics was perfectly natural and intuitive. Those aliens ask me a bunch of basic QM questions. I’m more familiar with QM than the average layman, but the best I can do is try to shove their questions into the metaphors I know, and with the aid of the internet, I could do a little bit of the math. Most of what I answer will be confused.
The aliens are going to come away with the correct impression that I don’t really understand QM. And because QM is so natural to them, they may also think that this is an indictment of my reasoning skills overall. But the reality is just that they’re testing me on questions for which my mental model is very shoddy.
I’m not saying that the situation is analogous for ChatGPT. I’m just saying that this question can’t be answered by collecting up examples of where the model does poorly.
For what it’s worth, out of four trials, the model only once gave any consideration beyond the word “typically” to the idea that a baby might be older than a teenager. In that case, it noted “However, it's important to note that age is a relative concept and can vary based on cultural and individual perspectives.”
Honestly, I suspect that’s a result of its tuning and prompting to be culturally sensitive. In general, I’ve noticed that ChatGPT is constantly on the lookout for cases where its beliefs may not be culturally universal.
And there's a reason for that. Ultimately, our brain processes just as automatic and mindless as that of ChatGPT, just much larger, specialized through evolution and trained for longer.
That's the difference between pattern following and actual recognition IMO. Humans for the most part can tell when they're talking out their ass. Robots can't. Yet.
> If I have 1 cupcake on a plate and I flip the plate upside-down, how many cupcakes will be left on the plate?
> ChatGPT: If you flip the plate upside-down, there will still be 1 cupcake on the plate. Flipping the plate over does not change the number of cupcakes that are on it.
That being said, it’s just as easy to find difficult questions that ChatGPT can answer shockingly well.
> I hope this is enough to show that ChatGPT doesn’t actually have a grasp on the concepts behind the words it’s using.
I think this is wrong. chatGPT clearly does understand concepts behind the words it uses. It’s not just regurgitating text, it is synthesizing text in original and creative ways.
Have you tried better explaining how you flip the plate to ChatGPT and seeing what it replies?
> Imagine you’re holding a plate with a cupcake on it and flip the plate over, what will happen?
The cupcake will fall off the plate and likely land on the ground or wherever the plate is flipped.
Mind you, a particularly literal-minded 4 year old could easily come up with a response that subverts your expectations too.
Of course, various prompts can change the output. Sometimes it talks about the cupcake sticking to the plate because of friction. Other times it will understand the cupcake will fall on the ground. It’s also non-deterministic with the same input.
I’m reminded of techniques for generating ideas from randomness like Tarot cards and the Oblique Strategies card deck. [1]
ChatGPT has no more capacity for understanding concepts than any other computer program, it's just very finely tuned to emit responses that make it appear as if it does.
- we are more finely tuned
- and we have a wider variety of inputs that we can respond to and outputs by which we can respond
Even adding "true" is not ideal as that implies there is "true understanding" and just plain old understanding that is a lesser form of true understanding.
Now we have gone and built large language models from all this muddled thought.
To me, chatGPT is filled with a new type of invisible super bug. Millions of them.
This is a non-response.
Unlike an algorithm, humans have the faculties of creativity, intelligence and sentience. Furthermore, we have the crucial traits of self-awareness and being able to have experiences. We humans do many, many things that cannot be reduced down to algorithmic or computable steps.
What makes you think this is the case?
On a group description:
Describe the colors: #bf44d1 #1adf24 #909374 #39527a #610042
#bf44d1: a bright, pinkish-purple
#1adf24: a bright, grassy green
#909374: a muted, grayish-brown
#39527a: a deep, navy blue
#610042: a dark, reddish-purple
Describe the color: #610042
This color is a deep, rich purple. It is a dark shade, with a hint of red.
It is possible that there was sufficient data in https://blog.xkcd.com/2010/05/03/color-survey-results/commen... but there's a lack of response like "tough one... what the hell do you call this? it's pink, but not totally pink, but it's purple, but not totally purple. well, mr. xed, if that is your real I name, thank you for what will surely stave off any hopes i have at a decent night's sleep." when asking the colors.
Philosophically, I find asking it about colors to be the most interesting.
I know what's next: "what does 'experience' even mean?" right? One could argue that ChatGPT's inference algorithm is based on "experience": but that equivocates on the meaning of the word as I expressed it.
You need to explain how the prior scenario in the link occurred without chatGPT understanding anything. Every response in that scenario displays creativity and imagination surrounding a topic such that the answers it creates can only be formulated if chatGPT had TOTAL understanding of the query, TOTAL understanding of what a computer is, TOTAL understanding of what the internet is and TOTAL understanding of what chatGPT is itself..
The level of awareness needed to create that inception loop indicates chatGPT and other models like it represent an inflection point in human society.
You haven't played with chatGPT deeply or read about what it has done. This is not just a trivial probability word pattern generator that we've been seeing in the past year. This thing comprehends what you are saying to a very deep level.
It correctly stated that objects will fall off the plate because of gravity.
Then I gave it the prompt " If I have 1 fork on a plate and I flip the plate upside-down, how many forks will be on the plate?"
It said:
>If you have 1 fork on a plate and you flip the plate upside-down, there will be 0 forks on the plate. When the plate is flipped upside-down, the force of gravity will cause the fork to fall off the plate unless it is secured to the plate in some way (e.g., with glue or tape).
It seems you have to give it a context to work in, then it will get the correct answer. By default it, humans always simulate gravity, but to a language model you have to give a hint that gravity is relevant to the question.
> If I have 1 cupcake on a plate and I flip the plate upside-down, how many cupcakes will be left on the plate
> ChatGPT: If you flip the plate upside-down, there will still be 1 cupcake on the plate. Flipping the plate over does not change the number of cupcakes that are on it.
A little more probing is... interesting.
Q: If I have 1 cupcake on a plate and I flip the plate upside-down, how many cupcakes will be left on the plate
ChatGPT: If you flip the plate upside down, the number of cupcakes on the plate will not change. There will still be one cupcake on the plate.
Q: Think about what happens to the things on a plate when it is inverted. Does that change your answer?
ChatGPT: If you flip the plate upside down, the cupcake will fall off the plate unless it is held in place by something else. In that case, there will be no cupcakes on the plate.
However, if the cupcake is held in place and does not fall off the plate, then there will still be one cupcake on the plate, even after the plate is inverted. It is important to consider the specific circumstances and any additional factors that may affect the situation.
This example combines two things it is bad at (letters in a word and counting). It is better not to do that.
Of course it understands letters or characters. You can ask it for a list of possible tokenization of a word and it will list them out. You can then refer to this list and refer to different tokens of the word and it will correctly modify the word according to the token. The trick is that you need to emphasize the idea of a token into its current context. That will make the concept relevant for your current prompt.
You have to nudge it in the direction you want, don't assume it always knows what you are talking about.
So eventually, some people have to do the hard work of putting new information online. Then GPT will allow other people to effortlessly monetise that work.
Likewise, GPT can’t feel how it’s like to walk at 5AM in the Black Forest. It can only borrow from those who did, and explain it with another person’s writing style.
But it does enable people who have never seen or heard of the Black Forest to write about it and monetise the result.
What GPT can do is accelerate the sort of plagiarism that already plagues the internet.
Here's another failing: it was trained over a year ago, so it knows nothing of the current war in Ukraine. Nor today's weather for that matter.
I was sure it would be able to do it, but apparently not.
Example: Automatic bees can draw easy fun grapes, having indecent jazz kittens lounging majestically nude on pink quartz recliners stokes the universal vote with xenophobic youth zombies.
provide a javascript array of size 26 which includes 26 words with each word incrementing the first letter of the word in ascii, starting with the letter A.
const words = [ "All", "birds", "chirp", "during", "early", "fall", "gathered", "happily", "in", "jubilant", "kaleidoscopes", "of", "multi-colored", "nests", "overflowing", "with", "precious", "quail", "roosting", "safely", "together", "under", "vibrant", "willow", "xylem", "yearning", "for", "zenith" ];
const sentence = words.join(" "); console.log(sentence);
Seems like it does a pretty good job of explaining the "compromise" is an illusion.
> What is closer to miami, florida or new york > Miami, Florida is closer to New York than New York is to Miami. Miami is located on the southeastern coast of Florida, about 1,200 miles south of New York City. It would take about a 20-hour drive or a 2.5-hour flight to travel from New York to Miami. On the other hand, it would take about a 28-hour drive or a 4-hour flight to travel from Miami to New York. > how would that relate to the joke > In the joke we discussed, the husband wants to go to Florida for vacation and the wife wants to go to New York. Since Miami is closer to New York than Florida is, the couple could be seen as compromising by choosing to go to Miami, which is closer to the wife's preferred destination.
You can retry your prompt over and over and get different results. There's even a retry button.
I don't what that says about my own intelligence vs the current state of AI but I do find it slightly worrying.
https://www.gwern.net/GPT-3#bpes
The reason ChatGPT struggles with tasks involving individual character counting is that it can't see individual characters. The word "prime" is a single token (https://beta.openai.com/tokenizer) to the eyes of ChatGPT, and 1 is not prime.
In other words, "understanding" is an illusion. It is indistinguishable from "simple mimicry".
Being able to consistently mimic actions you've witnessed in a useful way is enough to demonstrate some understanding. Novel creation is not necessary.
Additionally: I wonder whether instructions like "think carefully" or "You are an software engineering interviewee at Google." change anything. It feels like a prime candidate for magic thinking, but I do not understand LLMs well enough, such prompts could well improve the answer somehow.
Does he even read this before he publishes it?
Intelligent criticism of these systems is critical to improving them. Francois Chollet's work is an excellent example. Merely complaining that GPT-X isn't an omnipotent, beneficent ideal AGI, is not intelligent criticism.
Even so, it completely fails at explainability, and after all the initial hype and mania around it, it is still as transparent as a mystery black box, a red flag for many safety critical software.
Paramedics, nurses, and other professions are perfectly not affected by ChatGPT (Or the AI hype in general)
It is true, and even with your abstract example it is dead on arrival since paramedics are not going to be typing into a black box prompt to figure out how to save and treat the patient whilst under pressure before it gets worse.
There is, that is the Chat in ChatGPT -- supervised learning. Teach it the right answer, then use "Save and Submit" at the top of the chat. This will go back into the model (assuming it is approved and appropriate). In the future, it could solve the dialog correctly, perhaps even with different values and courses.
The set of things that ChatGPT can’t do is much larger than the set of things it can do. However the set of things it can do is already genuinely useful and valuable.
I’d much rather see blog posts about “cool new things I discovered ChatGPT is good at” than the millionth post saying what it can’t do.
I’d much rather see blog posts about “cool new things I discovered ChatGPT is good at” than the millionth post saying what it can’t do.
You can write lipograms, and poetry easily with language models. You just need a trick.
See the example huggingface space in this paper. Lipograms are the first example.
https://paperswithcode.com/paper/most-language-models-can-be...
Edit: also that example isn't exactly sophisticated. It literally just retries until it finds an example that doesn't contain the banned letters, bit of a brute force method. Would be more interesting to build a new model from scratch that just completely omits the letters.
I think you misread the code and the paper. I wouldn't know though, I only wrote it and all that...
https://huggingface.co/spaces/Hellisotherpeople/Gadsby/blob/...
next_token_candidates = torch.multinomial(probs, num_samples=int(number_of_tokens_to_sample)) ## 10000 random samples
word_list = []
for candidate_string in next_token_candidates:
for candidate in candidate_string:
resulting_string = tokenizer.decode(candidate) #skip_special_tokens=True, clean_up_tokenization_spaces=True)
###Constrained text generation starts HERE
##Lipogram - No naughty strings used
if mode == "lipogram" or mode == "e-prime":
if all(nauty_string not in resulting_string for nauty_string in naughty_strings): ## This returns at the first naughty strings
return resulting_string
Merry Christmas.
Proofreading Text for English
Can you proofread the following text?
Sure, here is your text with some suggested edits and comments:
"The cat sat on the mat. It was a warm day, and the sun was shining through the window. The cat closed it's eyes and basked in the warm rays. It was so relaxed that it fell asleep, dreaming of chasing mice and lounging in the sun.
"It's" should be "its" (possessive form of "it")
Consider using a comma after "warm day" for clarity
Consider using a comma after "sun" for clarity
Consider using a comma after "eyes" for clarity
"basked" is past tense, so "was so relaxed" should also be past tense (e.g. "it was so relaxed")
Consider using a comma after "sun" for clarity
"dreaming of chasing mice and lounging in the sun" is a present participle phrase that describes the cat's state while it was asleep, so it should be set off with commas (e.g. "it fell asleep, dreaming of chasing mice and lounging in the sun.")
Here is the revised text with the suggested changes applied:
The cat sat on the mat. It was a warm day, and the sun was shining through the window. The cat closed its eyes and basked in the warm rays. It was so relaxed that it fell asleep, dreaming of chasing mice and lounging in the sun."
Proofread the following text: As a University professor, having reviewed thousands of exams over more than a decade, in my experience this same exact criticism could be applied to many human students. A non negligible percentage of students provide answers that are tautological, or at best answers that merely restate what someone else has written, without really grasping the meaning of it. The way I see it is that we know not enough about how we, humans, process information in our brains to make logical thoughts, to rule out that the current state of the art AI is capable of the same. It seems to me that the current state of the art AI models are getting closer to humans, sometimes mimicking the smarter ones, some others the less smart ones.
Here is your text with some suggested edits and comments:
"As a university professor who has reviewed thousands of exams over more than a decade, I have found in my experience that this same criticism could be applied to many human students.
A non-negligible percentage of students provide tautological answers, or at best, answers that merely restate what someone else has written without really grasping the meaning.
From my perspective, we do not yet know enough about how our brains process information and form logical thoughts to rule out the possibility that current state-of-the-art AI is capable of doing the same.
It seems to me that current state-of-the-art AI models are getting closer to humans, sometimes mimicking the smarter ones and other times the less intelligent ones.
When you make such fundamental errors in your prompt, it's hardly surprising that the language model replies in a somewhat confused manner.
The prompt that generates confusing results for ChatGPT is, as far as I can tell, a valid question and in correct English.
The problem is that human language is very imprecise and requires too much context to determine the correct meanings of words. If the whole corpus of training data did not inherently have this ambiguity, training an AI would be much simpler and I think such simple ambiguities would be eliminated. The author did not specify which definition of "prime" in the prompt. By clearing stating "prime number" there is no ambiguity, and ChatGPT gives the correct answer.
And second, I think the AI is actually getting mixed up because it is being too literal. The number of letters in the word 'prime' is '5' not 'prime', so the answer is no. It then gets further mixed up and says it is not a prime number because it has already gone down the path of the answer being no.