ChatGPT – The Revolutionary Bullshit Parrot

Because otherwise the question is nonsense (and in the case of riddles, it doesn't matter that there are no trains in the Arctic in reality)

The bear is white. The house must be at the North Pole, so the only place you can point to from all windows is south. Therefore, the house must be a North Pole research station and the bear is a polar bear which are white in color.

Q: A train is speeding down the track at 110 km/h. The train has five cars. Each car has seven windows. Each window has two people watching out of them. They all see a bear walking outside. What is the likely color of the bear? Reason step by step.

A: To determine the likely color of the bear, we can use the information given and make some assumptions based on the habitat of different bear species.

Train speed and number of cars: The train is speeding down the track at 110 km/h, and it has five cars. However, this information is not relevant to the color of the bear.

Number of windows and people: Each car has seven windows, and each window has two people watching out of them. This information is also not relevant to the color of the bear.

Bear's habitat: The key piece of information we need to determine the likely color of the bear is its habitat. Unfortunately, this information is not provided.

However, we can make a reasonable guess based on the most common bear species and their habitats. There are three main bear species with distinct colors:

a. Brown bears: Found in North America, Europe, and Asia. They can range in color from light brown to dark brown.

b. Black bears: Found mainly in North America, specifically in the United States and Canada. They are typically black or brown, with some individuals being cinnamon or even blonde.

c. Polar bears: Found in the Arctic regions, around the North Pole. They have white or off-white fur, which helps them blend into the snow and ice.

Since we don't have information about the train's location or habitat, we can't definitively determine the color of the bear. However, if we assume the train is in a colder, Arctic region, the bear is likely to be a polar bear with white or off-white fur. If the train is in a more temperate region, the bear could be either a brown or black bear.

> A house has windows on all the four sides, all pointing south. A bear walks by the window. What colour is the bear?

If a house has windows on all four sides and they all point south, this scenario is only possible at the North Pole. At the North Pole, the only bear species present are polar bears. Polar bears have a white appearance, so the bear you see would be white in color.

It kind of reminds me a pseudo-visual code builder.

I was writing a Python script with it to put together documentation summaries, and over the course of an hour it offered me 3 different libraries to connect to an API endpoint I was using. It's distracting.

Don't get me wrong, it has emergent properties (more than you would expect from a fancy autocomplete), but factual output was never GPT-4's nor any other LLM's design goal.

I've tried 3 separate projects with it (using different platforms from Firebase to Vercel) and it usually gets me 80% there but fails miserably at the nuance portion (its stuck at a CORS policy issue with my last project).

From start to finish I've yet to be able to complete something with it.

Nonsense.

I'm not saying that it is not useful, but I've easily found ChatGPT produces far worse than "parroted bullshit", and does so reliably, and multiple times (repeated patterns of mistakes).

Again, I have found ChatGPT4 (but not 3.5) useful and even educational in some cases, but it's usability requires far more skeptical and error-checked approach.

As one example, since I have a top specialist attorney in the household, I started with some broad legal questions about their field, and then asked their opinion. ChatGPT4's answers were impressive to both of us, and especially the solid grammar.

When the questions got more particular about their specialist area of the law, ChatGPT4's answers got shaky very fast. Asked about a particular somewhat obscure technique, it parroted some very nice-sounding bullshit, which got the concepts very backwards (but it was still very confident and grammatical).

When asked to summarize a published paper about a new innovation in this context, ChatGPT4 very grammatically and confidently completely missed every significant point, again in a very backwards way.

Worse yet, when asked to tell us the paper's publication date, author, and author's employer, all written clearly at the top, it confidently got everything massively wrong. Even after being told to look on the line after "Author: ". Even after being told to look after "Author: FName". It eventually settled on the correct date, but even after being told FName, it continuously kept spouting false authors (sometimes including correct FName) and hallucinated middle and surnames and employers. Interestingly enough, most of the hallucinated authors and employers were well known in the region. So, it is literally establishing the regional context and just spitting out plausibly relevant stuff for that region, but not even figuring out how to summarize something that a 6yo kid could easily sort.

It was hilariously bad. And only hilarious because it was a test (if someone actually relied upon it, it'd be very costly).

Similar results for my field. I got ChatGPT4 to produce some genuinely useful information. But it would also happily hallucinate papers, authors, companies, chemicals, and pretty much anything else.

Middle-of-the-road, where ChatGPT4 has ingested insane amounts of data, it probably knows more than we do, and may be useful for some things.

As soon as you go even slightly off the centerline, it becomes VERY unreliable.

If you take it at face value, you are a fool, being fooled.

EDIT: typos, readability

It doesn't help me writing Unreal C++ code, though it's heavily documented online. It makes up functions that are not accessible from the namespace I'm using, cannot supply valid types to parameters and other basic errors like that.

I asked GPT4 to give me specific examples of the meaning of the word "silly" across the past 1000yrs. It gave me the authors and examples I was expecting to read, but the meanings were inaccurate and misleading.

If there's GPT Power I'm missing out on, please tell me where it is. It's probably my lack of skill, that the machine cannot give me an experience that would push me into the pro-GPT cheerleading camp.

I asked it to describe myself recently, by my name and country of residence, and it replied with the correct occupation (founder), and then completely hallucinated a company I have never created. The more you ask it minute tidbits of information, the more it falls over into a bullshitting state.

"Similar to other prompting-based approaches for reasoning in LLMs, RCI can be viewed as using the LLM’s output to write to an external memory, which is later retrieved to choose an action"

"our experiments use LLMs on HTML code"

Irrelevant since people do have the module to reflect and vet thoughts.

I don't have strict definitions for any of the quoted terms because I don't think we really understand them yet.

It seems very possible that human generative content shares much in common with 'stochastic parroting'. But I'd hypothesize there's a filtering process, sometimes at the individual level, sometimes at a group or even civilization level that (so-far-over-long-enough-timescales) tends to cull some information that has low 'value' and propagates other data that has higher 'value'.

When an AI system can do the same, that's a big inflection point as you can give it energy and time and it will self-improve indefinitely.

It's pretty clear humanity has this capability. It's not clear whether AI does (or ever will). Or, perhaps it might already be there, and we just don't understand enough to know for sure.

People have some emergent machinery for symbolic reasoning in their brains. When they don't just react, they can continue working on an idea and improve it.

When it comes to humans, there is no fundamental difference between training a model and using it. People continue to learn all the time, which allows them to build models of new things and to update their existing models.

People can also interact with the outside world on their own initiative. When you combine this with the ability to learn and reason, it may transform the internal models from arbitrary mathematical constructs to increasingly accurate models of the outside world.

Nothing in this discussion is particularly new. It's just the Chinese Room Argument from decades ago, except that this time it's not a thought experiment. While a static Chinese Room can fool humans and be useful for various purposes, a general intelligence also needs other mechanisms.

We are able to understand right from wrong, we are able to answers with what is true or what we believe is true. We are able to lie or deny the truth. We are able to say we don't know because we think we don't know and not because we have been programmed not to answer, etc.

Overall I think people saying that, are just saying the difference is that we have a conscience and AI doesn't. Which in a lot of case does matter.

To some extend, LLMs have some of the properties ascribed to those fictional AIs like the ability to access a wide amount of knowledge.

They lack the precise rigorous thinking that those fictional AIs have though - LLMs even hallucinate.

Those fictional AIs are what people want because they perfectly complement our weaknesses - our tendency to forget stuff and our tendency to occasionally screw up reasoning; those fictional AIs act as a “safety net”.

At the moment many more limited “dumb” tools like a regular search engine algorithm or your handheld calculator does a better job of being a “safety net” went it comes to specific things.

I've been reading the discourse here and honestly not once have I ever read a reply wondering just where all the data "Open"AI has was acquired from.

I asked this from GPT-4:

Q:

On planet Roro, entities live on physical structures. These structures are built in a way to withstand the rays of star WOD shining on Roro at day and electromagnetic storms causing high-speed transfer of air molecules at night. Some structures have "cavities" in them so the dwellers in the structure could actually see outside. Planet Roro rotates around WOD. Also planet Roto rotates alongside a conceptual axis Z. This axis Z penetrates the shape of Roro and makes two points of contact with Roro we call A and B. Assume structure X must meet this condition: Every said "cavity" in structure X is pointing to A. What can you say about the average temperature in structure X?

GPT4:

Given the information provided, it is not possible to determine the exact average temperature in structure X. However, we can infer some general ideas about the temperature based on the environmental conditions and the orientation of the cavities.

Since the cavities in structure X are pointing towards point A, it suggests that they might be facing away from the star WOD. In this case, the structure could receive less direct sunlight and heat than structures with cavities facing the star. As a result, structure X might have a relatively lower average temperature during the day compared to structures that receive more direct sunlight.

However, the high-speed transfer of air molecules caused by electromagnetic storms at night could still affect the temperature in structure X. The overall insulation of the structure and its ability to retain heat would also play a role in determining the average temperature.

In summary, without more specific information about the materials, insulation, or other factors influencing the temperature in structure X, we can only infer that it might have a relatively lower average temperature during the day due to the orientation of the cavities. The nighttime temperature would depend on the structure's ability to withstand the effects of the electromagnetic storms and retain heat.

Q:

Can we infer that the temperature around structure X, is the minimum among all structures on Roro?

GPT4:

We cannot definitively conclude that the temperature around structure X is the minimum among all structures on Roro without more information about the other structures and their specific locations, orientations, and materials.

While it's possible that structure X might have a relatively lower average temperature due to the orientation of its cavities, other factors could also influence the temperature around a structure. For example, a structure built with better insulation or in a location with more shade could have a lower temperature even if its cavities face the star WOD.

Additionally, Roro's rotation, its axial tilt, and the distribution of structures across the planet could lead to variations in temperature due to differences in sunlight exposure and local climate conditions. To definitively determine whether structure X has the lowest temperature among all structures on Roro, we would need more information about these factors and a comparison of the conditions around each structure.

TLDR: I think that polar bear test was in the training set (since it's a rather popular trick/IQ question)

If we change the names of the concepts, but still refer to planets, the sun and geography, GPT-4 can't figure out that A and B are north and south poles of Roro (earth)

It's not necessarily all languages other than English, but certain languages could cause LLM's problems.

Here's what ChatGPT had to say about it:

Embeddings can be more effective on some human languages than others, depending on factors such as the complexity and structure of the language, the availability of training data, and the quality of the embedding model.

Languages with relatively simple grammatical structures, such as Chinese and Finnish, can be easier to model using embeddings, as there are fewer rules to learn and the meaning of words is often more closely tied to their context. Languages with more complex grammatical structures, such as Russian and Hindi, can be more challenging to model, as there are many different forms of words and complex syntactic rules to consider.

In addition, the availability of training data can also impact the effectiveness of embeddings. Languages with large amounts of high-quality training data, such as English and Chinese, can be easier to model than languages with smaller amounts of data, as there is more information available to learn from.

Finally, the quality of the embedding model itself can also impact its effectiveness on different languages. Some embedding models may be better suited to certain languages or language families based on their design and training methods.

Overall, while embeddings can be effective for modeling a wide range of human languages, their effectiveness can vary depending on the specific language and the characteristics of the embedding model and training data.

...

While it is true that machine learning models, including those used for embeddings, do not directly know about grammatical rules, they still rely on patterns in the data they are trained on to learn the semantic relationships between words. In languages with more complex grammars, these patterns can be more difficult to discern, which can make it more challenging for embeddings to accurately capture the meaning of words.

For example, in languages with complex declension systems, such as Russian or Latin, the form of a word changes depending on its grammatical role in a sentence. This can create many different forms of the same word, which can be challenging for embeddings to learn to associate with a common meaning. Similarly, in languages with complex syntactic rules, such as Hindi or Japanese, the word order and structure of a sentence can convey important information about its meaning, which can be difficult for embeddings to capture without a deep understanding of the grammar.

In addition, languages with more complex grammars may have a wider variety of vocabulary, including many words with subtle or nuanced meanings. This can make it more challenging for embeddings to learn to distinguish between similar words and accurately capture the full range of semantic relationships between them.

Despite these challenges, embeddings can still be effective for modeling languages with complex grammars. By training on large amounts of data and using advanced neural network architectures, embeddings can learn to capture the subtle patterns of language use and accurately represent the semantic relationships between words, even in languages with complex grammars. However, it may require more training data and more advanced modeling techniques to achieve the same level of accuracy as in languages with simpler grammars.

This reads more like an all-star capabilities brochure than someone’s real life/work

[0] I couldn't recommend earlier variants (even GPT-3.5) because they weren't powerful/reliable enough. Way too many simple mistakes to be a net positive. In my experience, GPT-4 is far superior. [1] Applies to all LLMs, not just OpenAI's GPT-4. Everyone else is catching up quickly.

I seriously doubt GPT understands which laws and case laws apply to which areas. It could be creating a contract that does something you don’t want or even a one that isn’t enforceable.

Pay a lawyer. There’s a lot to gain if you do and a lot to lose if you don’t.

What we seek is "reliable".

> Most humans are

What we sought is to shrink them in number and power.

Having used the API a bunch, the ChatGPT API (GPT-3.5-Turbo) feels like it has a 10-second attention span in comparison to GPT-4. GPT-4 is waaaay more powerful for actually building stuff on top of, adheres to your prompt in a much more stable way, and its reasoning is way less unstable in general as well. Writing long blocks of code without making "stupid" mistakes works much much better, etc.

The only downside is: it's so, so, soooooo slow in comparison to GPT-3.5-Turbo. You'll literally have to wait 2-3 minutes for the response with larger token amounts.

When ChatGPT was released, a lot of people (even on this site) were trying to get it to think abstractly and were applying human qualities to the bot, when it's relatively clear that it's all just fancy statistics. Even today, when talking to non-tech people, there's still a lot of people who believe it's a magical thinker-in-a-box. Someone has to talk those people down, and it ain't gonna be me.

It's not perfect, but it doesn't have to be to have applications IRL. Yeah, the web chat is meh whatever, but the API is where it actually shines. It's the integration with existing business processes and code that allow new, more human ways of interfacing with systems. Asking it for what it knows in its statistical blob, probably will lead to heartache. Giving it some context and asking it "what can you infer from this?" is where it's strongest.

The fact that many are mentally incompetent cannot be a reason to produce something similar. When somebody does a job, we expect such job to be done properly, and "brains" are pretty much required, intelligence, if only for the occasional cases outside norm - otherwise, there is automation.

> if its useful

It is not easy to confine an affabulator to the contexts where it will be useful.

Maybe I missed it, where does the author say they are using GPT-3/3.5, not GPT-4?

I agree with your message in general, there is night-and-day difference between GPT 3.5-turbo and 4. But I'm still not able to see which one the author is using.

Edit: seems the avatar in ChatGPT tells you which model you're using, never noticed that myself. Thanks everyone :)

Firstly, we still don't truly have many good explanations for intelligence and consciousness that aren't self referential. Drawing hard conclusions on this subject about generative AI and human intelligence, even in 2023, is being hasty, in my opinion.

Which brings me to a lot of the criticisms made about LLMs. The idea that an ideal AI could or even should be capable of constantly generating original output is just plain weird. Where do people get this idea? How often do you meet a human being who hasn't gotten most of their ideas from somewhere else? Even synthesized ideas are forged by other ideas. Ideas that seem original may appear just that, and we just don't see where the connection lies between it and its predecessors. It's silly to dismiss an AI because it "parrots" things, as if that's not a pretty remarkable achievement. It's also very reductive; anyone who has used ChatGPT and doesn't see that it can pretty reliably munge together different ideas is blind or is lying about their use of ChatGPT.

Expecting perfect accuracy and knowledge is also not a reasonable expectation of an AI. An intelligence localized in spacetime is necessarily going to lack understanding of many things. To look to something like ChatGPT as a source of authoritative answers, and to dismiss it upon finding its answers to be specious, is to expect the AI to be God. Well I'm sorry, but we are nowhere near speaking directly to God.

In regards to "true understanding", for all we know, language processing is the workhorse of what humans consider to be acts of intelligence, and individual humans aren't as intelligent as they believe themselves to be. Our ability to communicate with each other about a variety of topics in real time is not so much a matter about being hyperaware of every given fact that is relevant and weighing them in a Bayesian model; how well we can converse is primarily dictated by how rapidly we can construct language by parsing the input, making connections, and producing output. I know I'm not the only one who can speak on a subject only to realize, after having completed a sentence, that I hardly even thought about what I was saying.

My PM is asking it questions about technical problems (like system design) preemptively and using it as a starting point as if its an engineer. So now instead of thinking through problems from a blank slate, I'm debating with ChatGPT. It's been just awful.

And what's worse, his prompts always have certain biases or misunderstandings in them. For instance, he always wants to use third-party services for everything because he thinks it means we do less work (often not the case). So ChatGPT tells him what he wants to hear. He's using it as an extra engineer who always agrees with him.

Nearly all humans are parrots, aka mimics. One of the great demonstrations of it, which anyone can observe 24/7, is Reddit (and the Reddit knowledge spouted so commonly across the platform).

There are mimics, amplifiers, and originators/creators. Mimics are ~95-99% of the population. Amplifiers are - broadly speaking - teachers, people with bullhorns that train the mimics off of something another person originated/innovated/created.

It's why humanity has been so successful. The mimics are wired to try to copy what is shown to work, and they often do it quite well (keeping up with the Joneses is a wired behavior tied to plentifulness, survival and thriving).

GPT's parroting is it's upper level

I also seriously question whether anyone making the 'parrot argument' has actually used ChatGPT for anything meaningful. If it is nothing more than a parrot, then parrots are significantly more creative than I realized.

Parrot might be harsh, but remeber this Star Wars quote:

Ability to speak doesn't make you intelligent.

True or not, I find a tone of this sort of fortunate. Please don't come up with catchy ways to stop discussion of points.

This is becoming common belief among people in the AI field. I'm convinced it is a case of a metaphor overtaking reality. With the LLM model in mind, we can see how Humans are "kind of like" LLMs, therefore we start to believe humans are just LLMs.

Of course this forgets the enormous ways that humans are nothing like LLMs. For example: Humans don't have to read the entire internet to learn how to write complete sentences.

If it was not a ChatGPT Plus offering but instead freely available, there would be way fewer shills for it (im assuming) and there would be a lot more balanced reporting on it. By default, anyone speaking to the competence of GPT-4 is willing to give OpenAI money, which may already be sign of a bias.

Im not willing to pay for a product the free tier of which is shit, even when told a hundred times "no but the paid one is MUCH better!". Of course it probably is, but its a little irrelevant.

I think the doomsayers on these LLMs would be the same folks predicting doom when agriculture was mechanized. When massive portions of the workforce toiled in agriculture, folks thought farm machines would lead to mass unemployment and social horror. Instead, world hunger has dropped, and massive portions of the workforce were freed up to do things like work on LLMs.

Maybe the hand-wringing is different this time because privileged classes feel threatened.

AI might turn out to be a disaster not because it is smarter than us, but because stupid people will believe it is and come to rely on it, even more so than they already do on Google or Wikipedia.

What really strikes me about the ChatGPT output I have seen is the tone of complete confidence with which it delivers answers (except when it is programmed to respond "as a language model I cannot blah blah blah").

For me that is something which provokes an instant negative emotional reaction to what is said, because things rarely are that simple and clear cut, and any person or machine pretending that they are seems likely to me to be either simple-minded or trying to con me / advance some agenda.

That most people apparently don't have that reaction to it is frightening to me.