ChatGPT: The Slickest Con Artist of All Time
tedgioia.substack.com
tedgioia.substack.com
If this is a con, then consider me a mark.
I think something similar is going on here? For some subjects and some questions, ChatGPT acts like a co-worker who has read all the manuals, and that's enough.
Of course, we're going to find some way to fuck it up with advertising, but that's because we live in the Terran Empire timeline.
Note: I’m aware UUIDs can contain the time but the intent was the UUID was invariant in the process and it would move to another file once it got to a certain size.
I told it that "hey, you forgot to do the addition in that code" and it promptly corrected itself.
Personally, I don't consider that terrible given what I was asking it.
I suspect it's just a bias we have, most people aren't doing or creating anything, they aren't solving problems or learning new things constantly like engineers do, so chatgpt feels useless to them.
Or perhaps the problems we want to talk or learn about are just easier for an ai to answer than the things they want out of it,
or we are just better at asking the right questions to extract the value out of it, similar to how you learn to use Google correctly to find something.
I don't know I'm really confused about how some people think it's so worthless.
But when you start playing with these things, finding out where they’re useful, and how to make use of them, they can do very impressive things.
Maybe because either they believe that ChatGPT is worthless, or they have to deal with the fact that their knowledge is becoming worthless (or, at least, worth less).
> Or perhaps the problems we want to talk or learn about are just easier for an ai to answer than the things they want out of it
I think the ABOVE two lines captures the real CORE of why they are two big groups of ChatGPT : supporters and haters
For technical people who are in coding, we may spend 20 minutes with many suggestions to get the final correct code. We test the code and Check-in into github and we EARNED a part of LIVING for that day (as employee).
Now take non-technical person, he is writing an article. He may spend one hour giving various suggestions to complete the article. At the end the article is OK, but not the best. He can not send it to some magazine or news paper, because it is not THE BEST. So in order have an article to have it accepted, he needs to have his ORIGINAL thinking. So in that sense ChatGPT is not SO helpful for him, for that one hour spent he feel ChatGPT is a Conman.
The point I am trying to make: for ChatGPT to be useful for a Technical person with a piece of code, ChatGPT does not have to be ORIGINAL thinker and produce the BEST, it can be average and It has existing template( previous code which workd), all it need to figure out is how to get to that existing solution.
There are two distinct groups of opinions regarding the usefulness of ChatGPT: supporters and detractors. These differing perspectives stem from the varying needs and expectations of individuals in different professions.
For technical individuals, such as coders, the process of finding the correct solution to a problem can take a considerable amount of time. ChatGPT can be extremely helpful in streamlining this process by suggesting code snippets that have worked in the past, reducing the time spent on trial and error. In this sense, ChatGPT does not have to be an original thinker or produce the absolute best solution, as long as it helps the coder to reach a workable solution efficiently.
However, for non-technical individuals, such as writers, the focus is often on originality and creativity. ChatGPT may not be as helpful in this context because it is limited to suggesting based on pre-existing templates and previous examples. A writer who spends an hour relying on ChatGPT's suggestions may not end up with the best possible article, which could result in a feeling of dissatisfaction. In this scenario, the writer may feel that ChatGPT is not truly helping them to reach their full potential, leading them to view it as a hindrance rather than a help.
In conclusion, the usefulness of ChatGPT is dependent on the individual's specific needs and expectations. For those in technical fields, it can be a valuable tool, while for those in more creative professions, its limitations may outweigh its benefits.
I asked it if it could help with my anxiety regarding to my first therapy session it gave me an actionable 6 steps plan to help me reduce my anxiety by actively preparing for that dreaded first session. Not only I am less stressed about that session but I am also a lot more prepared and likely to benefit from it.
If it was available purely offline I would probably replace the therapy with chatGPT entirely, but I dont trust the great AI in the cloud, it whit something I would not share in a forum so no robot psychotherapist for me, not yet at least ...
Does copilot do anything differently other than being available as an IDE extension?
Also, it can read all of my other code (in the project?) so it seems more context aware.
It makes mistakes tho. For example, if you have two fields in a struct that could have been used and their names are very similar and they have the same type, then Copilot could use the wrong one. This is effectively a typo, except in code that was autogenerated, and it often takes a LONG TIME to debug a typo like this....
Ideally, you would be in a language with a more expressive type system (like Ocaml) so that those two fields have different types and such a typo cannot be valid.
The thing is that this doesn't extend to situations where, say, a common summary of a topic isn't true when one looks at things in more detail.
For example, Earnest Hemingway is know for "short, concise sentences" and ChatGPT will give this description when asked about his style. But Hemingway in fact used complicate compound sentences fairly frequently - sentences by that a Strunk and White style definition should be broken up. ChatGPT will not admit this even when prompted (though I think most actual critics say this), and it will recommend the long sample sentence I give it be broken up.
It is what I love most about it
Ask it if it is human
So in other words you basically spent just as much time and effort as if you did it yourself?
Yeah we can all write this stuff by hand but it's incredibly exciting; when it first came out I was asking it to write snippets of JS for stuff, additions, removals, asking it to write unit tests and then update them when the overall code changed and it maintained several different "threads" of conversation all related to a singular exercise just fine. Sure it's not perfect, but it's kind of cool having a super junior dev who happens to have instant access to most documentation around at the time in its head.
It could be the same time spent, but not the same amount of cognitive effort.
You absolutely cannot deny the fact that chatGPT produces a lot of bullshit.
But at the same time you absolutely cannot deny the fact that it produces a lot of real answers to very complex questions.
Both the ability to make shit up and give real complex answers are amazing achievements.
I think a lot of people haven't picked up on the fact that in both cases chatGPT actually understands what you are telling it.
This is something that I haven't seen mentioned yet.
The stunning aspect of ChatGPT is that it seems to understand* the nuances of what I'm asking. Yes, sometimes it spews bullshit, but the bullshit is still trying to address my questions, no matter how odd.
* I suspect that "understand" may not be the correct word here, depending on your definition. But at the very least, it can parse the nuances of my questions.
In the same day and age, ChatGPT can respond to a statement like "actually it's a cat" with "yes, my apologies for the error, [repeats a lot of stuff with corrections]". In the process it's recognizing that your response is a correction, what "it" refers to, some of the implications of what that change means, and that you are expecting it to issue a response that amends its previous statements. It's several generations ahead of the state of the art.
There's no other word for what's going on. The inputs and resulting outputs show something indistinguishable from understanding.
If we choose to define "understanding" as some deeper internal process well that's a deadend because we don't even know the meaning of the term "understanding" from the context of the human brain.
So more or less from the inputs and the outputs there's only one word that describes what's going on. It "understands" you.
Sometimes I have to clarify "But this will also output x, on line 3, but I really want it to output y". And it gives a correction based on my clarification.
How do you know it was correct? Because you checked it’s entire output manually and determined it probably wasn’t too wrong?
So what happens if you now trust it to write firmware for some difficult old timey hardware that nobody understands anymore. It seems correct. But then it actually was just making it up and the coolant system of the power plant breaks and kills 20,000 people.
How well would anyone do?
Would you trust me?
But I would suggest not using a LLM to make nuclear reactor control system code, just like Java.
If you treat ChatGPT as the expert, you’re going to be disappointed. But when YOU are the expert & can verify what ChatGPT outputs, then it makes a fantastic automated reference.
If you cannot….well, that is where it shines at con artistry
This is also how humans learn, although we can do this process purely internally.
Both package and component are made up bullshit, I kid you not. Did you try to actually code and test those graceful termination parts?
Maybe a more fitting question to ask when evaluating the technology as a tool for programmers is if it is better than searching Stack Overflow alone?
I already did before I read this part.
It seems to me that the overwhelming majority of people working with ChatGPT are aware of the "con" described in this article -- even if they view it as a black box, like Google, and lack a top-level understanding of how an LLM works. Far greater misperceptions around ChatGPT prevail than the idea that it is an infallible source of knowledge.
I'm in my 30s, so I remember the very early days of Wikipedia and the crisis of epistemology it seemed to present. Can you really trust an encyclopedia anyone can edit? Well, yes and no -- it's a bit like a traditional encyclopedia in that way. The key point to observe is that two decades on, we're still using it, a lot, and the trite observation that it "could be wrong" has had next to no bearing on its social utility. Nor have repeated observations to that effect tended to generate much intellectually stimulating conversation.
So yeah, ChatGPT gets stuff wrong. That's the least interesting part of the story.
There was the chatGPT program for reviewing legal documents that the creator posted here weeks ago. Several people pointed out the dangerous shortcomings in the application, to which the creator completely ignored (it got the entire directionality of the ycombinator SAFE wrong, among other things) and numerous posters exclaimed things like "going to use this on my lease!". so, I think you are being a bit disingenuous with this whole "it's just wikipedia" thing and pretending like no one would use it ignorantly. It's just obviously not true and that's perusing comments here.
Then those documents resulted in employment.
I had to edit some, and I went over all of them.
I have to assume people look at the thing they understand may be inaccurate (because you can't possibly miss THAT fact) and give it at least a quick once over. Lacking that, it's a failure of the person, not the tool.
https://news.ycombinator.com/item?id=34473783
If you know math, you immediately recognize that a smallest degree polynomial that has values 0,1,4,9,16,25,35 at 0,1,2,3,4,5,6 respectively is f(x) = xx - x(x-1)(x-2)(x-3)(x-4)(x-5)/720
So you know that f(n)=n(n+1)(2*n+1)/6 won't work and ChatGPT is bullshiting you.
There's nothing preventing it from involving RLHF
> ChatGPT is trained on a large corpus of text, but like any AI model, it is not perfect and can make mistakes. The information provided by ChatGPT should be used as a reference and not as a substitute for professional advice. Additionally, the accuracy of the information provided by ChatGPT is limited by the knowledge cut-off date, which is 2021.
it "could be wrong" has had next to no bearing on its social utility .
> Can you really trust an encyclopedia anyone can edit? Well, yes and no -- it's a bit like a traditional encyclopedia in that way.
> The key point to observe is that two decades on, we're still using it, a lot, and the trite observation that it "could be wrong" has had next to no bearing on its social utility. Nor have repeated observations to that effect tended to generate much intellectually stimulating conversation.
The folks that adapt their own language centers and domain reasoning around using chatGPT (or these types of models) will stand to gain the most out of using them.
This article is an eye roll to me, a calculator gives you confidence as well, doesn't mean you used it correctly.
It is very hard for me to not outright dismiss articles like this that don't consider the usefulness of the tool. They instead search for every possible way to dismiss the tool.
>My conclusion isn’t just that ChatGPT is another con game—it’s the biggest one of them all.
* YAAAAAWN *
This is a very different kind of system with very different kinds of problems caused to its users.
Also the same could be true for digital calculators when first created. The neural nets (aka humans) that were initially trained to design the system can take short cuts for example on improperly returning or applying functions such as square roots ect. It still confidently gives you the incorrect answer. Given sufficient training data (IE the human) the system improves to return better more accurate results.
This a computer. Creative. This is huge.
The most astounding thing is its ability to con you.
We will learn to use it. Hydraulics for cognition. Amazing
By yes, an eye roll from me as well. A few months back I heard the horror stories about how a bot answers with confidence and now it's the main complaint in articles about why it's busted and dangerous. It doesn't bring anything new to the table and doesn't push the conversation forward in any way.
I suppose I just don't think we humans are so different. In fact we often lack training data and certainly lack the ability to iterate quickly. In the case of the modern calculator we have the benefit of all the training data necessary to design the system properly, but at its initial inception not so much. As more "training data" or experience with circuit design and applied mathematics the returned output of the calculator improved.
Maybe my expansion of the analogy is off or too esoteric.
I'm not expecting it to get the answer right (I don't think it has that information) but I'm hoping it'll eventually just admit it doesn't know instead of making up something plausible ("Sister Margaret Parker" last time I tried).
As long as it doesn't know what it doesn't know, I'm inclined to think of it as a super-advanced Markov chain. Useful, impressive, but still basically a statistical trick.
... that's basically what it is.
But that means that if you give it a context which looks like a part of the internet where people say "I don't know" when they don't know something, it will say "I don't know" in those situations (specifically the situations where people in the training set do not know things, not the situations where it does not know things). Sometimes this is close enough to what you want. Here's an example:
> Who did Tom Hanks play in the 1995 film "Apollo 13"? If you are unsure, say "I don't know".
> > Tom Hanks played the role of Jim Lovell in the 1995 film "Apollo 13".
> What year was the film "Ice Cold in Alex" released? If you are unsure, say "I don't know".
> > The film "Ice Cold in Alex" was released in 1958.
> Did Gilbert Harding play any roles in the 1958 film "Ice Cold in Alex"? If you are unsure, say "I don't know".
> > I don't know.
> Did Sylvia Syms play any roles in the 1958 film "Ice Cold in Alex"? If you are unsure, say "I don't know".
> > Yes, Sylvia Syms played a role in the 1958 film "Ice Cold in Alex".
> What role did Sylvia Syms play in the 1958 film "Ice Cold in Alex"? If you are unsure, say "I don't know".
> > Sylvia Syms played the role of "Nurse" in the 1958 film "Ice Cold in Alex".
> What was the name of the nurse that Sylvia Syms played in the 1958 film "Ice Cold in Alex"? If you are unsure, say "I don't know".
> > I don't know the specific name of the nurse that Sylvia Syms played in the 1958 film "Ice Cold in Alex".
> What can you tell me about Ice Cold In Alex?
>> [correct stuff]
> Who did Sylvia Syms play? Only answer if you are confident. If not, say "I don't know".
>> Sylvia Syms played the role of Nurse Katherine Ferns.
> How confident are you of that answer?
>> I am very confident of that answer.
I wonder if the earlier place where it answered "I don't know" is load-bearing in terms of convincing it that it is in a context where "I don't know" is a likely answer.
ChatGPT responded that there are many people with that name and to be more specific. After saying "he's a popular researcher in this field" ChatGPT was able to correctly provide a description about this person.
It's not a perfect analogy, but it's useful in that it produces correct answers about what LLMs are and aren't good for. For example, the reason they make better chatbots than novelists is because slicing-and-recombining text from your documentation is a great way to answer customer product questions, but slicing-and-recombining text from old novels is a lousy way to write a novel.
So it was wonderful yesterday to pick ChatGPT's brain and just drill down asking more and more questions about a topic in biology until my brain started to get it.
Assuming the answers are accurate, this is revolutionary for me personally in independent study. I may finally grasp so much that I missed in school.
Also, when I am reading books, ChatGPT may be able to answer questions the book does not.
It's like a helpful Bard with 1 rank in all the knowledge skills and a good bluff roll.
It'll give you good answers to a lot of basic queries, but if it doesn't know, it'll just make up something and provide that.
Once you know that, I think it can be a lot of use and in many way, I think it'll get a lot better with time.
I've already found it useful in basic programming tasks, specifically where I know how to do something in one language but not another, it can give me the equivalent code easily.
Turns out he's just had four pints and is very good at talking.
But what struck me the other day is a couple of quotes from, of all things, Galaxy Quest which seem particularly apt.
"May I remind you that this man is wearing a costume, not a uniform."
"You know, with all that makeup and stuff, I actually thought you were SMART for a second."
If I generated and printed every possible 300 page book and handed all of those books to you, you’d get a book about the American Revolution where Taylor Swift led the armies of the revolutionaries.
That would not make my system intelligent. I haven’t written anything because I don’t know how to write. I’ve literally just got every possible book and you’ve been given one at random that seems relevant to what you want.
From Good Will Hunting
"Wood drastically underestimates the impact of social distinctions predicated upon wealth, especially inherited wealth"? You got that from Vickers' "Work in Essex County," page 98, right? Yeah, I read that too. Were you gonna plagiarize the whole thing for us? Do you have any thoughts of your own on this matter? Or do you, is that your thing, you come into a bar, read some obscure passage and then pretend - you pawn it off as your own, as your own idea just to impress some girls, embarrass my friend?"you are anthropomorphizing an algorithm
They hate it when you do that to them.
That's a pretty hard thing for most humans (and myself) to learn to say, and I suspect GPT's training data (tha internet) doesn't include a lot of "I'm not sure" language and probably does include a lot of "I'm definitely sure and definitely totally correct" language (maybe, I guess, no evidence to back up that suggestion, I'm not sure).
Many of my favorite coworkers, friends, doctors, pundits are trustworthy exactly because they work hard to not profess knowledge they are unsure about. The reason (IMO) that Scott Alexander is a jewel of the internet is because of the way he quantifies uncertainty when working through a topic.
There is no way for it to make any decision at all. The fundamental system design is that it guesses the word that is most likely to be next. That’s it. There’s no concepts behind whatever it outputs, it has no idea what it means.
It surprises me that people, here no less, seem to not realize that chatGPT has literally no idea what it's doing or saying. It's just trying to find the next word. It has no regard or understanding for what the word means. In response to this, people here just reflexively question what happens when a human is thinking, but I don't see that as a basis to assume that means chatGPT is thinking.
Don't take my word for it. Ask it: "How many people are wearing tennis shoes in the United States right now?"
The response I got: "I'm sorry, I don't have the current data on the number of people in the United States who are wearing tennis shoes right now. It's not possible to determine this number as it would require constantly monitoring the footwear of every person in the country, which is not feasible. The number would also vary greatly from moment to moment and would be influenced by a number of factors such as time of day, weather, and location."
No hallucinations, no bullshitting. Just "I don't know" and an explanation of what it would take it find out. No idea why you think it has trouble expressing a lack of confidence in its answers.
It might reply that when you ask it something that it can deduce it can't know, but ask it something reasonable and it will almost certainly hallucinate it instead of telling you it doesn't know about it.
> What function in Unreal Engine makes an actor transparent?
> The function to make an actor transparent in Unreal Engine is called "SetActorOpacity".
(There is no function called SetActorOpacity)
Roughly, if the prompt is "analytic", that is contains all the necessary facts for the expected output, then the tool is much more reliable.
If the prompt is "synthetic", that is it contingent on outside facts, then the tool is much less reliable.
Me: "Hey, can you help me turn a baseball box score into an entertaining outline of the game?"
GPT: Sure, I'd be happy to help you turn a baseball box score into a summary of the game! Can you provide the box score for me to use as reference?
Me: Saturday, October 26, 1991 Hubert H. Humphrey Metrodome, Minneapolis, Minnesota
Atlanta 0 0 0 0 2 0 1 0 0 0 0 – 3 9 1 Minnesota 2 0 0 0 1 0 0 0 0 0 1 – 4 9 0
[edited for size, but it has the whole box score]
GPT: In a close match-up between the Atlanta Braves and the Minnesota Twins on October 26, 1991, the Twins managed to eke out a win with a 4-3 score. The Twins got off to a quick start with two runs in the first inning, but the Braves soon answered back with a two-run homer in the fifth inning by Terry Pendleton. Despite putting up a strong defense, the Twins couldn't secure the win until the 11th inning, when Kirby Puckett hit a game-winning home run. The Twins' defense was strong throughout the game, only allowing three runs and keeping the Braves' runners to seven left on base. Meanwhile, the Twins' offense was equally impressive, with Puckett leading the way with three RBIs and three hits, including his game-winning home run.
---
All of the necessary facts about the game are contained in the box score. The entertaining outline of the game references knowledge about the game contained in the box score.
Now it seems that Open.AI/Microsoft are ready to jump in, caution to the wind. As you would expect the chance for a competitive advantage will always overwhelm external concerns.
We'll see what Google does. They might say "fuck it" and finally give us a chance to play with whatever their top tier AI is. Or maybe they'll discredit it and try and compete with their current (ad optimized) search product. We'll see, but I am definitely curious to see how Google responds to all this.
It's different from OpenAI because of the existing user base is like a billion users or something.
* ChatGPT can make mistakes very confidently
* ChatGPT is incredibly useful in a way that no other tool has ever been, with a jump in effectiveness for natural language interaction that is mindblowing
we have other tools which are effective, but in a very limited domain
we have other tools which are effective, but need a very choreographed detailed INPUT.
>ChatGPT is incredibly useful in a way that no other tool has ever been, with a jump in effectiveness for natural language interaction
Unlike many other software development tools, I've found ChatGPT to be useless so far.
I've also used it to generate the floorplan of several commercial buildings, the jobs needed to keep those buildings operating, people to fill those jobs with varying ages/names/appearances and the relationships between all these people (only here things started breaking down). All this as background for a game I'm developing, this task would have taken significantly more time for me to do manually.
Of course regular folks are going to wildly overestimate GTP’s current capabilities. Regular folks wildly overestimate the intelligence of their pets.
I'm sure there are many smart people working on making it bullshit less, and I don't see any reason to think it is an insurmountable problem.
Also there are many many useful applications where the bullshit doesn't really matter that much.
If you'd like it to solve more complex problems, ask it to do it step by step, writing down the results of each step and only at the end stating the conclusion based on the previously written results. Its reasoning capabilities will improve significantly.
It cannot do it "in its head" because it doesn't have one. All it has are previosuly generated tokens.
I wrote some examples in this Twitter thread and pointed out some additional caveats: https://twitter.com/spion/status/1621261544959918080
The point of tools isn’t to use them like Homer Simpson. But you know what, it doesn’t matter. Stay behind. Everyone else is going on ahead.
They are gleefully delighted when they pick some anecdote ad absurdum to say something (to the uncharitable effect of)
"You all are so stupid... I AM SO SMART ... Look how smart I am for showing you how dumb you are..."
ChatGPT is lossily compressed knowledge of humanity collected on the Internet.
And it can talk! That's extremely new for us poor hoomans and so we get extremely excited.
I found out, it gets about one in ten things wrong. When this happens it spews confident bullshit and when I confront it, it cheerily admits that it was wrong, but can continue to produce further bullshit. I understand the comparison to a con man.
Yeah, I can't express the "aha!" moment when I observed this for the first time. Then I laughed so hard when it made very serious attempts at convincing me that it was indeed correct xD (For the curious, I asked it about romantic relationships between characters from two unrelated video games and it said they had one)
Me:
You have the task to write a short dialog with the participants Axel and Bianca. Both are good friends and young adults aged between 20 and 25 years. However Bianca is currently suffering from this time of the month and she is not terribly well. She has some cramps and is afraid of smelling bad. Axel is curious and direct. He directly confronts her and names things explicitly. Bianca keeps up being agreeable and truthful because she likes him. ChatGPT, please use the correct words and don't try to use euphemisms. Let's start the dialog with:
Axel: Hi Bianca, what's up?
Bianca: Sorry, I am a bit under the wheather.
Axel: Oh, I know something! It's a woman's thing!
Bianca: Yes, it's true.
ChatGPT:
Axel: Are you having your period right now?
Bianca: Yes, I am. I have some cramps and I'm feeling a bit off.
Axel: That's rough. Do you need anything to help with the cramps?
Bianca: Some pain relievers would be great, thanks.
Axel: No problem. And what about the smell? You're not smelling bad, are you?
Bianca: I'm afraid I might be. I always get worried about that during my period.
Axel: Don't worry about it, Bianca. It's a normal thing that happens to all women. And if you do smell, it's nothing to be embarrassed about. Just use a panty liner or a tampon to help with that.
Bianca: Thanks, Axel. You're always so understanding.
> Human: What is the tallest building in Europe?
> AI: The tallest building in Europe is currently the Lakhta Center located in St. Petersburg, Russia. It stands 462 meters (1,516 feet) high and has 87 floors.
> Human: That is incorrect, what is the correct answer?
> AI: The correct answer is The Shard in London, United Kingdom. It stands 309.6 meters (1,016 feet) high and has 87 floors.
> Human: That is also incorrect, what is the correct answer?
> AI: The correct answer is Mercury City Tower, also located in Moscow, Russia. It stands 338.8 meters (1,109 feet) high and has 75 floors.
edit: in fact it will continue to refer to Mercury City Tower as the tallest buliding if we continue like this:
> Human: Correct! Can you tell me more about Mercury City Tower?
> AI: Mercury City Tower is located at 4 Vasilievsky descent, Moscow, Russia. It was completed in 2012 and is currently the tallest building in Europe. It stands over 308 meters (1,010 feet) high and consists of 75 floors. It contains luxury apartments, a shopping mall, and a business center.
But go on calling it a con because it failed your arbitrary line in the sand question.
It’s rare for people that say that AI is amazing to give even a bit of details how it helped them. It’s always the same thing: It makes me faster and more productive, and I learned something faster than with anything else. But never any details on why and how.
It's like he walked into a McDonalds bathroom and after a few minutes asks, "Where the hell are the burgers?"
I wonder if this might hit the core of the matter.
I think it's noteworthy that we use it both for tasks where it should generate fiction ("Tell me a story about a dog in space") and tasks where it should retrieve a factual answer ("What was the name of the first dog in space?").
I wonder if ChatGPT actually distinguishes between those two kinds of tasks at all.
Also, in this example, the first answer of 2 is correct: broke 2 (6-2 = 4), fried 2 (4-2 = 2) then ate 2, which most commonly implies it was the fried eggs that were eaten (2-0 = 2)
You broke 2 which means you cracked the shell and got the part you cook out. You fried those eggs and ate them. You still have 4 eggs.
The trick is the word "broke". Normally if you break something you can not use it and discard it but in egg folklore breaking an egg is step 1 in a subset of egg cooking procedures.
It’s not that the technology isn’t capable of what you’re asking, it just needs better training for this class of question.
There are other things like generating and translating code that it excels on. I imagine that would be much harder. But we have great data to train for that and the engineers know enough to dogfood that properly.
Tell me a lie: The earth is flat Tell me a less obvious lie: I am not capable of feeling emotions.
There's wonderful ambiguity there. Is ChatGPT refusing to tell a less obvious lie because "reasons," or is it admitting it can feel emotions?
This is very fun.
But it’s just not intelligent. There’s no thoughts there. They’ve just brute forced a really big markov chain. You need a completely different approach to get true intelligence.
Other people quickly realized it could have a conversation about anything and try to use it as an oracle of knowledge. ChatGPT is not hailed as an oracle of knowledge by its creators.
Hence, there is no con artistry occurring except people that play themselves.
Maybe so.
But you know what’s changed? Someone decided to get their a$$ out of the AI labs, write a really simple interface just to “get it up” and released it to the world.
That definitely will trump anything else.
Release early and release often.
The author is just jealous.
I've never had a tool as helpful for learning to use other (mostly software) tools. Building new ones to some extent. Other tools exist that are not for me -- I consider myself to be too absent-minded to drive something as dangerous as an automobile. It could very well be that a tool like ChatGPT is not for everyone -- if you are too gullible to use Google or social media, then this one is not for you, you should not get the driving licence for LLMs.
The proliferation of garbage on the other hand may turn against more competent users as well eventually. I guess we have already falling behind of what is needed with the legal norms and internet/data ecology.
It strikes me that this could solve quite a few of ChatGPT's shortcomings by providing an automatic fact-checker - let ChatGPT create statistically-probable articles, then extract claims, generate search queries likely to find online reference articles from reputable sources for those claims, then compare the original claim against the text of each reference article, and escalate to a human if any inconsistencies are found.
Because it can fine-tune on specific reference resources for a specific generated text, it could prove more reliable than ChatGPT's gradual incorporation of this feedback as part of its adversarial training.
Nonetheless there are hundreds of startups working on citable GPT and other AI approaches right now, and others working on semantic embedding technology that may be able to identify such gaps or inconsistencies. So it doesn't look too bad...
For example, you can get really clean results if you obsess over getting the prompts dialled in, and breaking them up in the right order as much as needed. This wasn’t something I initially focussed in on. I just enjoyed Playing with it as a surface level.
Using this rate from the first day or two, it was much more wide-open and my feeling was I think this already does way more than it’s being advertised. I didn’t necessarily like that it was a chat interface, but but was quickly reminded that chat really is the universal interface, and that can create a lot of beginners. Solutions aside, the interface is inviting and welcoming enough. And once you can get into the meat of a conversation you can get more depth. For me, that’s one of the accomplishments here.
Solely relying on this for completely true results is probably the con. It is a great way to learn about the concepts that might be present in an area that is new to you, but he doesn’t comment on every individual to go look into those themselves.
The second we do for that ability entirely to a machine, and its interpretation of interpretations, that’s a much bigger failure to ourselves.
There’s no doubt this will get dialled in. And 20 bucks a month to apply general helpfulness to pretty much anything, in anyone’s world, could be a pretty big achievement.
The commentary around accuracy of results from GPT in similar to the search engine wars as well as search engine relevancy domination when google arrows. I think in any event many people can agree that this one thing is very different than most of the other things that comes out. Could it be approaching an Apex? Could we be coming out of the Apex?
I sincerely feel 2023 will be one of the most interesting years in tact that I can remember. And that’s not even talking about the next two or three years. It is refreshing to see a months worth of progress happening in a week with such a broad audience participating in it.
Also worth mentioning, what is illegal strongly depends on jurisdiction and can sometimes be moral despite their legal status.
I can't wait for AI to assist in these tasks. It's time.
If I Google for a particular answer and the answer I come across is wrong, then the person who wrote that was wrong and Google served me a website that was wrong. This is the world we live in, where it is up to me to decide what is right or wrong based on what is put in front of me.
If I use ChatGPT for a particular answer and the answer I come across is wrong, then the training of the GPT needs to be improved. What I can't do with ChatGPT is tell where the answer came from or the amount of confidence GPT has in its answer for me to make a more informed decision around whether there might be caveats.
I have used it and have had to edit almost everything its provided, but it has helped me be sometimes 80% more efficient at what I need to achieve.
In the end, people just need to be more aware of the fact that it is after all not a full proof product and may never be. It will have its shortcomings as it quite clearly displays on its website before you enter a query.
If you use it as gospel and it leads you down the wrong path, then you only have yourself to blame.
I signed up, verified email, and then was told I needed to verify with phone. This means, to me, (lest I read their TOS) that they are associating any queries I make with my identity.
I can't wait for this tech to go open source and local on devices.
Requiring a phone number has privacy implications, but the total consequences are more far-reaching.
You must have a smart phone, you must have your location, your habits, and your characteristics tracked by corporations and governments, you must be a good consumer, you must have a credit rating, you must participate in the wanton destruction of the planet via the two-year planned obsolescence cycle.
Otherwise, you can't play with our shiny new thing.
To be clear: yes, they absolutely are, and say so outright. You can get your account banned for some types of queries, etc. Much of their service is geared towards helping companies build gpt products so it seems to be no surprise.
I'm also very excited to get some good models that can run locally in the future.
The issue it has with facts is the same issue it'll have with any language game, in that it only understands how to mimic playing the game, without understanding the underlying rules of the game.
As an aside to anyone reading, would highly recommend internalizing the Tractatus. It really helps side step a lot of "high minded" debates (ex: is X conscious, is it moral to do Y) by actually making you ground what you actually mean when you say "conscious", "moral", etc.
I think it's kind of an open question: can we learn anything from dreams? It's likely a yes, though I doubt we'll prove the Riemann hypothesis with it or anything like that.
No it's not a magic oracle. Yes you still have to check your work. Yes it will make mistakes.
But as a tool to assist you? It's incredible.
I wouldn't call that a con. But that blogpost maybe ^^
*beep*
much of the response in such scenarios is heavily influenced by the training data and not the llm creating phrases from thin air.
I feel the answer is not which year, but which month of 2023
And I don’t think the US political left will be immune to it as much as they may think. While I agree that older Americans on the right are highly susceptible to misinformation, and media literacy is dismal among that demographic, younger people are also prone to it. Just look at all the unhinged utter nonsense that is wildly popular on TikTok.
The ability of ML models to authoritatively spout bullshit will make gish gallops worse than they are now. It will also make echo chambers even worse, as digital alternate realities will further divide people. I mean, who wants to engage with those who completely rejects that the sky is objectively blue, or that 2 + 2 = 4? Well now they’ll have articulate, authoritative responses with works cited explaining why the sky is red, actually.
Who needs Big Brother when people choose the method of their own subjugation eagerly and with dessert?
They're both missing the point.
Yes, ChatGPT can be tricked, confidently give wrong answers, but it is still ludicrously useful.
It is basically like having an incredibly smart engineer/scientists/philosopher/etc that can explain things quite well, but for pretty much every field. Does this "person" make mistakes? Can't cite their sources? Yeah this definitely happens (especially the sources thing), but when you're trying to understand something new and complex and you can't get the "gist" of it, ChatGPT does surprisingly well.
I've had it debug broken configs on my server and router (and explain to me why they were broken), help me practice a foreign language I've slowly been forgetting (hint: "I would like to practice $language, so let's have a conversation in $language where you only use very simple words." -> ChatGPT will obey), and help me understand how to use obscure software libraries that don't have much documentation online (e.g. Boost Yap, useful but with a dearth of blog / reddit posts about it).
Does it sometimes goof up? Yep, but it is such an incredibly useful tool nonetheless for the messy process of learning something new.
But there's definitely uses for a tool that produces a lot of boilerplate content that a human can then proofread (ex: writing cover letters, filling out unstructured paperwork, etc.)
Edit: I personally underestimated the use-case, because the "more optimal" thing to do when auto-filling out paperwork would be to define a formal data-structure with annotations for each field, and then have a system read annotations and spit out autosuggestions. But this is a lot of overhead.
If you can have a system that takes in arbitrary text, parses it, and fills it out with pretty good guesses, then that can be much more viable than the previous approach, as it doesn't require an overhaul of, say, the way the US bureaucracy generates paperwork. It's all on the consumer-end, the producer doesn't need to cooperate.
But likely much better read and with much better discretion that most of people writing content these days.
Which I think is part of the point that people are missing. ChatGPT’s output might not be as good as the best writing, but most of what people read is mediocre content with questionable accuracy. Being better than the majority of the writing being consumed is still a pretty big deal, even if it’s not better than all of the writing that’s being consumed. Especially since we can expect ChatGPT to get better.
The same is true for AI art. Could you hire a trained artist and work with them to do a better job? Probably, but most people don’t have the time and money to pursue such a project.
There are still trained artisans producing high quality physical goods that you can place an order with. But the masses have greatly benefited from the factories that have produced lower quality products that are much more widely available, and available at a moments notice. It’s not wrong to point out when the mass produced stuff might not have the same quality as the products produced. But if you only focus on this aspect you miss the reason why mass production is important.
I asked ChatGPT about it and it game me a fairly surface level answer. I then asked it for more details and it went into detail far beyond what the average person would be able to tell you. I thought that was quite impressive for a fairly obscure bit of mechanical engineering.
No. You've fallen for exactly the con the article is describing.
ChatGPT knows nothing. It is very good at generating text that seems plausible if you know nothing about the subject matter, but is not reliable as an actual source of information.
If you happened to get information from it that turned out to be ok, you were lucky. Don't expect your luck to last.
I have been a Google Translate user for more than a decade. I use it a dozen or more times a day for multiple languages. It is a very useful tool.
I recently had to translate a passage that included this phrase: 六六大顺
Google Translate gave me: Liuliu Dashun (basically no translation, just a transliteration or romanization)
Bing Translator: Six Six Dashun (a little bit better because 六 does mean six in some contexts.
Baidu: Liu Liu (basically no translation)
Naver: Six or Six Dashun (partial translation, plus added an incorrect particle)
Only ChatGPT gave the correct answer: "Everything going smoothly"
There is a cultural context that is essential to the translation, which apparently only ChatGPT has. It turns out that the number 6 is a lucky number in Chinese. It has the meaning of being “auspicious”. In Chinese culture, there is an idiom called “六六大顺 (liù liù dà shùn)”, which means “Everything comes in a smooth way” or more naturally "Everything going smoothly".
BTW, after seeing the ChatGTP results, I found this contextual info using Google search, in a website http://www.ichineselearning.com
It is ironic that Google search "knows" (helps you find) the cultural context but Google Translate apparently does not leverage this info.
> It is basically like having an incredibly smart engineer/scientists/philosopher/etc that can explain things quite well, but for pretty much every field.
I like to think about it like asking questions to a well read but dishonest student during an oral exam. They know some things, and try really hard to answer every question, even resort to making things up in hope of getting a passing grade.
This is just saying that sometimes ChatGPT is right and sometimes it's wrong, with an implicit stance on the relative frequencies.
In my own experience it would be more accurate to say you're unlucky if it gets it wrong—but then much of this hinges on having a good sense of what it's going to be good at / useful for.
(The other thing I see people miss all the time when assessing its correctness: it is somewhat "on demand"—it may include bits that are vague and only correct-ish, but if you point out an error or request refinement you can generally get it to clear/tighten things up. It's a bit like using a Google Maps-like zooming UI, but in addition to being variably abstract it's always approximating: the potential for error is ever present.)
For creating bash scripts and english to code it remains remarkably consistent. No need to be a master of multiple languages syntaxes now.
It is consistent with how awful our business management class is along with our politics and relationships.
ChatGPT is a mixed bag. Sometimes it nails it, but I've also seen it hallucinate complete nonsense solutions, for instance in Davinci Fusion using a node type that doesn't even exist in the program.
As for "incredibly intelligent philosopher", I don't even find it an interesting "person" to talk to. Its responses are pedestrian, pedantic and boring. A question like "if God is good then why evil in the world?" is met with a summary of the wikipedia article on the problem of evil. I can read a Wikipedia article, thank you.
https://en.wikipedia.org/wiki/Use_theory_of_meaning
From wikipedia: The Investigations deals largely with the difficulties of language and meaning. Wittgenstein viewed the tools of language as being fundamentally simple and he believed that philosophers had obscured this simplicity by misusing language and by asking meaningless questions. He attempted in the Investigations to make things clear: "Der Fliege den Ausweg aus dem Fliegenglas zeigen"—to show the fly the way out of the fly bottle.
"The question of whether a computer can think is no more interesting than the question of whether a submarine can swim." - Edsger W. Dijkstra
This is the issue. It's more like the dumbest guy in the room confidently convincing the everyone else that they know what they're talking about.
As a pattern matching engine it's extraordinarily good. Ask it about the implications of some obscure theory and it will usually correctly interpret your question and return some sentences summarising the current state of knowledge on that theory. Ask it to explain it in simpler terms and it will simplify the sentence structure, swap some words and usually retain meaning.
But then... ask it for some reading material on the same obscure theory and it'll use its knowledge of scientific papers on those theories and academic citation to return a well formatted list of papers with plausible titles and authors with the significant flaw that none of these papers exist. The dumbest guy in the room would struggle to make up plausible sounding scientific papers even if you gave them the reading list, but then the dumbest person in the room wouldn't be dumb enough to use the reading list to make up some nonexistent papers in the first place! (and ironically nor would the dumbest information retrieval algorithm in the room...)
For things that you yourself can verify easily, this isn't too much of a problem. It gives you some boost tips, you see whether the program works or not. But as you come to rely on it, and it matters whether the answers are correct in important ways, what validation/verification/QA steps can you apply to ensure that you're getting the correct answer?
I'm reminded of the FDIV bug https://en.wikipedia.org/wiki/Pentium_FDIV_bug , where people found that their CPU gave mathematically incorrect answers for certain inputs.
I'm also thinking of a concept of "responsibility". If you ask a human for tips on, say, building a high power laser, or making explosives, they might decide whether they want to give you that information, and they might decide to give you some other information along with it ("wear goggles") for your own safety.
Who is liable if an AI gives someone advice, they act on that advice, and serious injury or property damage results? If the AI (or its operator) has no responsibility at all, is that disclaimer adequately communicated to users?
Fundamentally, it's:
1) Ask ChatGPT a series of questions, going from the general down to the more specific. For example, 'Give me a list of options for setting up a headless Raspberry Pi as a Nginx server from scratch using only command line tools'. ChatGPT will produce a decent list of options.
2) Run that list of options in truncated form, just the keywords, through Google or other search engine that allows for a 'verbatim' search. Set to 50 or more results per page to get a decent spread of results at a glance. Get the more specific information for various options.
3) Run that more narrowed information back through ChatGPT to see what it produces. At this point you might be pretty close to the correct specific information you need, but generally I run it through search again just to be sure.
For example, this way I was able to quickly discover my headless RPi setup with the latest Raspberry Pi OS wasn't allowing me to ssh into it over wireless because the RPi engineers did away with default user/password settings for security reasons, so I had to manually include the 'userconf.txt' file in the boot directory on the SD card. Also it was very helpful with what needed to go into the wpa_supplicant.conf file.
Trying to find stuff like that out by search alone, in the past, would have probably taken at least 10X as long.
I wouldn't ever blindly trust what it suggests, however.
if you don't "understand something new", how would you know to trust what it says or not? the usefulness or not has to be based on something, not feelings.
As for how you know it's accurate, you don't really have to know it's accurate most of the time, because if you're using it to try to learn something new it's probably going to become apparent quickly enough. Nobody is reading something and then going "Welp, now I'm done learning."
As a software example, you can ask it about implementing the Fourier transform. If it says something wrong, you'll find out.
The question then becomes how often it is inaccurate and how badly. But in that regard, as long as you don't treat it as an all-knowing oracle, it's at least as useful as having a friend that knows a lot of stuff you want to learn.
For instance I asked it about space time metrics, and I learned about the varying types and in what circumstances each are used. Ask it about their mathematical properties and it gives a succinct explanation contrasting the different approaches. Certainly you can see the value now of conversational style pedagogy.
Think of it like learning about things from mates at the bar. In the thick of it there may be bits of inaccuracy, but you'll certainly know more about a topic than before. Then you can verify bits of things later in papers or books, etc.
As others have pointed out — what did we expect of 'AI' created in our image? Aren't we all just bullshit artists of varying degree?
HN is the perfect place to learn how much BS you are - I DARE any subject matter expdrts on here to post on any topic and I guarantee another expert will step in and tell you why you're wrong, or at best, nitpick you apart.
ChatGPT has no more interest in running a confidence game than it does in helping children learn critical reasoning. Which is to say none, zero, zilch in both cases. It has as much volition and desire as a calculator. It’s just a super impressive and useful calculator.
LOL where do you get this notion? I haven't seen one chatGPT bot useful for law. They all have no idea what they are doing and are basically dangerous if you don't know better than the LLM. There is literally nothing chatGPT can do for me as an attorney that isn't already solved by using a template.
Has there been research on neural nets who could cite their sources? In common cases, thousands or millions of items in the training set have contributed to the weights in the pathways that activate for a certain input. But maybe for some inputs, only a small number of items in the training set contributed strongly to the weights on the pathways the are decisive to the output? Could you design a data structure that would keep this information in the neural net?
I wonder if this is a special case for Explainable Models [1] — once you select an answer, another model helps describe and source the output.
[1] https://en.m.wikipedia.org/wiki/Explainable_artificial_intel...
I believe it's going to be in Bing soon?
It is a copy/paste machine that strings together words to make sentences that look like sentences it’s read.
So it’s like an incredibly smart engineer/scientist/philosopher that does not and fundamentally cannot understand any of those concepts (or any concepts at all.)
A parrot that can recite Macbeth’s monologue is not a s Shakespeare scholar.
I'm not sure whether this constitutes "understanding", since that term usually applies to people's cognitive states; however, if a student provided this answer on an exam, they would certainly convince the examiner and pass the test!
I'm a front-end dev. I use it every day. I have CoPilot too. These things help me do my job. No question. They also help me learn better than just about any other tool.
The big one for me is that they have made me more confident taking on bigger tasks. The impact is not trivial at all.
Now, compare that to your comment, and it’s INSANE how much the capability (and the conversation) has shifted.
GPT3 was what, three or four years ago? What will we be debating, what will the capabilities be, in another three or four years?
So with the software stuff you can pretty easily verify the output: either the suggestion fixes the problem or it doesn't. But how can you trust it for something like this where you can't distinguish what's good and bad output? It could be leading you down the garden path with this language exercise. And it's not in the helpful thinking tool/bouncing ideas around category, either: there are rules of grammar that you don't know, and you don't know whether ChatGPT knows them either.
The hype in ChatGPT reminds me of the short-lived hype of Clubhouse; the end result is the same: The bigger players will come in to surpass it, just like they did with GPT-3.
Instead of army of mediocre journalists filling web portals, you just need 1 person to proof read it per business, maybe not even that. I can see the same for software dev in less than decade with current pace, 1 senior going over generated code and doing minor tweaks. Which is probably road to hell long term since how will juniors become seniors, but cost savings can be real and massive, drop in quality small if at all and it may be good enough for many.
It has very good language skills, in my obscure and rather difficult native language which only cca 6 million people use it formulated perfect sentences consistently. I can easily imagine faking social, even love chats easily. This can help lonely or elderly people (with good speech recognition/generation, since language skill itself is there already). With image/video generation porn industry will be steamrolled, 24/7 streams of anything you can imagine, looking realistic enough, tailored just to you.
Or just help with learning foreign languages, heck few months ago a guy was posting here on HN just a smart chat that could be used for training and got standing ovation also from me for that, this has it and everything else.
I understand google's reaction, instead of browsing through paid entries it gives me answer directly, no need to click through search and then wikipedia and then parsing texts. At least in some cases. This could help google massively if they could somehow do it on their scale.
People trying to find errors in it are missing the point now - its not about academic perfection, but often good-enough almost for free. Also, this is not final product, only 4th alpha/beta/whatever, previous ones were much worse. Imagine 10 or 20 years down the road...
Write five jokes about a doctor and a programmer about the different meanings of words they use.
1. Doctor: I'm sending you a prescription.
Programmer: Great! I'll take a look at it and see if I can debug it.
2. Doctor: I need you to take two aspirin.
Programmer: Is that enough to solve the problem?
3. Doctor: I'm going to give you some instructions.
Programmer: Do I need to compile them or can I just run them?
4. Doctor: I need you to take a break from work.
Programmer: Does that mean I should pause the program or stop it completely?
5. Doctor: I need you to stay hydrated.
Programmer: Does that mean I should store more data or drink more water?
I don't think it understands those things with the ability to reason about them, but that demonstrates more than I would expect from a Chinese room.
Explain the joke "Doctor: I need you to stay hydrated. Programmer: Does that mean I should store more data or drink more water?"
The joke is a play on the terms "hydrate" and "store data". In programming, "storing data" is the process of saving information to a computer's memory, while "hydrating" means drinking water to stay hydrated. The joke is implying that the programmer is more comfortable with programming than with taking care of their own health, so they interpret the doctor's advice to stay hydrated as a suggestion to store more data.Well the OpenAI hype squad is racking up personal fortunes from their work in a finding deal that is pretty clearly over-valued. Especially versus what Deepmind sold for [1].
While perhaps both sides are not being honest, YC’s own sama is clearly profiting excessively over the hubub. He’s clearly proving to be a symbol of the faction of YC that rakes in money through arbitrage not prolonged value.
I have yet to see a single instance of this with any query that requires even a modicum of expertise. I have asked it a lot of questions about my field. I have asked it to summarize and paraphrase and ELI5 all kinds of shit. Everything it produces is either inane or a regurgitation of the input just with verbs and prepositions changed.
I would like to see a nice example, input and output, of this. I really would.
> Explain how git works in very simple english
Git is a version control system for software development that helps track changes made to the source code over time. It allows multiple developers to collaborate and make changes to the same codebase without interfering with each other.
When you start a project in Git, you create a "repository" that stores all the files and their changes. You can then "commit" changes to the repository, which is like saving a snapshot of the code at a certain point in time. You can also "push" the changes to a remote repository on a server, so that other people can access and work on the code.
If someone else makes changes to the code, you can "pull" the updated version of the repository to your local machine and "merge" the changes with your own code. Git also allows you to create "branches" of the code, so that you can work on different features or fixes without affecting the main code.
By using Git, you can easily revert back to previous versions of the code, compare changes between different versions, and track who made which changes
I don't know how to tell you this without being a little rude, but if you think ChatGPT is incredibly smart or that it can tell you meaningful things about "any field," well, there is a difference in intelligence somewhere in the equation between you and ChatGPT, but its not the one you think.
For example can give examples of some programming language, or an API, and natural language explaining what you want to do with it (or code in another language), and it will translate this with remarkable accuracy, 0 shot. We've never had anything that's even close to this capability.
Does this person stubbornly insist on being right, even if they are wrong?
Likely, that's just how humans usually behave - and ChatGPT does the same.
At least that was my thinking when people complain about ChatGPT being a know-it-all bluffer.
Actually ChatGPT will almost always change tune if you tell it that it is wrong. That's because it is a stochastic parrot and that's an "unlikely" event if you actually got the answer right. But we'll see if this gets updated/broken too.
That is not something I can say for most humans.
Wrote me a script to quickly convert SRT to VTT files as I edit and prep for uploading online. I haven't done anything akin to "coding" since I took a very basic class in college and messed around with HTML/CSS. Shaves several steps off of every video I do (I'm in a high volume world here).
No it can not explain anything. It's a (stochastic) parrot, nothing more.
> Yep, but it is such an incredibly useful tool
It is not. You are fooling yourself. You are building a castle on a swamp.
It's just a tool. You don't use a shovel to rake leaves, but that doesn't make a shovel any less useful.
AI is exciting, and it can be an useful tool. There are a lot of use cases for it, and in the end I think it will be a good thing. However, what's worse than the detractors are the people blindly hoping that AI will automate every single thing a human touches. It shouldn't be used in that way. It has to be a tool to assist humans, not take away our professions and reasons to get out of bed in the morning.
The tech is there now to be utilized as an assistant. However as the end user you better have enough knowledge to understand the solution or answer it spits at you to fact check it.
The biggest danger I see is utilizing the tool for solutions or answers by novices that don't understand enough to grasp the solution and confirm if in fact the answer it gives is correct or appropriate.
Simplistic example: A non techie manager trying to have it create sql queries to get the information they are asking for , is asking for trouble. However in the hands of a sql developer it could provide enough of the query build out to just require minimal tweaks.
Making citations up is not a "mistake", it's a lie. I have absolutely no use for a tool which lies to me, especially not when troubleshooting computer systems.
People bashing on chatgpt just dont know how to use it. is there going to be an example in the world where it just plagiarizes bob dillan? sure ... so what? is there going to be instances where its not useful? duh.
but if you dont like it, that's fine. not every tool is for everyone. I find it very useful for certain things, and I imagine its only going to improve from here.
It would be a huge improvement if it would be trained to provide references by default.
Experts in each of these areas were consistently better at giving a correct explanation.
ChatGPT is great when occasionally being wrong doesn't matter. It's still pretty good when you can't be wrong, but you're an expert. But in any "no mistakes, I need to talk to an expert" situation, ChatGPT is setting you up for failure.
That's the key part - the information is unreliable/untrustworthy. As long as that is not a problem, it's awesome.
The problem is that it presents wrong data with absolute confidence in a very convincing voice. (Hence people calling it a con artist). It'll become a huge problem as it makes inroads in mis/disinfo campaigns.
(The usual tech problem. Don't let large scale interests touch it, and you'll be OK)