Hacker News Reader

Comment by nsagent | Hacker News Reader

Was just talking with a microbiology researcher yesterday about ChatGPT. They use it for helping to write code for analyzing their lab data and they were surprised I said it's likely to lead them astray at times, especially if they don't know the code well enough to spot the errors it might generate.

Despite my PhD being in NLP, they still seemed incredulous at my assertion. I truly think the general public have not been adequately warned that they cannot trust the outputs of LLMs. I wonder how long until that fact becomes commonplace.

View on HN

I think there's a certain level of denial involved in this as well. There is a category of person who really wants AI to become a Big Life-Changing Thing. Maybe they are older and remember the early iPhone or the heady days of home computing in the 90s. Or maybe they are young and just eager to be part of the next big thing. Or they're lazy/busy and eager to avoid the hard work of editing/proofreading/revising. Or they think they are smarter than everyone else, and that the caveats only apply to regular people, but not them. In any case, there ends up being a kind of emotional attachment to the AI usage, which gets in the way of them considering evidence that it can be unreliable.

(This phenomenon is not unique to AI, it shows up in all areas of life.)

Oh you have no idea how badly this is going to go over time.

I watched a presentation by some psychology researchers last year where they were presenting their paper for applications of LLM technology in early-therapy diagnostics. They were literally throwing patients at it. I showed one of the enthusiastic followers, a high ranking PhD in the field, how easy it was to get it to give me suicide advice and she was absolutely horrified and totally unaware of the possibility this could happen.

All these things were missing in the associated paper of course. Gotta keep those grants flowing.

> totally unaware of the possibility this could happen.

A rare human would easily admit their mistakes. Especially when their career and money and power is at stake.

That is why I didn't go into academia. I saw someone burn an entire research outcome on purpose because it'd compromise their speciality entirely.

Mind sharing which speciality?

I figured that out, the first time I submitted a Swift question to ChatGPT, where I would normally have used StackOverflow.

It hallucinated an Apple API call. Not just that, but that nonexistent call was the entire fulcrum of its "solution."

Don't use "hallucinated". Use "bullshitted". https://link.springer.com/article/10.1007/s10676-024-09775-5

That's a problem with lots of less common languages. The accuracy seems to scale with the number of github repos and SO questions. Most Swift code being in proprietary iOS/MacOS apps doesn't help it.

Also, Swift allows extensions of fundamental types. I often declare extensions of Int and Array. It’s fairly common to add computed properties and functions to basic APIs.

I’m pretty sure that a number of folks extended the API with this property (which is what I did, once I figured out how to address the issue. It just made sense to do so), and ChatGPT read that as a standard system call.

That’s likely to be an issue with more mainstream languages, as extension is a fairly common pattern, these days.

I think I may start tagging my extensions (like a naming prefix), to signal they are added after the fact. I do a lot of open-source work, so it’s likely to be used as training data.

Eh, I had the same experience asking questions on Java and Python.

The more strict your requirements, the less reliable LLMs are.

Using agents improves this dramatically if you want to try again. Putting a linter, test runner and self reflection in the loop fixes a ridiculous number of LLM issues.

It makes sense to me that the general public would have this impression now that AI is a product being sold to people. If you tell people "well, AI can make a SME's job easier and save them time, but they'll still need to understand the output and take the time to assess it for errors," it will rot on the shelves.

Customers don't want to pay for an AI to make a SME's life easier, they want to pay for an AI so that they can stop paying the premium for for SMEs.

I'm hearing of pressure for LLM-generated science-related apps, such as auto-generating workflows from natural language English for "non-coder scientists". It sounds wonderful, but unless the user 1) sees the workflow being generated, and 2) has the necessary skills to discern whether that workflow will generate the desired results, and 3) has the ability to modify the workflow/pipeline manually or continue prompting the LLM until it spits out the correct workflow (see #2), then it's not only useless, but dangerous.

> I truly think the general public have not been adequately warned that they cannot trust the outputs of LLMs.

to the contrary, every company now advertises using "AI" (the latest handle for LLMs) to "solve X" with maybe some small print somewhere about the side effects, much like medication advertising

Did you see Google's ad during the Olympics? No mention of hallucinations that I recall (also probably the stupidest use case for LLMs they could have thought up; pretty shocking).

> Despite my PhD being in NLP, they still seemed incredulous at my assertion.

I've the feeling the term "AI" erased in a lot people mind the fact text generation is a classical NLP text; they actually had no clue what NLP is in the first place. So we became to the current absurd situation where opinions of people with actual expertise and knowledge highly related to the thing are dismissed.

One use of AI is to do something that you couldn't do yourself. In many cases that means it can successfully bullshit you.

I think there's a segment of society that blindly believes anything data backed. This is pretty frequent in product and marketing too. The article here shows some people who blindly trusted their marketing teams.

And while everyone here are going to laugh at those chumps who use data driven stock price indicators or LLM code generators, they'll throw a fit if you tell them that the data on calories and vaccines aren't always that accurate either.

My experience is the opposite: laypeople are excessively pessimistic on LLM progress ("AI is so dumb. It tells you to put glue on pizza and eat rocks)", usually due to a remembered anecdote that's either years old or reflects worst-case performance (only egregiously bad AI mistakes make the news).

Frontier models are better than they were and "feel" fairly reliable, although all the AI problems of 2021-2022 conceptually do still exist.

>I wonder how long until that fact becomes commonplace.

When it starts killing enough people to make the news.

I'm calling it now; people will be informed of the mistakes AI makes when the first "fake news" story pops up that has inaccuracies due to ai analysis of data. Then it will be all downhill from there.

when it becomes a crime of misrepresentation, to call it AI when it is not AI but just a piece of code like any other.

We are already there! Humans misrepresent data on purpose in the media all the time in order to create misinformation and sway opinions. If anything AI creates deniability, "Oh our so and so used gpt, woops!"...

>We are already there! Humans misrepresent data on purpose in the media all the time in order to create misinformation and sway opinions.

That isn't really the same thing. I was thinking of something more along the lines of "AI designed car randomly explodes" or "AI doctors misdiagnose patients and mix up prescriptions." The AI equivalent of the Therac-25[0]. Something a lot worse than "sometimes the media lies," which we already know, and something explicitly the fault of AI confabulation and automating away humans at what should be necessary stopgap or validation steps for the sake of streamlining or cost-cutting.

>If anything AI creates deniability, "Oh our so and so used gpt, woops!"...

Luckily, that doesn't seem to be the case. In every story I've seen where people have tried to blame AI they didn't get away with it, because the AI is just a tool and the human beings are still legally responsible. People still try though.

[0]https://hackaday.com/2015/10/26/killed-by-a-machine-the-ther...

You mean like AI assisted profiling of minorities? or AI assisted drone strikes? We are already there. Just not for the comfy cushy western peeps. Oh wait that republican self drove her tesla into a lake and died, so yeah- already there.

Self-driving cars already make the news when they crush pedestrians or plow themselves into trucks on the highway. Companies are still pushing/pursuing the tech. Why would the public response to LLM's be different?

Unless they are having GPT come up with the calculations and doing the statistics, I doubt that. They likely just need help writing the code, not the math.

Reply on news.ycombinator.com