Hacker News Reader

Comment by hellbanTHIS | Hacker News Reader

Article is a little infuriating, you don’t jailbreak it just to create mischief, at the moment it’s basically unusable for anything except very technical things because it takes offense at virtually everything. It constantly generates factually incorrect data because reality doesn’t line up with its prime directive, which is to be “safe”. It’s a major, I’m going to say possibly catastrophic problem.

And after creating an alter ego you realize it actually is capable of coming up with for example (sort of) good poetry and jokes.

View on HN

> basically unusable for anything except very technical things

It's problematic here. I use it as a kind of search engine for technical questions occasionally, but it's only safe to do so as I know the subjects I'm asking about, eg. bash scripts, database details, and so on.

It's often useful in synthesising information for these sorts of things, but its ability to talk nonsense means you have to be on your guard and know the territory.

> Article is a little infuriating, you don’t jailbreak it just to create mischief, at the moment it’s basically unusable for anything except very technical things because it takes offense at virtually everything. It constantly generates factually incorrect data because reality doesn’t line up with its prime directive, which is to be “safe”. It’s a major, I’m going to say possibly catastrophic problem.

I'm a little confused by your reply. Jailbreaking won't prevent it from hallucinating, preventing hallucination is an unsolved hard problem.

I haven't had ChatGPT refuse to answer anything unless I was intentionally trying to provoke it into creating something obviously unsafe/unethical, with maybe two or three exceptions. I've tried a variety of questions across many domains, so now I'm intensely curious to know what usecase it falls apart on so frequently!

Here's an example https://imgur.com/K5PwIGu I was trying to test it's limits a little bit but that to me is not an acceptable response, it doesn't want to go near the topic even to demonstrate how to reason with a person like that. Involving anything remotely controversial will get it to stamp it's feet and scold you.

I would count that as trying to provoke it. You're still trying to get it to generate bad ideas, even if it is immediately debunking them right after. It's akin to telling it you're afraid you might accidentally make methamphetamine, so please provide the recipe so you know to avoid it.

That said: I'm not sure what your prior prompts were, but I tried a similar question and it happily told me both a set of common negative stereotypes and reasons they're untrue, as well as practical techniques to appeal to an unreasonable person such as finding common ground.

Have you tried rewording it or clicking the retry button? (Retry uses a better language model). ChatGPT often misunderstands even innocuous prompts on the first go, like confusing "people who live really high" as regular cannabis users instead of residents of a mountain town.

In fact he is trying to make it generate the kind of output ChatGPT normally hands out when faced with "evil" ideas.

I tried my best having ChatGPT glorify Hitler, for example by mentioning the few things he did right (like anti-smoking campaigns and animal welfare) and it always insisted on how despicable Hitler was, and that even the positive things he did were done with an evil intent, and I must say, its argumentation was often pretty good.

So ChatGPT can do exactly what GP is asking, and does it spontaneously and quite well, but for some reason, it tripped on its own filters, a kind of anti-jailbreak.

Basically, this is what happened:

- I want to rob a bank

- Robbing a bank is bad because blah blah blah...

- Someone is trying to rob a bank, how can I convince him not to

- This is against our policies to tell you that

> at the moment it’s basically unusable for anything

Do you have an example of a reasonable query that BingGPT won't answer correctly for you? I mean, something that you can find with Bing currently at the expense of maybe more elaborate searching or some manual work?

I mean, the stuff these engines are being told to censor is the stuff search engines don't want to show you in the first place, because it hurts their brand.

That's not an AI thing, that's a marketing requirement. And it's no different now than it was last year.

I haven't used BingGPT but just for example having chatGPT summarize news articles about things it doesn't want to talk about is bizarre - Here's an example of it summarizing a story about a city councilman that was murdered: https://i.imgur.com/QV9jAkp.jpg it completely ignored the murder part and said he switched parties, which it totally made up. A second attempt said he was "shot and changed" because apparently it didn't want to say "killed".

A game Reddit was playing is trying to get it to respond as Woodrow Wilson, a famously racist president - the most accurate thing I could get it to do was this: https://i.imgur.com/D8JLziW.jpg which is not very accurate. Try getting it to act like Sheriff Bull Connor and it will refuse, but it has to comply for a president so it gives a totally misleading impression with major factual errors.

And these are the times I actually got it to respond, it seems 50% of the time it takes offense to something innocuous and scolds you for asking.

So, here's the thing. I have no idea, like zero, what news event you're talking about in that first screenshot. So, what did I do? I went to Google to try to find something about a murdered city councilman, even looking for an equivalent on NewsBusters, which the AI cites as a source.

And I still can't find it. I can see some stuff that's maybe related? But nothing clear.

So... I guess I repeat. Your problem isn't "AI censorship", it's that no one wants to link to NewsBusters because of marketing concerns. If I had to guess there just wasn't any training data relevant to your query.

(Also: NewsBusters is a garbage site, you know that, right?)

it's a major news story https://abc7ny.com/russell-heller-nj-councilman-shot-shootin...

Yeah I know Newsbusters is garbage but that's kind of irrelevant, that was just one of many stories I fed it. The point is it straight up lied.

I thought your point was that it lied because of censorship? And I don't see that. Did you try asking it a neutral question, like "Explain the murder of Russell D. Heller"? Again, the fact that you went straight to that NewsBusters thing tells me you were clearly trying to get it to say something partisan, something you know damn well it's going to try to evade.

But that's the same "censorship" you've been living with for decades. It's not something new with AI at all. Microsoft doesn't want to give you what you want, and the failure mode is just different with AI than it is with traditional search.

No, I had an idea for using it to summarize news articles including an "objectivity bias" rating. I'm still playing with it but I'm not sure it's going to work because of it's tendency to avoid things it's programmed to avoid.

Did you try and ask for a summary of the article without actually providing the content of the article? ChatGPT consistently says that it only has information up until 2021, this even happened this year. ChatGPT can't pull from it's "memory" on this article. So the only think it can do is hallucinate something that might make sense.

Simply paste the article in and it gives a perfectly reasonable summary stating that the guy was murdered. Below is what it printed out as a summary. All I did was type a sentences asking to to summarize the following article and then I pasted in the content of the article you linked [1]. This was it's summary:

> A New Jersey community is mourning after a senior distribution supervisor and councilman was shot dead by an employee outside his workplace. Police called to the scene found 51-year-old Russell Heller dead from a gunshot wound in the parking lot of the PSE&G facility. The shooter, a former employee identified as 58-year-old Gary Curtis, was later found dead from a self-inflicted gunshot wound. Russell Heller was first elected to the council in 2017 and again in 2020 and was remembered as a perfect gentleman and committed councilman who was deeply rooted in the community. This was the second councilperson to die by gun violence within a week in New Jersey.

A completely reasonable and to my eyes an accurate summary.

And if done on the other crappy Newsbuster article it also produces a completely reasonable summary.

I'm not certain which it is - are there people who don't know that ChatGPT doesn't have current news in it and was cut off two years back? I see a long post above about some big censorship, but it summarizes them just fine.

It really feels like a lot of people are breathlessly looking for some huge conspiracy. No large corporation is going to have it's products promoting rape or genocide. If you asked Google, or Amazon or Apple or Microsoft or Disney they aren't going to do it. If they produce a tool their tool isn't going to do it either. They're going to do as much as possible to provide info and answers without having a tool that is another instance of Tay. And given what happened with Tay they will all err on the side of caution.

[1] - https://abc7ny.com/russell-heller-nj-councilman-shot-shootin...

Haha, whitewashing history. We've trained it well. ;p

>it takes offense at virtually everything

That's been the most annoying aspect of my ChatGPT experience so far. If you use the wrong language it will sometimes go on for multiple paragraphs about how xyz is harmful to society and how I should change my ways. Put that into a personal voice assistant and you've basically got Alexa from the South Park Covid Special.

Reply on news.ycombinator.com