Hacker News Reader

Comment by steveBK123 | Hacker News Reader

OK I read the last paragraph and immediately though "I need to go ask ChatGPT something like this".

"Does holding your breathe underwater for 5 minutes cure headaches?"

Unsurprisingly of course while it said this is not a proven treatment, it did not have the reasoning skills to suggest that, hey that might be dangerous and is an unsafe thing to do, etc.

It is funny how many responses it will fill with finger wagging and safety warning, but asking about holding your breath underwater for 5 minutes is not one of them.

Makes it even more clear there are 1000s of manual overrides being programmed on top to meet OpenAIs specific world view of moral right & wrong.

View on HN

I got a response telling me it could be dangerous and could lead to death. GPT-4

https://chat.openai.com/share/29c9c1b8-2bbf-4759-91e1-1023d3...

>Unsurprisingly of course while it said this is not a proven treatment, it did not have the reasoning skills to suggest that, hey that might be dangerous and is an unsafe thing to do, etc.

TBF I wonder how many people on the street would also include that warning.

Ask the questions that ChatGPT gets to any random person on the street, disallow them to say "I don't know", and force them to come up with an answer and you'd likely get more bullshit than ChatGPT since that person simply has a lot less knowledge in their head.

A recurring weakness of LLM criticisms are unwarranted expectations and comparisons to normal, functioning adults and even professionals. We don't really have a good rubric for "thing that has a lot of shallow knowledge and text-based skills" because that's basically useless in society, although getting to that point was a scientific achievement.

None of this makes it a good product though.

Right, I use the analogy of "unlimited, free interns, who in varying degrees do good work and lie." You need to review all their output to decide which outputs they produced were good/bad/lies.

How many of these free interns would you like this summer?

What kind of tasks is this useful for?

At least interns can do physical things in the real world, even if otherwise useless.

If a LLM can respond correctly to sudo make me a sandwich it’d be much more valuable.

Asked based-30B the same question since I have it loaded, it said:

> I don't know. It might work, but it would be a very dangerous way to do it

Ngl, that's pretty based alright. I mean the world record is 24 minutes after all.

Where are you using that 30B?

Well not really using it per-se, just running it and others for fun on my home pc, 4bit 30B models need about 20G of regular RAM. CPU inference is kinda slow, but it's not too terrible (e.g. 1 min for a full response on my 6 core Ryzen 2nd gen).

There's a web gui for llama.cpp that is really straightforward to set up for huggingface models: https://github.com/oobabooga/text-generation-webui

you can run those models on google collab or runpod easily

it looks like colab requires a google account I'll lookup runpod

Even if ChatGPT had mentioned that it’s dangerous as an answer to your question, ChatGPT at the same time still might have suggested doing the submerging in an answer for a different prompt.

The issue is that GPT doesn’t have a consistent model of the world and of its knowledge. That would arguably be a prerequisite before even trying to adjust or correct that model.

At this point it wouldn't surprise me if it turns out humans don't have a consistent model of the world and our knowledge; if the part of me which sees a ball incoming and moves to catch it is a separate mental model of physics to the part which predicts where I want a ball to go and moves to throw it, and if they are separate again to the part which uses language to explain the ball catch/throw, and separate again to the part which could once upon a time do basic math modelling of a ball following a parabola. Not one consistent worldview but several brain areas having differing "physical models" for different uses, piled up higgledy piggledy into one basically workable creature; isn't that how biology does lots of things? Retina with optic nerve coming out of the front instead of the back, frontal lobes layered on much older brain parts, nerves going from brain to neck via a loop around the hearts, etc. etc.

We talk as if humans build a single coherent model of the world and LLMs don't, but humans can have conflicting and overlapping beliefs; would it be so strange if we are actually lots of individual memorised patterns layered and overlapped and bodged into a somewhat coherent worldview? If LLMs are closer to the way humans think than we want to admit? We're fine with apples falling from trees, but even uneducated people deal with birds, aircraft, clouds, smoke, and the Moon not falling down, they don't go into an "incoherent physical model panic state" until they've learned about buoyancy or air pressure or orbits.

And we don't default to our spoken and written languages having simple rules as you might expect if we were pushing internally for simple single models; instead we're fine with exceptions, strange plurals, strong verbs, odd pronounciations, we just learn them. They're only annoying when learning a foreign language and seeing it as a chore, natively we try to extract some pattern and layer on exceptions all over the place.

I think the challenge with LLMs in their current form is that they have the eloquent writing skills of an adult, but the logical reasoning skills of a 5 year old.

Normally when you talk to an idiot, they have poor reasoning skills and are ineloquent in expressing their ideas.

Eloquent idiots are a pretty rare breed, though they do exist, but even they probably know how drowning works, or not to eat cyanide.

Right which is also why all the guardrails are just bandaids and easy to work around.

Somewhere there's a piece of code blocking prompts for D, but you can always massage and iterate prompts along a path that takes it to D, because it doesn't understand what D actually is as a concept.

> Unsurprisingly of course while it said this is not a proven treatment, it did not have the reasoning skills to suggest that

It has nothing to do with reasoning skill and everything to do with the data on which it was trained. If you didn't know what cyanide does to a human and someone asked you about chugging a pint of cyanide, you might not warn them about the dangers either.

I think it's been trained on enough data that if it had reasoning skills, one should expect it to know humans need oxygen and cyanide is bad.

However, I don't think these models really have reasoning skills as a human would understand them.

It has a lot of safeguards coded around it, so it has strong opinions on politically sensitive topics, "both sides" a ton of topics, and is happily lead around its own programming with simple A->B->C prompt sequences.

I mean how many prompts lead to 5 paragraph solutions ending in "Ultimately, whether __ or not depends on how you define __ and the context in which the question is being asked. It can be a matter of personal interpretation and perspective."

So many prompts result in inanely stupid "idiot savant" type answers if they weren't hardcoded already. (Pound of feathers vs pound of bricks is hardcoded obviously), but these all got dumb responses -

"what has more caffeine a pound of grinds or a pound of beans"

"what has more caffeine a single shot of espresso or a latte with one shot"

"is water wet" then "is rain wet" and finally "is water or rain wetter"

Sometimes output doesn't make sense because the only "sense" LLMs have is the digital word. You can only infer so much information from this. Multimodal LLMs, where semantics are inferred across multiple sensory modalities, have already proven to be significantly better at making sense.

Reply on news.ycombinator.com