Hacker News Reader

Comment by letmevoteplease | Hacker News Reader

So it doesn't have an understanding of sippy cups. If you tell it the sippy cup has a lid, it says:

>If the sippy cup had a lid on it when you turned it upside down on your bed, then the diamond would still be inside the thimble which is inside the sippy cup. So your diamond would now be on the kitchen counter where you placed the sippy cup. Is there anything else I can help with?

View on HN

> It doesn't have an understanding of sippy cups.

I know that's not what you're doing here, but arguments of people saying that's it's nothing special are sure getting more and more interesting.

I haven't touched GPT4 yet, but there are legitimately things GPT3.5 does not understand. Often simple, easy, intuitive things. Pretty much anything that isn't being optimized for as a marketable product. Ask it law questions, perfect it passes the bar. Ask it for an ascii drawing of San Fransisco (or any other text encoding of geography), it fails flat on its face often placing things labelled "East" to the left and vice versa or listing the same place twice. Ask it to make a transcript and it often messes up which speaker has which role and what is appropriate for them to say. Ask it to do any sort of recursive transcript, that is a transcript inside of which another transcript will be generated and discussed within the conversation, and it blatantly ignores you. All this and so much more.

GPT4 is much better in that regard. With GPT3.5 I've had very similar problems. Plainly ignoring very simple directions (e.g. Do not output sentences starting with "In conclusion"). GPT4 unfortunately still does that, but it's like 10% of the problem that it used to be.

I also happened to test nested contexts, and at least when going meta-meta it's doing noticeably better than it did before.

I would have never guessed such performance is achievable with this kind of token predicting text-only model.

If current trends continue, in 10 years time I’ll no longer be able to pass a Turing test.

So the lid on the cup changes whether the diamond stays in the thimble? Or is the diamond-in-thimble guess entirely random?

Reply on news.ycombinator.com