Hacker News Reader

Comment by angarg12 | Hacker News Reader

I keep reading examples like this, but every time I used ChatGPT at work it ends up saving me a bit of time completing my task (at best), definitely nothing as impressive.

I wonder if these examples are heavily cherry-picked, or if I'm terrible at using the tool.

View on HN

Typical Zero shot Q/A is like coming up to a person, shouting a question and then taking the first then that comes to mind as the answer. LLMs work quite well in zero shot despite this. They undoubtedly work better with a. Chain of thought b. Feedback

Gpt-3 performance on MultiArith goes from 18% zero shot to 93% with chain of thought and feedback (or 79% with just chain of thought)

All of this is to say that complex chains will produce better results. Probably they cherry picked but that's definitely not the only thing going on

Chain-of-thought is still zero-shot performance. It's just prompting "thinking step by step" at the start of your prompt.

It can be that yes but the first CoT paper introduced few shot examples to induce the actual CoT and those generally produce better results. But yes you're right.

> the first CoT paper introduced few shot examples

It doesn't actually. See https://arxiv.org/pdf/2201.11903.pdf

Perhaps you are thinking of the GPT-3 paper ("Language Models are Few-Shot Learners") which has plenty of these examples.

What do you mean by Feedback here? Is there a paper you’re referencing for Chain of Thought with feedback?

Feedback is a. pasting output of code back in b. For the benchmark inporovement I was talking about, they basically had gpt generate an algebraic and python solution of. Proceed if the output of the two matched, try again if it didn't. Guess I should have added another one - Consensus

I'm definitely getting better at asking questions, but the tool still seems to have some very rough edges.

Here is my most egregious example. I asked ChatGPT to produce code for a very specific approach. It produced ok code, but I didn't like the approach, so I asked it to do it a different way. It produced a second function, but I ended up liking the first one better.

The first code had a bug though, so I asked ChatGPT to fix it. It apologized and applied the fix... to the second code (which didn't make sense). I pointed this out, and it profusely apologized... and spitted out the same code again. I corrected it once more time, and did exactly the same thing.

After 3 tries I gave up, fixed the code myself, and started a new chat with this code.

Had it been a person I'd think it was a troll, a blithering idiot, or gaslighting me.

Same here. It sounds really impressive until i try try it. Ive been trying it as help for coding a web app this week. Sometimes it's really helpful. But often it wouldn't work, I kept correcting it and telling it which errors i got. So it keeps giving me new suggestions that also didn't work, and it keeps adding code that doesn't help. So i end up with a bunch of long code from all the attempts that still doesn't work. So i have to fix it by hand anyway. Like i said, sometimes it's really good, but either I'm also bad at asking, or they have cherry-picked examples.

I suspect that there is a huge selection bias with regards to ChatGPT since reporting impressive results gets the most clicks.

True, but I think this is totally fine and expected. We’re in an experimentation phase — a new discovery has been made, and everyone is experimenting, poking around, and testing it out. This is the only way we figure out what it will be good at (and what it will bad at!) Someone has an idea, tries to come up with a way for GPT to execute it and succeeds or fails. Along the way, we start developing better techniques and answers to the question, “what is it really useful for?”

And in a while, we’ll know a lot more. But right now, we’re learning and discovering. IMO, this is a useful lens for viewing the news around Chat GPT. (Well, more useful than saying “it’s the future!” or “it’s useless” or “it’ll be the end of us!”)

It started as cherry picked but as you come up with better solutions and handling unexpected situations it becomes magically good at building things

It's a bit of both. The savings that ChatGPT did is impressive. And these examples are quite cherry-picked.

100% cherry-picked usually i cant even get the complete boilerplate of a project, because it losses its context.

let alone the fact that you cant get around the token limit.

Part of it that this is ChatGPT + tools (given via langchain). This opens the door to a lot of power.

This is a good example: https://langchain.readthedocs.io/en/latest/modules/agents/ge...

I spent a ton of time hacking together something like that only to just find langchain supports this: they definitely need to bring the tools use case to the top of the documentation since it's by far the most tedious part of what it offers to figure out on your own

Reply on news.ycombinator.com