Hacker News Reader

Comment by smusamashah | Hacker News Reader

I have kind of pet peeve with people testing LLMs like this these days.

They take whatever it spits out in the first attempt. And then they go on extrapolate this to draw all kinds of conclusions. They forget the output it generated is based on a random seed. A new attempt (with a new seed) is going to give a totally different answer.

If the author has retried that prompt, that new attempt might have generated better code or might have generated lot worse code. You can not draw conclusions from just one answer.

View on HN

That doesn't seem to be the case here. Reading through the article and twitter thread, the impression I get is that between moyix and the author, a decent amount of time was spent on this. A valid criticism that could have been made is the use of Claude Sonnet but based on the twitter thread, it looks like opus was what @moyix leveraged.

Yes – it's a bit hard to follow the various branches of the thread on Twitter (I wasn't really intending this to be more than a 30 minute "hey that's neat" kind of experiment, but people kept suggesting new and interesting things to try :)), but I gave Claude Opus three independent tries at creating a fuzzer for Toby's packet parser, and it consistently missed the fact that it needed to include the sequence number in the CRC calculation.

Once that oversight was pointed out, it did write a decent fuzzer that found the memory safety bugs in do_read and do_write. I also got it to fix those two bugs automatically (by providing it the ASAN output).

> A new attempt (with a new seed) is going to give a totally different answer

Totally different...I'd posit 5% different, and mostly in trivialities.

It's worth doing an experiment and prompting an LLM with a coding question twice, then seeing how different it is.

For, say, a K-Means clustering algorithm, you're absolutely correct. The initial state is _completely_ dependent on the choice of seed.

With LLMs, the initial state is your prompt + a seed. The prompt massively overwhelms the seed. Then, the nature of the model, predicting probabilities, then the nature of sampling, attempting to minimize surprise, means there's a powerful forcing function towards answers that share much in common. This is both in theory, and I think you'll see, in practice.

Depends on the question. If you asked for a small fact, you are going to get almost the same answer every time. But if it's not a factual question, and answer is supposed to be a long tangled one, then the answer is going to depend on what LLM said in the first lines because it is going to stick with that.

e.g LLM might have said for some reason the writing a fuzzer like this isn't possible and then went on presenting some alternatives for tge given task.

I have only experience with GPT-4 via api but I believe at core all these LLMs work the same way.

You're absolutely correct, in that it's never guaranteed what the next token is.

My pushback is limited to that the theoretical maximal degenerate behavior described in either of your comments is highly improbable in practice, with a lot of givens, such as reasonable parameters, reasonable model.

I.e. it will not

- give totally different answers due to seed changing.

- end up X% of the time, where X > 5 say it is impossible, and the other (100 - X)%, provide some solution.

I have integrated with GPT3.0/GPT3.5/GPT4 and revisions thereof via API, as well as Claude 2 and this week, Claude 3. I wrote a native inference solution that runs, among others, StableLM Zephyr 3B, Mistral 7B, and Mixtral 8x7B, and I wrote code that does inference, step by excruciating step, in a loop, on web via WASM, and via C++, tailored solutions for Android, iOS, macOS, Android, and Windows.

I still think it depends on the subject you are prompting. If LLM knows that thing very well it will stick to the answer, otherwise it can go in a different direction based on how different initial assessment was.

Yesterday I asked it to write a simple VB script to show a reminder that I will schedule via command line using Windows task schedular. In first attempt it suggested to create VB file for each message based on initial reasoning that I can not pass arguments to VB file like that. It didn't seem correct (confirmed via Google) then resubmitted the same prompt but this time it said that I can simply pass my reminder message as arg to VB script and the next code was based on that. (I don't know VB or ever used task schedular before)

This was GPT-4. You are not wrong about 'maximal degenerate behaviour' but initially generated assumption can lead to different answers overall. Chain of thought prompting stems from this exact behaviour.

Reply on news.ycombinator.com