Hacker News Reader

Comment by PaulHoule | Hacker News Reader

It's part of their seductive ability that makes people think the glass is half full instead of half empty, why some people seem to think ChatGPT-4 can program better than they can while scientific evaluation shows that it's like: writes a program 80% of the time, the program compiles 20% of the time, the program works 5% of the time.

I can only imagine that somebody with poor judgement like Elon Musk has to have some people around who tell them what they want to hear and suck millions and millions from them.

View on HN

>while scientific evaluation shows that it's like: writes a program 80% of the time, the program compiles 20% of the time, the program works 5% of the time.

Where was this "scientific evaluation" published, on what LLM, and what size programs did it concerne?

> scientific evaluation shows that it's like: writes a program 80% of the time, the program compiles 20% of the time, the program works 5% of the time.

Can you share the source for this?

There’s no source because he just made it up. Oh sorry, I meant that he hallucinated it. That’s the term we’re using now, isn’t it?

I could swear I saw this the other day but I couldn’t find it in YOShInOn, here is an older paper which gets results like this for C/C++ but does a lot better with R, see Figure 1

https://arxiv.org/pdf/2308.04477.pdf

The results of this kind of eval could be across the board, you could pick out a set of examples worse than what I said (Games, C++) or pick one out that is really good (Algorithms, Ruby)

There was this one also

https://arxiv.org/abs/2310.12357

which showed some pitfalls…. In some case the LLM could say which project the source code was from which meant it had seen it in the training data and the code ought not to be in the test data.

The glass is always full. Just depends on what you value. Here on earth, water is often more important than air...

Retest that hypothesis in orbit...

If you don't have enough reaction mass (for which water sometimes qualifies) to get out of orbit, having air but no water (rather than water but no air) just means you die three or four orders of magnitude more slowly. Maybe someone can lauch a rescue mission within a matter of days, but given how slow most space programs move these days, I'm not optimistic.

So, do we have a replication crisis?

Copilot is great for explaining a bit of code, or suggesting an approach especially with a language I'm less familiar with. But, if I committed what it made, my code would turn into unmaintainable spaghetti.

Reply on news.ycombinator.com