Hacker News Reader

Comment by doctoboggan | Hacker News Reader

In the keynote @sama claimed GPT-4-turbo was superior to the older GPT-4. Have any benchmarks or other examples been shown? I am curious to see how much better it is, if it all. I remember when 3.5 got its turbo version there was some controversy on whether it was really better or not.

View on HN

My early benchmarking seems to show that it's somewhat better for coding.

https://aider.chat/docs/benchmarks-1106.html

I'm really surprised your benchmark shows gpt-3.5-turbo-0301 outperforming gpt-4 (non tubo) on first try coding problems.

One thing to notice is that the gpt-3.5 bars are blue and the gpt-4 bars are green. This is because aider uses different prompting strategies for GPT-3.5 and 4.

GPT-3.5 is only able to reliably edit a file by returning a whole new copy of the file with the edits included. This is the "whole" edit format.

GPT-4 is able to use a more efficient "diff" edit format, where it species blocks of code to search and replace.

All of this is described and quantified in more detail in the original aider benchmarking writeup:

https://aider.chat/docs/benchmarks.html

The original article benchmarked both models using both edit formats (and some others). And indeed, gpt-4/whole beats gpt-3.5/whole. But it's very slow and very expensive to ask gpt-4 to return a whole copy of any file that it edits. So it's just much more practical to use the gpt-4/diff, even though it performs a bit worse than gpt-4/whole.

Aider will let you do gpt-4/whole if you'd like to spend the time and money:

  aider --model gpt-4 --edit-format whole

Once OpenAI relaxes the rate limits, I will benchmark gpt-4-1106-preview/whole.

Its unclear to me if the 1106 is the same as the turbo model.

Ya, OpenAI's naming schemes don't seem very consistent. But my read of the announcement is that gpt-4-1106-preview is the new turbo model.

GPT-4 Turbo is available for all paying developers to try by passing gpt-4-1106-preview in the API and we plan to release the stable production-ready model in the coming weeks.

https://openai.com/blog/new-models-and-developer-products-an...

Ok great thank you for pointing that out.

It definitely feels worse to me. In the way that GPT3.5 felt worse than GPT4 in the past. Somewhere between 3.5 and 4. Similar to with the Bing plugin activated. Somehow "shallower" and does not seem to grasp my intent as good as before.

It seems like the "Turbo" models are more about being faster/cheaper, not so much about being better. Kinda similar to the iPhone "S" models or Intel's "tick-tock"

Reply on news.ycombinator.com