GPT4 vs. GPT-4-0613: Worse reasoning, but improves if you use the function API

twitter.com

9 points·reissbaker··8 comments

There's been a bunch of discussion of gpt-4 "getting worse"; I tried gpt-4 vs gpt-4-0613 (the latest release with function calling), and noticed that while the latest release is much faster, its generalized instruction following capability indeed declined compared to the original... But you can get it to regain some lost reasoning capability by using the new function call API. Kinda weird!

We desperately need more advances on open models. Having this space controlled by corps that decide what we can or can not use of the SOTA is very frustrating

Yeap. We noticed some issues with the new version as well. We went back to the previous model.

ask gpt the same question three times, you'll get three different answers.

a 'correct' answer one day and an 'incorrect' answer later demonstrates nothing.

Zero the temperature if you want the same answer every time.

I redid the test several times! GPT4 consistently got it right, and gpt-0613 got it wrong every time.

Anecdotally gpt-0613 has generally been "worse" for me on non-function-calls (but equally-ish good and much faster on function calls), but I wanted a consistent test for it. Feel free to try and duplicate on the Playground though!

it's just a question of temperature that's a choice

Really wish they didn’t block the population out of more real value

Reply on news.ycombinator.com