GPT-4-turbo preliminary benchmark results on code-editing
aider.chat
aider.chat
I tried Claude because of the larger context size, but I've been disappointed so far. I find Claude much more likely to just compliment my writing, whereas GPT will identify strengths and areas that could be improved.
Anyone else seeing any evidence of this?
It's full of memes and people complaining its not as "good" as it was yesterday when it fails at completing their homework.
I would take anything said there with a big grain of salt, and stick to benchmarks.
"I benchmarked on SAT reading, which is a nice human reference for reasoning ability. Took 3 sections (67 questions) from an official 2008-2009 test (2400 scale) and got the following results, here a SAT-like test:
- GPT3.5 - 690 (10 wrong) - GPT4 - 770 (3 wrong) - GPT4-turbo (one section at time) - 740 (5 wrong) - GPT4-turbo (3 sections at once, 9K tokens) - 730 (6 wrong)"
Source: https://twitter.com/wangzjeff/status/1721934560919994823?t=P...
You seem to be suggesting it got a bit worse, and the aider article seems to suggest gpt4 got a bit worse, although much faster at being a bit worse, while gpt3.5 got worse, then better, while faster.
However, in my opinion the first attempt score is more important, and Turbo does genuinely seem to lead there. There's still a possibility the updated training data has tainted the results.
well, that's how it was for GPT-3.5 anyway. The "turbo" flavor was faster and cheaper, but seemed to have slightly worse output (again, this is all going by subjective measurement; it could entirely be the imagination of AI bros)
If you "have enough compute" available -- which OpenAI definitely does -- the best current technique is to use mixed precision with post-quantisation fine tuning to restore performance. That's most probably how all of the "turbo" models work. Take a model that was initially 16 or 32 bits per parameter during training, quantise it down to a mixture of 4, 8, and 16 bits, and then fix it up with an additional training pass that uses the original full-fat model's predictions as the loss function. With access to the raw parameters, it's possible to do this training such that all of the output weights are considered and adjusted during this phase instead of just the top word. Third parties fine-tuning against GPT4 chats can't do this, even with the collected samples, because they only have individual selected tokens/words instead of the full probability distribution.
It's like the AI equivalent of "the rest is left as an exercise to the reader" you'd find in old textbooks.
gpt4 should really be called DecartesGPT with this bs.
Are there any other programming assistant packages that use the chatgpt api like this?
Regarding rate limits, it might be an idea to have configurable delays built in to the testing code to prevent hitting limits.
Does it really make sense to play with this kind of power?
Better to be excited and learn the tool as it develops than to stick your head in the sand.
A world where AI has put literally everyone “out of a job,” meanwhile, is still so far from our current reality that IMO, it’s not worth making practical day-to-day decisions on unless you are directly involved in the development or regulation of AI.
2 years ago something like ChatGPT (as limited as it is) was “far from our current reality”.
I think it’s worthwhile to think ahead.
But two years ago, GPT-3 was already astounding. ChatGPT made some pretty huge gains, and GPT4 more still, but the thing that exists today was relatively predictable, compared to a world where human effort is no longer important.
Or do you think there will always be space for humans to provide value to other humans, even if machines surpass us in intelligence?
Even if AI can do everything, you'd still want the authentic human experience.
99% of people are far smarter than horses but people still pay to ride horses.
I don't see why someone wouldn't want to pay to... ride me... uh, in a matter of speaking, of course (of course). I mean, look at me.
How many working horses are there today vs before automobiles?
Heck, how many domestic horses are there today vs before automobiles?
Well, exactly. The less working horses there are, the more expensive and exclusive it would be to ride them.
There could be 1 trillion automobiles and I bet you, none of these automobiles would compare to riding a real, live horse.
Similarly, there could be 1 trillion AI robots, they could do everything better than a human, and yet I bet you'd still want to ride (or otherwise experience) a real, live human.
My point is that if automobiles were always better than horses in every way, then nobody would want horses. But even today, with the amazing automobiles that we have, some of which even faster and more reliable than most horses, it's clear that we still want horses.
My question is, if horses were as intelligent as us and they could have their basic needs met extremely cheaply, would they be willing to work at all, apart from the occasional ride? Because the horse labor pool would shrink immensely if they didn't really want to work.
So you expect there to be less of us?
Imagine everyone who is selling labor today out of a job.
> So you expect there to be less of us?
Less working people, not less people.
> Imagine everyone who is selling labor today out of a job.
If it's because they don't need it anymore, that's great!
What are the people not work going to eat?
We though AI would replace the low level jobs first, but it seems creative jobs are gone first (art, software developers, etc). Bear that in mind.
Frankly, if it's the higher end jobs getting replaced first that would likely spill over to the lower end ones as the those people who lost their jobs resort to taking lower end work to survive, flooding the market.
The shape of my career might change, but I doubt I'll be unable to find a job.
I think it's critical to be thinking about how to make sure wealth isn't simply funneled to a few capitalists who own everything simply by virtue of them being first, because it seems that's the future we're heading for if we aren't careful.
I think you, and I, and everyone on HN will be fine (more or less...) but I am worried about a wide swath of people who will get "left behind."
It would be pointless. Do you even know that 99% of horses can't speak English?
Anyway, my own experience tells me they're mostly just horsing around.
A lot more leisure and a lot less work than a few centuries ago...
Good times!
https://energyhistory.yale.edu/horse-and-mule-population-sta...
Clearly unsustainable, so I'm glad that problem solved itself. Phew! Can you imagine what it'd be like if they continued reproducing like a plague?
Anyway, the real question is: were they happier in 1910 when they were our slaves or are they happier now that they have more leisure?
How do you even measure happiness in a horse?
How do you measure happiness in humans?
Is happiness even real?
What is the meaning of life?
What does "42" mean?
In the past, new automation technologies often open up new possibilities in production capabilities in turn creating new jobs - specifically jobs that have not been automated yet.
AI though promises to be the universal automation, i.e. it can do any job. Thus even if new jobs show up, they will be taken over by AI too.
Then what?
> The shape of my career might change, but I doubt I'll be unable to find a job.
Question you should ask is why would anyone hire you when they can get AI to do the same job.
My guess is that some people will always be stubborn and prefer me over an AI giving them massages every now and then, even if I won't do them as perfectly.
But at that point in time, I think me giving one massage would perhaps be enough for me to retire for life, as long as I can abstain from buying luxury goods or services, i.e. those provided manually by other people (instead of AIs, whose labor would be practically free).
But why you? There are billions of other people who are out of work they could hire for the “authentic human experience”.
They already gave one massage, so now they can afford AI-made stuff for the rest of their lives, including AI massages for themselves.
AI labor is effectively free at that point, didn't you get the memo? :)
It’s not. Machines need power to run. Even if it was super cheap, there is no reason for their owners to share its output with anyone unless they have something to give in return. You know “trade”.
Did you forget about all the endless solar panels and wind turbines that the AI robots progressively built over the years, for an increasingly cheaper price? Energy is abundant now.
They also built a bunch of AI robot factories, so now we have effectively limitless AI robot labor. Everyone has AI robot assistants/friends now.
> Even if it was super cheap, there is no reason for their owners to share its output with anyone unless they have something to give in return. You know “trade”.
But they have something to give in return. These owners want authentic human experiences, which can only be provided by other people, not AI robots.
Authentic human experiences (or hand-built goods) are thus now much scarcer, since most people don't need to work anymore. They're sort of luxury goods/services. In this world, imagine how much you'd be willing to pay to have an exclusive, authentic house maid or hairdresser for one day, rather than just another AI robot, no matter how good it is.
Or imagine dining at a restaurant where every waiter and every cook are authentic humans, wow! An unforgettable (and prohibitively expensive) experience.
Of course, not everyone could afford that, unless they themselves are willing to provide such a service. Most people would eat at home (with food farmed, prepared/cooked and delivered by the AI robots and drones, of course), or just go to regular AI robot restaurants if they're feeling especially fancy.
There will also be an over supply of people willing to offer it because they have nothing else to offer in trade for basic necessities.
Rich people are notorious for seeking luxuries.
> There will also be an over supply of people willing to offer it because they have nothing else to offer in trade for basic necessities.
I don't know, perhaps you are right. I do hope you are wrong, though... :)
Also, I'm hoping if such an extreme situation would present itself, governments would be forced to step in and somehow ensure people would always have their basic needs met by, say, a strategic national AI army dedicated to provide those basic needs.
Speculating is fun, isn't it? lol
That would depend on who is your government.
An AI can't do any job that requires a body, so clearly that isn't true.
If you mean mental jobs, this could be true in the future, but right now, it's hard to say. You can imagine scenarios where an AI does 95% of a job but a human is still required for the last 5% (without which the first 95% is useless).
These tools are recent, but not THAT recent. GPT3 is nearly 4 years old, Copilot is 2 years old, ChatGPT is a year old, GPT4 is 8 months old. No large-scale employment changes resulting from AI are visible so far.
No reason they wouldn’t be provide physical bodies eventually.
This is the kind of scenario I was talking about in my initial comment. The naive assumption about this scenario is that it would result in 95% of jobs disappearing, but based on history it's more likely that we just do 2000% more stuff with the same amount of human effort.
And if AI really replaces 100% of human labor, are we really going to retain an economic system that only works when humans are doing labor?
Somehow that didn't happen, though.
Agriculture's share of the labor force went from something like 70% in the early 20th century to something like 5% now. You have a higher purpose than doing a robot's job badly -- or a farmer's. Find it.