Hacker News Reader

Comment by UncleOxidant | Hacker News Reader

I just gave Bard a try. It's still nowhere near as good at coding as chatGPT (and GPT4 specifically). I asked it to write a Python program to solve the Traveling Salesman Problem using ACO. The code GPT4 gave me for this prompt runs out of the box which is pretty impressive. The Bard code, on the other hand, has errors like forgetting to pass an argument to a function, or calling a method on a class that hasn't been defined on that class. When I point out the error to Bard it presents code that it says solves the problem, but often doesn't. In some cases when it fixes a problem in one area it reintroduces a previously fixed bug elsewhere.

Also, chatGPT always goes the extra mile and includes instructions on compiling/running the code it provides (including sample data). Bard does not.

Bard still has a ways to go to catch up.

View on HN

I've been using it extensively and never once found ChatGPT to produce working code when the ask is just above basic. It might have to do with subject matter. It usually is close enough but my experience has been what you've found with Bard - missing arguments, phantom methods, etc.

Haven't bothered with Bard at all.

did you try with GPT4? I was able to run one GPT4 query/day for free on poe.com

Yeah, given the random nature of these LLMs it's possible that I won the lottery in that particular case.

"But did you try with GPT4?" is becoming a bit of a refrain for me when talking about this with people. I can get GPT-4 to produce working code from agile user stories surrounded by a well-crafted prompt.

In my experience, GPT-4 does get lots of textbook/common problems right in common languages on first try. But when I tried to solve a just mildly complex problem (finding a ring of cells in a Godot GridMap using GDScript), it produced nonsense: Solved the problem wrong, called functions that don't exist and so on. It did correct some of the mistakes after I pointed them out, but I couldn't get something working without manual editing. Then I asked it to optimise the solution and got nonsense again. For simple data wrangling in Python it worked a lot better.

I have a suspicion there might be some... showmanship at work where all kinds of test problems and common tasks are made sure to work well.

Also it generates code slower than I would type (if I know what to write), which makes me mad for some reason.

I think it has a lot to do with how common code examples in the language are. I bet if you tried with languages detailed here you will get a lot better results than more obscure languages (like GDscript)

https://octoverse.github.com/2022/top-programming-languages

> I just gave Bard a try. It's still nowhere near as good at coding as chatGPT (and GPT4 specifically)

Because its a much smaller model IMO. Its like comparing a car with 2.0 engine, with a car with 3.0 engine. Also ChatGPT made a lot of errors in code just a few months ago. Google seems to be catching up.

Worth noting that Google i/o is next month and many are expecting announcements related to Bard. There is that rumor floating around that they are training a very large (30T) model

The only source I can find is a reddit post from April fool's day, do you have a better one by chance?

There's almost no way they make a 30T model that's useful at least? That's over 60TB of memory at 16bit, 15TB at 4 bit. That's 15TB that have to go through either VRAM or TPU Memory (TPUv4 is 32GB I think?). Yes you could distribute it and chunk it, but that would be insanely slow/expensive to run for anything more then a research demo.

GPT-4 is most likely 1~2T from what I've heard, and they've been struggling to run that profitably/quickly judging by the crazy pricing and 25messages/3hours if you pay for premium chatgpt.

EDIT: The only way this is possible is if they've pulled off some crazy optimization, which is definitely possible, but a lot of the research from them such as their own Chinchilla scaling suggest smaller models that are more fully trained are better.

According to their chinchilla paper they would need over 330TB of training data for that, and an absurd amount of training time.

See my other comment: https://news.ycombinator.com/item?id=35657567

I've found that GPT4 does very well for problems that have examples on the web, but I have yet to make it display a 3d cube using opengl (any version) in linux using c++ or python.

So it depends heavily on the task and how much it has stored from the training.

Now if you ask them to do a teapot, it sometimes do it since there are so many examples of that, even if you often have to tinker a bit to get it to compile and run, at least it is a teapot and not some weird random geometrical object.

I have yet to try it with bard, will test when I get home to a proper computer.

Reply on news.ycombinator.com