Phind (code beating GPT4) seems to have used WizardLM's finetuned checkpoint | Hacker News Reader

Phind (code beating GPT4) seems to have used WizardLM's finetuned checkpoint

twitter.com

73 pointsankit21952 comments

Phind's reply: https://twitter.com/phindsearch/status/1696715973007671529?t...

Why is this reply written like a teenager was sending a text message?

To preempt this drama, Phind claims that they were inspired by WizardLM's technique of generating datasets using GPT-4, but didn't use their model or data.

Sounds like parallel construction here.

They did not beat GPT-4. They beat an old measurement. It's still like 16% behind.

Would be happy to hear otherwise.

More specifically, they've beaten an old measurement on one specific benchmark. In my personal experience however GPT-4 is still ways ahead CodeLlama and its derivatives. They are really good for what they are, but you just can't beat the parameter count GPT-4 has and in more complex tasks it does show.

Edit: An example just from a while ago (I know it's not a strictly programming question, but still, c'mon):

>>> Is 10000 bigger than 10050?

> Yes, 10000 is bigger than 10050.

Edit 2: I just asked it to write a simple Python app to render a rotating 3d cube. This is something GPT-4 did almost flawlessly few months ago for me. I used the exact same prompt. I got a program that not only didn't work, but had syntax errors (mismatched parenthesis), referenced variables before assigning them, etc.

Why are people constantly using examples with number comparisons and calculations? That's the one use-case I'd never use an LLM for.

It’s like how people keep discovering that LLMs are bad at letter substitution ciphers. Yes, they are. Because you’re asking it to reason over the inputs and outputs of the embedding model, which inputs it never sees.

Because GPT-4 gets this right. Also please look at my 2nd edit which is a concrete programming example that Phind's model gets wrong on several levels.

> Because GPT-4 gets this right

And a Threadripper CPU can render Crysis, but that doesn't make it a meaningful metric for other processors. Maybe GPT4 happens to get it right in some cases, but in practice it's one of the most inefficient and unreliable ways to do calculations on a computer and is not an intended feature.

Arithmetic logic is *very much* a sought after goal for LLM.

The “intended features” of any sufficiently advanced technology always move forward.

GPT, Bard, etc all want to nail that issue and are dumping cash into it.

Mismatched parenthesis, specifically using )) instead of ) is a common failure mode I observed in all of my codellama testing.

I get the repeating close parenthesis a lot too. Figured it was due to my temperature settings, prompt style or quantization though, but this was my first time running one locally

Yup. I use GPT for unit test generation and Wizard Coder is nowhere near even GPT-3.5 let alone GPT-4. HumanEval performance is not representative of general model capability.

Thanks, that's really all I needed to know. Glad they are trying and no doubt someone will but you have to believe OpenAI has a response in waiting when that happens and they feel the need to deploy it.

You are right, they didn't. The reason I said that was because of this post: https://news.ycombinator.com/item?id=37267597 and how people on twitter were talking about it. The goal was for easier recall of the earlier post as this discussion is in continuation of that. The title says that and so does the blog post.

Yes, Paul Graham posted it without questioning, and that's how many people came across it. I attempted to correct him, but I noticed it late (https://twitter.com/LechMazur/status/1695401777863000425).

And what if they used it? Does the license say they can’t? Even if they didn’t credit them, does that violate the license? And if they used it, what’s the problem if they acknowledged it?! It sounds like high school drama, just a little bit of accountability please.

Their problem is not that Phind seemed to have used their work but that they deny doing it and refuse to credit them, and going so far to say WizardLM uses multiple accounts to discredit them.

You sure have a lot of questions. Yes, it violates the license to use it without attribution.

They didn't acknowledge it though - that's the problem. That's not high school drama - that'll get you expelled from university for plagarism or lose you your job at a research institution. There's nothing wrong with using their work - wizard is encouraging them to use it! But you MUST attribute. If you don't, that's plagarism.

WizardCoder is fantastic. I use it as my local "code assistant" LLM[1].

I'd really like to hear from the Phind team about this. Using someone else's work without attribution isn't cool (assuming this thread is true).

------

[1] In fact, I put out a set of instructions on how to get it to work with VS Code (with the help of Continue + llama.cpp) a couple of days ago: https://news.ycombinator.com/item?id=37296728

Can you share more about the usage? Was it really useful, in what cases? Would you recommend it over gpt-4 or codepilot?

Sure. So, my startup[1] does data processing for clients in aviation and finance. Due to security, legal, and regulatory requirements, sending our code on to a third party (e.g., OpenAI) is a no-go. Thanks to CodeLlama (the WizardCoder fine-tune), I've finally been able to get AI assistance with my code.

My fundamental programming skills are solid (30+ years programming!), but I only started using Python extensively relatively recently. I can get the language to do what I want it to do—but often not in the most "Pythonic" way. With CodeLlama, I'm able to check if I've done things in the "right" way, and I'm also able to ask questions when I'm trying to figure out how to implement something.

As mentioned above, I cannot use GPT-4 for my startup, but I have used it for personal stuff. My feel so far is that wizardcoder-python-34b is surprisingly good. I certainly haven't thought "oh, I wish I had GPT-4 instead"—at least, not yet! (I haven't used Copilot either, for the same reasons listed above.)

--------

[1] Shameless plug: https://www.amw.ai

Humorous observation: amw.ai sounds like an LLM MLM. ... "Can I be an Amwai distributor?"

Isn't using someone else's work without attribution the whole point of (modern) AI?

I don't get any attribution for the code or messages I wrote that GPT4 was trained on, and there are multiple examples of LLMs regurgitating people's code (even copyrighted code) without attribution.

https://news.ycombinator.com/item?id=33226515

Excellent point.

The training dataset is just upstream of the fine-tuning step but both are supplying vital data to the whole pipeline.

thieves are never rogues amongst themselves

Hahaha you beat me to it.

Still rude though. Rude on rude.

Well by that standard how many of us can say we learned to code without seeing anyone else’s code? And how many of us “regurgitate” copyrighted code? I don’t check my own output, and across decades of writing, odds are good I’ve produced verbatim copies of some copyrighted code.

Point being, there’s a legit conversation to be had about AI training data, but it’s not the same conversation as humans intentionally creating a derivative work. I think it’s too reductive to insist those are the same thing.

The link I provided shows copilot regurgitating a 21 line function pretty much exactly. The comments are changed from //comment style to /*comment*/ style and some of the variables have been renamed, but thats about it. I can say with pretty high confidence I have never plagarised anything to that level, and I think the same is true of most human programmers.

As a university lecturer you don’t get attribution for everything your students do after they graduate.

I fail to see how anyone with your viewpoint isn’t flat out being intellectually dishonest. It is understood that it is both (usually) legal and ethical to not attribute someone when they teach someone an abstract skill set which they use to develop something “new”. LLMs inarguably bring us closer to reaching that point without implicating human effort in the creation process. “You did it with a computer, so copyright!”is such a naive and frankly wasteful take as it threatens to forego massive societal advances in favour of intellectual property rights. What sort of society is that!? An overly American one, that’s for sure.

This isn't a student learning from a teacher, its a student copy-and-pasting an answer they found. It isn't learning an abstract skill set which it is using to develop something new, its literally just copying the code.

It is utterly dishonest to make the comparison you do in your comment.

Is there a service that provides it?

This is neat! I'm going to give it a shot!

AI is the end of copyright. Hopefully such accusations like "X stole Y" will end in the future and we can appreciate a society where we all have unlimited access to knowledge.

With no copyright and intellectual property knowledge wouldn't be free, it simply would cease to be generated. If you can't profit (at least modestly) from your work, there's no incentive to perform that work.

Lots of things sound good in a perfect world, but we don't live in one.

Music composers existed before music sheets were copyrightable - and that was what copyright was invented for...

Without copyright, some business models would have to change, but to say that knowledge "would not be generated" is simply false.

I was responding to an obvious exaggeration in kind, which was a bad move on my part.

Yes people can and will still continue to do stuff for the sake of it. Some of it will be amazing. The huge things like building the next 5 CPU generations though, is that going to be practical? Who will fund that, and why?

Trying not to get political but the end of copyright feels like communism to me. No private property and all that. While I didn't live through it myself, the associated trauma WRT the USSR certainly left some deep scars that people still haven't healed from.

In any case, if -- and it's a big if -- AI continues to develop at the current rate, a solution like UBI will have to be seriously considered. Economies probably don't function when people don't create value, earn, or spend.

There is some confusion between copyright and patents here. Copyright forbids transmission of knowledge; patents forbid manufacturing. The incentives to manufacture CPUs would still be there if copyright were to be abolished, because patents would still work.

What AI demolishes is the pretense that human art is rare and precious, a scarce resource to channel through forced rivers in order to make money.

What machine-learning will probably result in, is a U-turn from the "knowledge economy", based on gatekeeping information, back into manufacturing-oriented systems, based on actually making stuff.

I bet you didn't even get paid for writing this comment.

I would contribute to a lot of open source projects if it took a few minutes at a time. Working full-time hours over many years is a different question however.

> With no copyright and intellectual property knowledge wouldn't be free, it simply would cease to be generated. If you can't profit (at least modestly) from your work, there's no incentive to perform that work.

A certain group of people would stop creating content while a new bigger cohort would start, I'm sure it would be a societal gain.

See my other comment for a more expanded take, but let's go with Intel for example. If they had to do all work out in the open, and share all progress they make, all they could compete on would be timing and pricing.

Pricing isn't going to be something they are likely to win on, and the timing is either time to market, or time to release IP. For the former, why buy i9 for $1000 when you can be an Indian alternative for $500 in, let's say, 6 months? For the latter, if companies can release findings after some fuzzy period like "when the prototypes have been validated," that's still IP but with a timed secrecy.

Isn’t the end of copyright also the end of IP? Then what?

Cool, I’m going to walk in your house later and enjoy unlimited access to your very enlightened refrigerator.

Breaking into someone's home and taking food they paid for doesn't seem equivalent to copying someone's idea.

If we were talking about someone breaking into an office and stealing whatever they were working on then your comment would make more sense.

So if the person was depending on selling the idea to pay for their food would you find that equivalent enough to engage with or should I try again?

We'll see where the dust settles but I'm kind of shocked that so many people seem to think that using wizardlm without attribution would be fine if they did. Before getting to that though - seriously why are people fighting over twitter? In public? What is this nonsense? Just email them privately.

If they did use their work without citation - that's plagarism. That'll get you fired. Your career will be over. If you pull that sort of stunt on a publication and you're caught, maybe you'll get away with it. But many places have very strict zero tolerance policies for that sort of academic dishonestly and will expel/fire you. You're more likely to end up with your life ruined for plagarism and dishonesty like this than fraudulent data because you can claim bad data is a mistake, while if you actively take credit for someone else's work, yeesh. Don't mess with plagarism. Cite your sources and give credit for ideas. Scientific fraud doesn't have a clear tangible 'victim' to center the story around but plagarism does and the person you ripped off is motivated to achieve justice. Like stealing from the poor vs the rich.

I think it's only a matter of time before people start adding trap street[1] like text to their training data to catch plagiarists who fine tune models on their checkpoints without crediting the original work.

[1]: https://en.wikipedia.org/wiki/Trap_street

This is a pretty fun food for thought issue. Does phind need to credit wizardLM for this? While it’s totally possibly they used the EvolveInstruct, the output is still from GPT-4 which never has a chain of attribution to begin with.

Maybe I’m reading this wrong, but the title suggests phind used their checkpoint but didn’t seem that way? They used their says generation technique.

This is serious.

But damn the emojis and text formatting makes it cute and hilarious.

I hope they settle it out, both are great help to the industry.

Reply on news.ycombinator.com