Hacker News Reader

Comment by doctoboggan | Hacker News Reader

I know there seem to be a few different models to choose from, does anyone have a resource that collects them and shows the strengths and weaknesses of the various options? I’d be particularly interested in models optimized for apple silicon.

Also, how do they compare to copilot?

View on HN

Unfortunately, Copilot is a lot more capable. Most important is that it works with many more languages out of the box, is continuously updated, has more mathematical plus scientific knowledge and is better at understanding your comments.

As for which to use, you'd want to take the largest model that can you can fine-tune (not necessarily on your machine) and fits comfortably in your machine. The small models aren't as flexible and are basically just a cleverer autocomplete. Copilot is a lot smarter than that and smarter than all models I've tested up to 6B. It can often cleverly work out a lot of local patterns for what you're doing and this is true surprisingly often for novel languages, math, understanding graph notation, categorization policies and much, much more.

I'd gone in hoping I could remove my dependence on Copilot but left disappointed. The small models up to 6B just don't compare. I don't have the resources to run the 13B model which is probably behind anyways due to not having trained as long on as much as copilot (which is continuously updated). I also suspect Copilot has a great deal of hand-coded hacks and caching tricks to improve the user experience.

Yep, I definitely agree that the 6B and below models are worse than Copilot. The 16B ones are pretty good! But IMO still undertrained compared to Copilot, and of course much less accessible (though see elsewhere in this comment section; I think INT8 and even INT4 should be doable – this won't help much with inference latency but it should help most people fit the 16B model locally).

I have high hopes for the BigCode project too; they will get to take advantage of a lot of things that have been learned about how to train code models effectively.

One last note – I think that many of the improvements to Copilot since its release are attributable to "prompt engineering" and client-side smarts about what context should be included; I'm not certain, but I can believe the underlying Codex model used hasn't changed much (if at all) since code-davinci-002.

>I'd gone in hoping I could remove my dependence on Copilot

You already depend on Copilot?

Reply on news.ycombinator.com