varunkmohan

Posts like these are always awesome to look at how much we can push consumer hardware.

It's hard not to really appreciate some of the devices we have today. For instance, an RTX 4090 is capable of 660 TFlops of FP8 (MSRP 1600). Would not be surprised if we soon have laptops that can do petaflops of computation soon!

Wow, very smooth to use! Would be extra neat if this could support plugins as well.

We're building out a free AI code completion tool, Codeium that will support Neovim as an IDE and would love to hook into this as well.

Interesting, I guess we implemented this op properly :)

It is strange that cub does not guarantee determinism here - it's not too hard an op to implement in CUDA.

It's very strange that they don't guarantee determinism. We trained a model at Codeium that does have determinism at inference time - it's not super hard if you don't just use random seeds everywhere.

This makes it a lot harder to rely on OpenAI's API if you are doing any sort of caching / evaluation workloads.

Really thorough post! It seems hard to prevent these prompt injections without some RLHF / finetuning to explicitly prevent this behavior. This might be quite challenging given that even ChatGPT suffered from prompt injections.

There are a bunch of really good ideas used to train this model - multi query attention, infilling, near deduplication and dataset cleaning.

I do wish that the demo was a little more interactive (not needing to click buttons to create a generation) since it makes it hard to see the full power of the model.

One of the things we tried at Codeium for our playground on browser was to make it super clear how well the model performs by making the experience interactive - https://www.codeium.com/playground

That's not the issue here, it's just saying any odd number is prime, which is false

The magical number for performance is actually memory bandwidth which is actually lower for TPUs compared to A100s. They have more aggregate compute, but it's not trivial to use that to get very low latency on a per request basis.

The neovim plugin mostly actually communicates with a node-js service seen here (https://github.com/github/copilot.vim/tree/release/copilot/d...). This is why they require you to install node for using the plugin and allows them to share logic with the vscode extension (also in javascript). I think all the features should be available even in neovim.

Very true, I think the issue though is unless that output is very likely to be 100% correct, a user would always prefer something that is incomplete but quicker to iterate on. It would be interesting to see if we can get to a paradigm like that.

Curious, what functionality did you find the most useful? Sometimes on edits, I find it adding more than expected (potentially entirely new files) which causes me to not accept the suggestion. Explain does work well sometimes though!

This is a pretty cool idea even for just engineering the prompt! It's a complicated tradeoff to decide what should go into the context and what should be selected from other files (2000 tokens is a lot but sometimes not long enough for the longest files). Previous cursor location is a great signal directly from the user, compared to metrics like Jaccard similarity. I'd actually like to try this out for our next release of Codeium.

Most likely latency and cost reasons. A model that's 10x as big requires 10x the hardware to serve at the same latency. Since most generations are not too long, a smaller finetuned model should work well enough.

A 1T model would be capable of much more than what the current version of Copilot in terms of autocompletion and even code correction. However, at that point, even with a lot of model parallelism to speedup inference, it's likely to be atleast 10x slower on the generation side. From my experience working on Codeium, a Copilot alternative, this would be too frustrating for users. It could be useful as a tool that runs asynchronously that modifies all your code at scale.

It's an embedding model so it generates vector embeddings not text generations. That's to be expected.

Yes, that's on the list of plugins to create. We're keeping folks updated on the timelines on our Discord.

We are currently working on making suggestions more tailored to your codebase without having to persist / upload the codebase.

We will have updates on this shortly!

Interesting - would love if you could join our Discord (https://discord.gg/3XFf78nAx5) and we are happy to help figure out what's going on!

Thanks for the feedback!

Please join our Discord (https://discord.gg/3XFf78nAx5) and we are happy to help debug this for you!

Thanks for the support! We are actively looking to hire experts in program synthesis and ML infrastructure to help deliver this technology at scale

We have gotten requests for more IDEs, and are actively working on creating support for them so that as many developers as possible can leverage this technology!

We're actively thinking of ways where we host the service within a customer's tenant - more coming soon!

We understand the concern! We describe our security policies in detail on our website, but we offer users the option to opt out from code snippet telemetry. This means after the inference, we do not store any code-based information on our end. We also know that you might not take our word for it, which is why we are working on getting SOC2 attestation for these policies.

Also, to answer your pricing point, we are actively working on additional features beyond code completion. We want to try to make the code completion itself to be free, and definitely free forever for early users as a thank you for using Codeium from early on.

Thanks! We've made a lot of optimizations internally for latency. We've gotten a lot of feedback that it is faster than Copilot but obviously they operate at much larger scale.

Right now, we are providing a free alternative to Copilot and expanding the set of features we support. We are actively working on improving the quality of suggestions and would appreciate feedback.

We guarantee that all users that join now will be able to use the product free forever. We explicitly allow users to opt out of sending data to us post inference. We believe we're in the early innings of code assistants and would like to democratize the technology so we don't have plans to monetize this tier of usage.

The model is hosted internally and trained on public code. For more details on the specifics, please take a look at our FAQ page on the website.

Completely agreed. For some of these large language models, it would take a long time before inference spend dominates training spend.

I think you'd probably always want to go with T4's since they are the same price unless there's just no availability for them.

Yeah, we should have a public release very soon for people to deploy internally. We will have support for all the commonly-used frameworks and different versions.

Yup, exactly. It's a good point that for self-supervised workloads, the training set can become arbitrarily large. For a lot of other workloads in the vision space, most data needs to be labeled to be able to used for training.