varunkmohan

They did not, Microsoft made the bet that there would be some application in the future that would require distribution to all developers. Turns out that application was Github Copilot.

Hi from the Codeium team. It's awesome to hear you are allowing other code LLMs to be used on the Replit platform (we're big fans)! We'd love to enable our free chrome extension on Replit.

We explicitly state in our privacy policy that we don't sell user data to third parties.

The language server is the common binary cross ides that actually processes local / repo context and communicates with the cloud service to generate completions. For now, that is closed source, similar to other tools like Tabnine and Copilot, but we may change that in the future.

Hi, Varun from the Codeium team here. This is fairly common thing to do to not support these sorts of domains to prevent abuse. Also, Copilot downloads a language server binary as well for their vim extension.

Happy to discuss how we could improve.

Sad, this whole thing where companies train chinchilla optimal models has to stop - I'm always happy to get open source models but it would be great if it was useful rather than a benchmark.

Exactly why our product Codeium (Copilot alternative) supports self-hosting.

OpenAI has done the reasonable thing of not exposing the probability distribution per generated token so it's very hard to use that to completely map to their models. Ultimately, you still do need a very large base model to compete.

This looks very cool and is a massive improvement. If only OpenAI would also publish what they are doing to make 32K work.

Curious if anyone has actually used this. It's quite slow for me and feels more like a cute idea rather than a useful product.

Thanks for the feedback. Currently there is not, but we will look into setting up custom keybindings or some other approach.

It isn't zero, that's for sure! Before Codeium, we as a team were building scalable ML infrastructure for some of the world's largest ML workloads, so we have a lot of experience in building infra (especially ML serving infra) that optimizes computation on GPUs/mixed compute resources to drastically reduce serving costs. We will probably talk more about these infra side optimizations in future blog posts!

I don't really want to argue but we have been pretty open about the fact that we want to keep this product free for users going forward as well. We have been pretty clear about our plan to monetize additional features but the focus now is to democratize this technology.

We are a free product.

We are actively iterating on our data cleaning to deanonymize any user information. Of course some cases are tricky to find, but we recognize these issues are serious and will continue to work here. All of our training data is public, so we know that we won't ever produce private user info.

Elixir is not a language we are currently confident in, but in the very near future (O(weeks)), we believe we will have a model that is better than Copilot for Elixir and other niche languages.

We don't rely on any 3rd party APIs in order to keep serving costs down (which is necessary to keep this as a free service). We only use permissively licensed public code to train.

No the model actually runs remotely. We would love to have some way to run a lightweight model on the user's machines but it's just not feasible given model size and amount of computation required.

Sorry to hear that. Point noted, we will take a look - most of our testing has been on chrome / safari.

This is one of the next IDEs we will be supporting! We're keeping folks on our discord (https://discord.com/invite/3XFf78nAx5) updated on this.

Yes, good point. Interestingly, our VSCode extension also works on VSCodium since we uploaded it to the OpenVSX registry!

Would be curious to see your benchmarks. Btw, Nvidia will be providing support for fp8 in a future release of CUDA - https://github.com/NVIDIA/TransformerEngine/issues/15

I think TMA may not matter as much for consumer cards given the disproportionate amount of fp32 / int32 compute that they have.

Would be interesting to see how close to theoretical folks are able to get once CUDA support comes through.

I'm not sure any of this is accurate. 8 bit inference on a 4090 can do 660 Tflops and on an H100 can do 2 Pflops. Not to mention, there is no native support for FP8 (which are significantly better for deep learning) on existing CPUs.

The memory on a 4090 can serve extremely large models. Currently, int4 is started to become proven out. With 24GB of memory, you can serve 40 billion parameter models. That coupled with the fact that GPU memory bandwidth is significantly higher than CPU memory bandwidth means that CPUs should rarely ever be cheaper / lower latency than GPUs.

Find it hard to believe developers would pay $100/month unless the product embeds deeply into their workflow. I do think it's an interesting idea to place some limitations and still have a free tier.

I think it's likely the other way around, a product like Copilot is actually much cheaper than if you were to build off of OpenAI's API. ChatGPT, the product, will be much cheaper than if you actually ran the API yourself.

Some context is that I work for a company that builds a product called Codeium, that's a free Copilot alternative. If we were to use OpenAI's API, we computed our costs would be high 100s of dollars a year per user (we've opted for something more custom / in house).

The model runs on a remote GPU as most users don't have powerful enough GPUs to run run the model with reasonable latency.

Posted this on a comment above but systems like Whatsapp likely sent an insane amount of data as well but used only 16 servers over 1.5 billion users at time of acquisition. Modern NICs can handle millions of requests a second - I still feel there is a lot of excess here.

Genuinely asking, why do you think Twitter needs 24 million vcpus to run?

This is not apples to apples but Whatsapp is a product that entirely ran on 16 servers at the time of acquisition (1.5 billion users). It really begs the question why Twitter uses so much compute if there are companies that have operated significantly more efficiently. Twitter was unprofitable during acquisition and spent around half their revenue on compute, maybe some of these features were not really necessary (but were just burning money)?

Good analysis. Obviously, this doesn't handle cases like redundancy and doesn't handle some of other critical workloads the company has. However, it does show how much real compute bloat these companies actually have - https://twitter.com/petrillic/status/1593686223717269504 where they use 24 million vcpus and spend 300 million a month on cloud.

This feels like what a company that hasn't accepted the future would say. Yes, ChatGPT doesn't do exactly what Google does. Can it be augmented by well understood search retrieval engines to generate a much better response? I think for a lot of cases that seems to be undoubtedly true.

Any plans to be able to directly run this in the browser similar to DuckDB. Nice to see more options in the space.