carlosfm

Great and timely contribution! Will be following the space

I figured I'd tell you a little bit more about the project: In a previous life I was the lead AI researcher in a healthcare startup, and whilst my team and I loved the challenge (this was pre GPT craze), it was super frustrating that everytime we showed a prototype, it would take ages, if at all, to bring the model to the product, so it can be actually useful to the user.

My personal struggles were with access to hardware (GPUs, I'm looking at you) but also about the fragility of the entire process of putting LLMs into production; the industry is flourishing, and the toolsets, though awesome, are evolving rapidly. This meant what's supported today, won't be tomorrow. And the cost of switching to new libraries was too high.

Kalavai is my solution (literally, a solution made for me) to all of these. Use any hardware to build up an LLM pool, and get out of the box templates to plug and play components of the LLM stack without affecting anything else. Yes, it supports the usual model engines (llama.cpp, vLLM and Petals for now) and you can swap them out without affecting the API layer.

I'd love to see if this is useful to the community. We are targeting those that are struggling like I was. We just want to tinker with new models, not figuring out how to install CUDA on a VM to make pyTorch work.

I think like with any technology, it’s not so much “we need it” but “who has access to it”. Unless you consider genAI totally useless (which is a stretch) effective access is an impactful consideration, society-wise

I tend to agree, with two caveats: - leaving genAI to just a few, given that is demonstrably the biggest leap in performance around perception, is arguably bad enough - worse even when we consider the funding sink that genAI has in the field -sucking funding for almost anything else

Thanks for pointing it out! We've added Apache 2.0 license. Kalavai is free to use, for commercial and non commercial use!

I’ve been working for a while on Kalavai (https://kalavai.net), a project to make distributed AI easy without a devops certification. There are brilliant tools out there to help AI hobbyists and devs on the software layer (shout out to vLLM and llamacpp amongst many others!) but it’s a jungle out there when it comes to procuring and managing the necessary hardware resources and orchestrating them. This has always led me to compromise on the size of the models I end up using (quantized versions, smaller models) to save cost or to play within the limits of my rig.

Today I am happy to share the first public version of our Kalavai client (totally free, forever), a CLI that helps you build an AI cluster from your everyday devices. https://github.com/kalavai-net/kalavai-client

Our first use case is distributed LLM deployment, and we hope to expand this with the help of the community.

I’d love for people interested in AI at scale (bigger than a single machine) to give it a go and provide honest feedback.

Do you share our motivation? If you tried Kalavai, did you find it useful? What would you like it to do for you? What are your painpoints when it comes to using large LLMs?

Disclaimers: - I am the creator of Kalavai - This is my first post