Show HN: Can your GPU run this LLM?

github.com

15 pointsALTRC03204 comments

Hey HN, I made this simple GitHub tool to check how much vRAM you need to train or inference any LLM. It supports inference frameworks (huggingface/vLLM/llama.cpp) & quantization (GGML/bnb) I Made this after getting frustrated when I couldn't get 4bit 7b llama to work on my RTX 4090 24GB gpu even though the model is only 7GB.

What would really be helpful here is adding information about how much inference you can do on a given GPU.

Since transformers & attention scale quadratically this is quite important.

Knowing that I can load X model on a given GPU is good, knowing that I can expect to be able to perform Y inference at N tok/sec on that GPU with however much concurrence.

It's so over, I cant even run the lowest LLama model; It never began for iGPUcels https://ibb.co/jy6B0sf

I was actually looking for something that could test what LLM I could run, thanks a lot haha

just nvidiamaxxx bro.

Reply on news.ycombinator.com