Hacker News Reader

Hey HN, I made this simple GitHub tool to check how much vRAM you need to train or inference any LLM. It supports inference frameworks (huggingface/vLLM/llama.cpp) & quantization (GGML/bnb) I Made this after getting frustrated when I couldn't get 4bit 7b llama to work on my RTX 4090 24GB gpu even though the model is only 7GB.

View on HN

What would really be helpful here is adding information about how much inference you can do on a given GPU.

Since transformers & attention scale quadratically this is quite important.

Knowing that I can load X model on a given GPU is good, knowing that I can expect to be able to perform Y inference at N tok/sec on that GPU with however much concurrence.

Reply on news.ycombinator.com