Hey HN, I made this simple GitHub tool to check how much vRAM you need to train or inference any LLM. It supports inference frameworks (huggingface/vLLM/llama.cpp) & quantization (GGML/bnb)
I Made this after getting frustrated when I couldn't get 4bit 7b llama to work on my RTX 4090 24GB gpu even though the model is only 7GB.