> and $10,000+ of compute hardware per inference session.
What hardware would you need to run it at home?
What hardware would you need to run it at home?
Step 1: https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/blob/ma...
Step 2: https://github.com/ggerganov/llama.cpp
Step 3: you're welcome
That is not true. A common macbook with lots of RAM (>32GB) is enough. Or any x86 computer with lots of RAM. llama.cpp is CPU only and quite fast