Hacker News Reader

Amazingly enough, Hacker News decided to show this to me as the top comment, I'm also running an rtx 5090 and trying it out now, thanks for the tip!

Edit: Absolutely blazing fast! Getting 163 tokens/sec on WSL and it generated a pretty sweet Pelican.

Amazing. I'll give it a shot on my 5090. I already tried using vLLM but it ran out of GPU memory. I guess it's likely Llama.cpp will work.

Just enable MTP on llama.cpp and you will get the same decode speeds.

It there anything similar for RTX 3090 and RTX 4090?

I'm not sure about a 4090 but there is a fork for 3090s: https://github.com/Don-Chad/ninfer-3090

I'm running it using a 4090 on using llama.cpp with Q5_K_S and its running at ~33 t/s

Yep, same, testing it now and it flies!