Hacker News Reader

How fast are these small models on a 4090, is it like 100ms ? 500ms ?

Mistral-7B gives you 80 tokens/second on 4090. So this one will be faster...

It’s about twice the speed