That's really fast. But it mostly seems to be because they made a custom chip.
I want to see an LLM that is so highly optimized that it runs at this speed on more normal hardware.
I'd pay quite a bit of money to have a Mixtral box at home, then we'd all have our own, local assistant/helper/partner/whatever. Basically, the plot of the movie Her.