DeepSeek V4 Flash with DwarfStar: https://github.com/antirez/ds4
The 2 bit quants are really good. I have a lot of memory so I can squeeze it all in at ~80gb.
The 2 bit quants are really good. I have a lot of memory so I can squeeze it all in at ~80gb.