Hacker News Reader

Forgive the ignorance, but how are you running a 763B parameter model on 64GB of RAM?

SSD weights streaming. The actual weights is 552B, then engram.

which engine?

By “quantitizing” it

The smallest quant is still over 100GB, so I don't know how that fits in 64GB RAM. It would have to be partly streamed from disk.