> If you can do 8bit inference, don't use a GPU. CPU will be much cheaper and potentially also lower latency.
Good advice. Does that mean that I can install like 64 gb ram on a PC and run those models in comparable time?
Good advice. Does that mean that I can install like 64 gb ram on a PC and run those models in comparable time?
And for a datacenter, a few $100 AMD CPUs will beat a single $20k NVIDIA A100 at throughput per dollar.