Hacker News Reader

Comment by 37ef_ced3 | Hacker News Reader

AVX-512 neural net inference on inexpensive, CPU-only cloud compute instances: https://NN-512.com

An AVX-512 Skylake-X cloud compute instance costs $10 per CPU-core per month at Vultr (https://www.vultr.com/products/cloud-compute/), and you can do about 18 DenseNet121 inferences per CPU-core per second (in series, not batched) using tools like NN-512

GPU cloud compute is almost unbelievably expensive. Even Linode charges $1000 per month, or $1.50 per hour (look at the GPU plans: https://www.linode.com/pricing/#row--compute). It's really hard to keep that GPU saturated, which is what you need to do to get your money's worth

As AVX-512 becomes better supported by Intel and AMD chips, it becomes more attractive as an alternative to expensive GPU instances for workloads with small amounts of inference mixed with other computation

View on HN

What does this have to do with https://gpu.land/ ? I've been very impressed with NN-512 in your past postings but I'm failing to see the direct salience w.r.t the top level post.

Perhaps readers would benefit from an apples-to-apples comparison of V100 to some CPU in a training per dollar metric? Preferably using something like MLPerf. You do mention inference but I think most people looking at https://gpu.land/ will be far more interested in training rather than inference.

I think the most direct competitor to this ShowHN would be https://vast.ai/

You are correct that vast.ai is pretty close.

The biggest difference is probably security / guaranteed uptime. With vast you're getting what it says on the tin - a machine from a marketplace. Could come from anyone / anywhere. No idea what else is running on it. Ours are hosted in a professional DC, managed and secured as they should be.

If anyone's curious, there's a detailed comparison page with other platforms here - https://gpu.land/versus

I think Linode/Paperspace/AWS/GCP/etc. are closer to "direct competitors". I would be much more comfortable trusting a single entity that owns its servers than a GPU rental marketplace like vast.ai.

The suggestion is that you may not want to use a GPU if you can use AVX-512 CPUs for your machine learning workload. Reducing the cost of the GPU helps, but it's still relatively expensive

Here's an article about ROBLOX running BERT on cpus versus GPUs.

https://blog.roblox.com/2020/05/scaled-bert-serve-1-billion-...

They're not training so I think that's the difference.

"AVX512 instructions cannot be sustained on a high clock speed for very long -- the processor downclocks right away if you send too many of them within a short time frame, probably for thermal reasons. This will probably be much more true for laptop processors." https://github.com/pytorch/pytorch/issues/4825#issuecomment-...

How much does this apply to the Xeon CPUs found in servers?

Reply on news.ycombinator.com