Also, we tried to get into NVIDIA Inception and were rejected for the company being too small, so even if we wanted to do GPU deployment, we wouldn't be able to buy 3090s for testing. Scalpers / eBay are not an option in Germany because for company purchases, I need a proper tax invoice from another VAT-registered company.
At the time, the price of the A4000 was similar to scalper-prices for similarly-performing gaming cards, but:
- Had ECC RAM and was generally designed for ML/enterprise/data workloads
- Was more power-efficient
- Had 16GB RAM, which you won't get on anything short of an A3090
I'd taking a consumer card at MSRP over the A4000, but not at street price. For what I'm doing, an A3060 Ti would probably even have been fine. It has an MSRP of $330, but a street price of $500-$1000 depending on the month you look. The enterprise cards tend to sell at MSRP (which is just around a grand for the A4000).
Note that NVidia reuses part numbers. With "4000," there is also an older NVIDIA Quadro RTX 4000 (no A) selling for the same price on eBay / Amazon / etc. as the Ampere-based A4000.
You can start with things like switching the weights and activations to fp16. A100 also supports BFloat16, which is even better and should work right out of the box.
GPU instances being absurdly pricey is not a good enough reason?
The biggest hurdles are learning the dev tech stack and setting up the development environment.
The last time I tried out CUDA, it took me a couple of days to correctly setup the environment; just to spend an hour playing around parallelizing math operations.
The best way I could describe the first impression dev experience: It’s a huge hack, adding non-standard language extensions to C/C++
And so relying on their own compiler or linker (iirc), instead of the standard C++.
Couple this with NVidia not playing nice with OpenCL to force you into their playground, and I lost interest pretty quickly.
There are cases where the random access to RAM are very deep in the algorithm. Consider e.g. sparse matrix multiplication.
Some easy-to-parallelize math and vector operations can yield impressive results without it being anywhere near optimal.
I do agree that optimal requires mastery of GPU programming; but I also think GPU programming can yield good results without said mastery, and be quite accessible
If anything it should be easier than writing raw multi-threading code, because GPUs aren’t (weren’t?) general purpose calculation units.
So more bare metal concerns, but less to reason about overall.