RTX A6000 has more VRAM than any other GPU types currently offered by public cloud providers AFAIK.
159 karma · joined June 2, 2012
As bubblethink already mentioned, the article is arguing that batch processing jobs (e.g. those characteristic of Deep Learning) don't benefit substantially from cloud infrastructure.
High availability, edge processing, fast network upload, and the lego blocks for creating redundancy are moot.
Most training algorithms automatically recover if a node goes down. That covers all required failure handling.
Admin:
Time (and cost) it takes to install software, drivers, etc
These tasks aren't made unnecessary by cloud. Yes, with cloud, once you've done the work, it can be forever encoded into an image or container. However, the same applies to on-prem using container solutions like Docker.
Maintenance / Capitalization / Finances:
3-year? What is the useful life of this? When does it seemingly become obsolete?
With GPUs of recent history, obsolescence is not a concern in a 3-year time frame. On AWS, people are still using K80s (released in 2014!). The GTX 1080 Ti, which was released 2.5 years ago, is selling for substantially above MSRP. This may change if competition in the GPU space increases and NVIDIA loses its monopoly.
AWS will continually upgrade their hardware and you keep paying the same
True, but this concern is mitigated by slow rate of GPU obsolescence.
Hardware can be capitalized, which means you can push it to the balance sheet (for tax or valuation purposes)
Can you say a little more about this? Not sure what you're getting at.
Spending $90k instead of $184k in year 1 with the option to turn it off if you want (no longer need). This could be very valuable for a startup who wants elastic spending patterns.
This needs to be evaluated on a case-by-case basis. A poorly-capitalized start-up with an unpredictable compute workload is not a good candidate for buying on-prem. A well-capitalized start-up that consistently uses GPUs is a better candidate.
Returns, breakage, warranty in case of a hardware failure
Any reasonable hardware provider will include an option for 3-year warranty.
Many benefits of running infrastructure in the cloud are lost for offline batch processing jobs. Training machine learning models doesn't require low response times, high availability, geographic proximity to clients, etc. Yet, with cloud, you're paying for all this extra infrastructure.
The main benefits of cloud for tasks with machine learning type workloads are cost saving (if utilization is low) and no electrical set-up.
On the other hand, cloud is extremely expensive for groups that require high base levels of GPU compute. The article is arguing that such groups can save a huge amount of money by moving infrastructure on-prem.
Our company used to run a cluster of ~1000 gpu servers for inference and training. We used caffe and torch. Provisioning and software maintenance was a huge hassle.
We realized how painful it is to get a machine set up for deep learning, so we decided to release our debian packages to the public.
We sell computers built for deep learning researchers and figure the more people we can help get started, the better :)
https://lambdalabs.com/lambda-stack-deep-learning-software
We wrote debian packages for every framework, including cuda and cudnn. Using our Debian repository, you can install all these frameworks using apt/aptitude.
When a new version of a framework comes out, we usually have it available in 1-2 weeks.
Dreamscope doesn't actually make money :) It's a little under break even. It brings in revenue through a $9.99/mo premium subscription, which gives customers higher resolution images.
Bottleneck depends on the workload. If you're training a small/fast network, data bandwidth is a real problem.
That being said, for most cases, a workstation build that provides every GPU with 16 lanes is far less cost effective.
The server we sell is packaged with software we wrote that makes administering significantly easier. We also provide technical support and even a limited amount of free machine learning consulting. The customers who purchase this server want a headache free solution and aren't as price sensitive as a lone researcher.
If you don't require a rack mounted server, a cluster of workstations like NVIDIA's DIGITS DevBox is far more cost efficient (and less noisy). I run a compute intensive business (Dreamscopeapp.com) and we opted to build a cluster of desktop-like machines instead of using a rack mounted solution. Another benefit is you don't run into the power issues mentioned in the post.
My start-up actually sells the machine described in this post: https://lambdal.com/nvidia-gpu-server
And a machine inspired by the NVIDIA DIGITS DevBox: https://lambdal.com/nvidia-gpu-workstation-devbox