Hacker News Reader

Comment by andrewon | Hacker News Reader

"Training was done on eight NVIDIA A100 GPUs for 1000 epochs" "trained on four NVIDIA V100 GPUs for 300 epochs" "model was trained on a NVIDIA DGX-1 cluster using eight GPUs"

No doubt these are model innovations. But can't help wondering how much ready access to arrays of GPU might give them advantage... You know how many different models need to be tested before getting to the winning ones.

View on HN

Disclosure: I used to work on Google Cloud.

I dunno, their A100 results took about 20-30 minutes on 8 x A100s [1]. 8xA100s is like $24/hr on GCP at on-demand rates [2]. So like $12/run.

The efficiency was okay but not linear, so if you were more cost constrained you might go with 1xA100 for $3/hr and have ~2.5hr training times.

Getting that performance out of a GPU is more challenging than getting access to the GPUs. All the major cloud providers offer them.

(Nit: GCP deployed the 40 GiB cards rather than the later 80 GiB parts, but let's ignore that).

but it often doesn't matter

[1] https://github.com/NVIDIA/DeepLearningExamples/tree/master/P...

[2] https://cloud.google.com/compute/gpus-pricing

That is the final model right? Not all the training while hyper params, or structure, was developed.

No matter if it takes you a single run or ten thousand runs; qualified manpower to code and supervise it costs more than $12/run, so with prices like that the limiting factor is how much people-time you can devote to it.

I disagree. I've participated and even won a few of those "ML in sciences" competitions in the past, and all other things being equal, the one with the most compute wins. 12$/hour sounds doable, but as soon as you need to grid over more than 4 hyperparameters (and that's small), you're looking at 100s to 1000s of runs in parallel. And you'll iterate over that process several times. Eg It's no coincidence that deepmind (with all of alphabets compute power) won CASP with alphafold. Yes they have a lot of very smart people, but so does everyone else. But no-one has the power to experiment with such large models so extensively for so long.

If you're doing grid search on more than 4 hyperparameters, that probably explains why you think computation wins. Even random search with a constrained budget will do pretty well without breaking the bank.

Pretty well? Sure. Win a competition? No.

If you're not running standard models (where you only have learning rate and regularization to grid over) but you're trying more specialized stuff/developing new models, 4 hyperparameters are not too many.

A non-trivial amount. But, this is true of basically anything. Teams are resource constrained on hardware, man-power, intelligence, etc.

Intelligence to create innovative models, hardware to test/iterate models, etc.

AlphaFold is a clear example of what's possible with big money.

One of the other contenders agrees: https://twitter.com/HassanMohyudDin/status/14466175365614018... (Saving you a click, they trained on a single 2080Ti)

Should GPU choice affect model performance? AFAIK it should only impact training time

When you can train and evaluate quickly you can iterate through experiments faster and produce better models...

The choice of hardware (here, type of GPU) could also impact the model architecture, potentially affecting model quality. For example, if GPU RAM is limited, one might have to reduce a model's depth or its receptive field just to be able to train on smaller hardware.

Reply on news.ycombinator.com