slizard

Hard no. Amazon EFA can barely come close to a dated HPC interconnect from the lower part of the top500 (when it comes to running code that does use the network, e.g. molecular dynamics or CFD), Azure does offer Cray XC or CS (https://azure.microsoft.com/en-us/solutions/high-performance...) which can/will be set up as proper HPC machines with fast interconnects, but I doubt these can be readily rented in the 100s of PFlops size.

Check these talks from the recent ISC EXACOMM workshop if you want to see why HPC machines and HPC computing are an entirely different league compared to traditional data center computing: https://www.youtube.com/watch?v=9PPGvqvWW8s&list=WL&index=9&... https://www.youtube.com/watch?v=q4LkF33YMJ4&list=WL&index=7

In fact, _better_ than Chrome for most websites (exceptions perhaps google's). I used to struggle with Chrome/Chromium hang and chew on CPU cores, crawl to a halt, crash, or just simply run out of memory or a regular basis. Switched to Firefox ~67-69 and such issues are completely gone. I still have significant CPU usage from background tabs occasionally even in Firefox, but the rest of the issues are all nearly inexistent (even though I have on average 30+ tabs open).

Without HBM or more memory channels, the top SKUs will be rather hard to feed considering the (claimed) at least ~3-5x increase in instructions/s/socket while only increasing memory bandwidth by ~20%.

> But that has little bearing on either what’s possible in the future,

There is certainly a chance that 10-100x more flops will appear on the fah network, but as I said I just don't believe it will happen.

> nor why they aren’t scheduling compute time on Summit, right?

They are. The researchers who get to use the FAH network certainly do have access to traditional supercomputing resources too. Some types of problems require strong scaling (and reliable resources) which requires HPC iron.

> > Also AFAIK garbage flops also count.

> That’s always true in these peak performance measurements, both for distributed projects and for supercomputers too. Also true for CPU & GPU peak flops specs.

I think you're misunderstanding me: I literally meant that the data files returned by the FAH contributors may often be garbage due to data corruption (OC'd cards, no ECC, overheating hardware, poor storage etc.). Not sure to what extent is this still the case today.

> I just don’t see any reason why Folding@Home would or should refrain from distributed crowd computing just because supercomputers exist. They can both happen. We don’t need to try to come up with efficiency numbers or compare utility/flop in order to see that Folding@Home is producing some useful research results, right?

In principle you're right. There are some questions around it, though. Briefly: access is a privilege of few; oversight?; inefficient use of hardware (low Flops/W), just to name a few.

> Maybe it’s worth pointing out that compute time on TOP500 supercomputers is not free,

No, it is not, but access is granted by grant agencies with at least some transparency and oversight as well as scientific review; also, those machine are far more efficient (flops/w).

> and Folding@Home is not a job you can run for 1 day or 1 week and be done.

No, Folding@Home is not the "job"; a "job" at least in HPC sense is a molecular dynamics simulation (or many); a set of such jobs is what is typically required for a project to be completed result of which would end up in a publication. The computational work corresponding to such a project/paper, depending on the size of the machine, can in fact be run in 1 week, and on a large machine even in 1 day.

Unfortunately typically there is no way to directly donate funds (especially not $10, even if there are 10000 people who'd do that on a monthly basis) to some research (group) of one's choosing. Therefore the choice is whether to donate to an @home project and trust that what they do is meaningful and the donated resources are used in a responsible manner.

In that respect, the responsibility of whether to ask for and how to make good use of donations lies solely on the teams that receive the donation. Without oversight it would however be foolish of them to be overly critical on their own shortcomings as there is a great benefit to having these cheap FLOPS (and good PR) that F@H brings.

I was about to suggest that it would be great to set up a merit-based funding scheme somewhat akin to the governamental funding agencies, but one run independently by the "council of the people". I'm however uncertain how effective could such an organization be at awarding the funding in a responsible and effective manner.

> I do agree it's difficult to formulate how to ask for donations (money or cpu-time), you need to seem optimistic otherwise nobody would do it but research takes a huge amount of money and cpu-time and calendar time so there surely is a gap here between expectations and reality..

And there is a tricky balance between over-promising and under-delivering vs being realistic and lacking enough "flashyness" to get funded/thrown money at.

On a second thought, projects like the Exscalate4CoV [1] might have a chance to contribute even short-term as they do have the explciit goal to also focus on "Identify virtually and quickly the drugs available, or at an advanced stage of development, potentially effective"

[1] hhttps://www.cineca.it/en/news/exscalate4cov-hpc-platform-win...

We're talking about Folding@Home not any (or all) distributed computing project(s). I've not seen recent stats on the size of the FAH network, so not sure how does it compare to large supercomputer resources, but plots available through an easy google search show that in the past it was never even larger than the biggest machines on the TOP500, let alone multi-oom larger.

(Also, AFAIK garbage flops also count. As far as I recall talkin to a researcher working with FAH a qhile ago, a large-ish fraction of results returned were not usable due to data/files being broken.)

My colleagues run numerous projects that transfer knowledge between computational research, including molecular simulations, and wetlab research as a routine. Computational and wetlab experiments are reguarly designed to complement each-other. Examples of numerous bio-phyiscs/bio-chemistry research done around me (ranging from research on ion channel dynamics [1] to drug permeability through the skin barrier [2,3], or cryo-em structure refinement just to name a few) strongly contradicts your experience as well as your final statement -- and than we've not even mentioned the drug binding affinity computations which is these days routinely done using molecular dynamics by both academia a pharma industry.

[1] https://doi.org/10.1016/j.celrep.2018.03.10 [2] https://doi.org/10.1016/j.jsb.2018.04.005 [3] https://doi.org/10.1016/j.jconrel.2018.05.026

> But Folding@Home is setup for home,

Nothing prevents running the fah client on nodes of a compute cluster -- in fact my colleagues did that (while running a local F@H server), though that was a number of years ago just because they wanted take advantage of the distributed computing facilities provided by the client-server setup and built-in algorithms.

> crowd sourcing has the potential to scale to many orders of magnitude larger than what can be done in a data center.

Potential it does have, but I am skeptical of the "many orders of magnitude" claim ever having a chance to materialize. I'd love to see a cost / benefit analysis on the effective amount of useful work contributed vs the cost of the same in a data center.

I agree.

In fact, donating in other ways, e.g. giving free programming, HPC etc. education to more researchers can have a higher impact on the long run.

> Does anyone else do it better?

To that question, assuming by "anyone" here you are asking about other donate-your-cycles-distributed-computing-projects: I am not too familiar with how well-optimized the codes of different @home projects are.

Taking a few steps back, perhaps the efficiency of these codes is the lesser issue and to be honest, in some (many?) cases other forms of donation/contribution may further more scientific progress than simply crunching numbers on one's home PC.

> When donating money or compute time, I want my donation to be used efficiently and not be wasted by high overheads.

Fair point. I think this is something you should bring up with the authors of Folding@Home. I do not work on that project.

My personal view on this is that there is always a cost/benefit balance that one has to strike which is often tricky especially given the considerably constrained resources as it is typically the case in academic computational/simulation tool development.

It is however, as you point out, a great responsibility of the researchers and developers of codes to make sure that choices made and action taken (or not) do not lead to disproportionate waste of resources donated by volunteers or awarded through a grant by research funding agencies.

I do not know the detailed reasons why F@H chose to not update CPU "core" since FahCore_a7 (AFAIK based one GROMACS code from 2014), but it is likely related to the aforementioned cost/benefit analysis done in their team. One of the motivations could have been (just hypothesizing) that the software engineering efforts estimated to be required to update FahCore for CPUs (which generate a very small fraction of the "points") would have taken away resources from the GPU FahCore.

Sure, I'm aware that this is fundamental research (likely not on actual protein folding, but haven't looked at the details), which will probably not have short-term outcomes, not on the timescales required for a 1st gen vaccine (perhaps not even 2nd gen).

That does however not take away from my main point, if anything it is in support of what I was trying to emphasize: such computational research is better suited for a compute cluster.

Can you please elaborate? Are you referring to the F@H CPU "cores" being outdated and inefficient?

It would be nice if NVIDIA donated compute time on one of their large in-house machines the DGX SuperPOD [1] or the DGX SAturn V [2] (#20 and #67 on the last Top500, resp.) for COVID-19 research. Running simulations in a data-center is far more efficient.

[1] https://www.top500.org/system/179691 [2] https://www.top500.org/system/178928

[Disclaimer: How do I know this stuff? I'm a core developer of GROMACS, one of the major molecular dynamics simulation FOSS community codes. Molecular dynamic is the algorithm/method behind the F@H simulations. I work as a researcher in high performance computing and have (co-)developed many of the parallel algorithms and GPU acceleration in GROMACS.]

20-200x is simply not true. Typically, such numbers are a result of comparing unoptimized CPU code to moderately or well-optimized GPU code which is often misleading. (Such differences are however perfectly reasonable when comparing hardware-accelerated workloads like ML/DL). If you compare actually well-optimized codes, you'll see more like ~4-5x difference in performance for FLOP/instrction-bound code as it is the case for well-optimized molecular dynamics.

Case in point, I recently pointed out the huge difference in CPU performance of two of the top molecular simulation codes, one of which is 8-10x faster on CPUs than the other, solving the same problem [2].

F@H relies on GROMACS as a CPU engine [1] which happens to be the same code as I quoted above as the fast one. The trouble is that F@H has not updated their CPU engine for many years and distribute CPU binaries which lack crucial SIMD optimizations to allow making use of AVX2/AVX512 on modern x86 CPUs as well as the years of algorithmic improvement and code optimization we made. These two factors combined lead to _significantly_ lower F@H CPU performance compared to what they had we're they using a recent GROMACS engine.

Consequently, due to the combination of an inherent performance advantage of GPUs and the severely outdated CPU engine, it is indeed not worth wasting energy with running F@H on CPUs.

[1] https://en.wikipedia.org/wiki/List_of_Folding@home_cores#GRO...

[2] https://twitter.com/twilard/status/1235142089156984832?s=20

Edit 1: adjusted wording to reflect that the performance difference between running outdated GROMACS version and subotimal SIMD optimizations on modern hw can have a range of performance difference, depending on hardware and inputs. Edit 2: fixed typo + formatting.

Hahha, good one. Just that by the time that one dev implements everything, you'll be overtaken by the competition -- at least that's what everyone's fear is, and it's partly warranted.

Well, those who were willing to pay the ~$150k for the DGX-1 (Volta-16Gb upgraded IIRC), won't necessarily find it too much -- and NVIDIA is after quick money before the markets start slipping down the hyper curve.

You also do get quite some meat compared to the DGX-1: the equivalent of 2x DGX-1 in terms of GPUs and NICs with 4x HBM2 size, plus the NVlink fabric. Plus a more SSD storage, Xeon Platinums to round it off.

Oh, and 350 lbs, let's not forget. :D

Expensive, yes! Will it sell? I'd be very surprised if it didn't!

Wild guess / conspiracy theory: Intel, afraid of the damage to their image just made worse by diminished performance advantage compared to AMD )due to Meltdown), fearing long-term market loss, quickly found ways to tackle the issue by, instead of pedaling to regain trust, damaging a competitor's image. It seems like a reasonable long game to support and perhaps steer the disclosure of AMD vulnerabilities that CTS-labs had been investigating. Or maybe is was Intel investigating themselves, had some cards up their sleeves, but needed some other entity to do the public disclosure.

Other theories discussed here seem less far-fetched than the above, but in any case, it does smell funny.

First indications of early KAISER fix testing in the wild..?

(which might have also been a test to see how will customers react)

That's correct and it is a fair way to see it. However, Intel dragging two of its main competitors name into the limelight is a rather uncool move.

research != deep learning

> should make cloud GPU time cheaper and/or more efficient.

No way. These will be not put in data centers of cloud providers. I'm not even sure many regular data centers will have anything like this -- especially if NVIDIA keeps pushing and blackmailing server vendors to stop selling servers packed with GeForce cards if they don't want to loose big [1,2].

[1] https://www.pcgamesn.com/nvidia-geforce-server [2] https://www.digitimes.com/news/a20171027PD200.html

I'm saddened by the arrogance you display. Does your definition of "world" really not go beyond the "coastal cities" and "North America"? Are you seriously assuming there are no ML researchers outside of the cozy little world you're referring to? People in the Middle-East, Balkans, South-Asia, etc. should not worry about "high-end kit", they should stick to "hobby" stuff?

(In case if this all leaves you baffled, e.g. assistant professors will often earn 2-4 TITAN V's worth of money per year in places like Eastern Europe -- and that's still on the map of the "Western world" by most definitions.)

> It’s an expensive bit of equipment for an independent researcher or student, or someone in a poorer part of the world.

Allow me to correct you: that is _much_ (if not most) of the world.

> But why would anyone expect the best kit for hobby prices?

Strawmen. Please read my point again (and see the concrete examples of affordable "kit" that can be used to target the high-end HPC iron).

CUDA version compatibility, for instance. Every new release requires new series of drivers.

Your story sounds very familiar. Add to that list the regular regressions in new driver release that require rollback, but the binary blob drivers sometimes forget to clean up properly and you can end up in a library mixup hell... no fun.

On the other hand, in terms of robustness and reliability on the compute side of things, the NVIDIA drivers are decent. I've only experienced 1-2 hard kernel crashes the last year -- which used to be an at least weekly event 5+ years ago.

I still wish they'd split their driver and contributed upstream parts of it if for nothing else, to make compatibility layers and shims more robust, but knowing NVIDIA I doubt that's on their roadmap for the coming century. :)

> It is strange that you expect to be able to buy the server grade GPUs for GeForce series prices Respectfully, you are clearly missing the point. You can buy <$500 USD Xeon Silver parts (and fairly affordable barebone workstation or servers to go with) and you'll get the Xeon-SP architecture to develop for. You can even buy the rebadged i9 parts too for ~$1000-$1200 with both FMA unit enabled, but admittedly those are also rather expensive.

> and are outraged when reality doesn't meet your expectations.

Untrue. I want an option, any option that allows researchers to develop for the fancy high-end chips -- especially when they claim that they want to and are providing such an option. Computational scientist and HPC researchers who aim to write code for, or porting community codes to machines like DOE Sierra, Summit or future V100-based machines need something that is within the reach of a junior PI, a researcher in countries less well off than the rich western Universities and research institutes.

In contrast, jack up the price this high means that the Facebook et. al will still find it worthwhile, but the grand majority of actual researchers, those that they claim to be enabling and helping with this card will not be able to afford it.

It all too easy to forget how the "democratization" of supercomputing was NVIDIA's slogan very ambitious and cool [1] that has over time turned into a marketing claim [2] that offers little in terms of bottom-up enabling the community.

[1] http://gpgpu.org/static/asplos2008/ASPLOS08-1-intro-overview... [2] http://assets.nvidia.com/nv/tesla/pdf/NVIDIA_Accelerate%20Yo...

Please show me where can you buy a $800 card that can be used to develop for the _compute_ architectures; i.e. the GV100 or the GP100 for that matter.

Also, please show me how can you use those $800 cards for serious computing/research without potentially running into silly issues and a lot of pain (from not being able to use the "supported" standard OpenCL to running into cooling, fan speed, clocking issues that makes reliable perf tuning a pain, etc.).

"With TITAN V, we are putting Volta into the hands of researchers and scientists all over the world"

Screw you NVIDIA. I've been vocal about this for a while but this is now getting utterly ridiculous, and frankly insuting to the research community.

I don't mind that NVIDIA is ripping of the big derplearning -- and more recently "data" -- companies (Goog, FB, Baidu, etc.), but this greedy bullshit attitude led to the Tesla cards all of a sudden going from expensive, with a 2-2.5x price jump, to the ridiculously priced. In the meantime, they've been "enabling" research by throwing peanuts at the science community with cancer stunts [1]. However, that was just the facade, the real story was that they'd found the cash-cow and the enthusiastic HPC/sci-comp crowd that helped NVIDIA grow out of their sexy-witch-on-the-front-panel-of-the-GTX company box and step into the "big compute boys' league" is no more than just another means to convert peanuts to marketing material.

After years of complaints that there are no sensibly prices _compute_ cards devs and independent researchers can afford, constant battles with NVIDIA to stop crippling their management tools on GTX cards, to allow cheap cards to be used productively for compute and development, etc. they step up their game and release a $3k "affordable" card for "researchers and scientists". This claim is beyond ridiculous, revolting and, as a researcher (working in HPC on a large FOSS simulation project) this is insulting. Which researchers are they talking about, the ones employed by the big ad, finance, companies? Definitely not me, not us!

Time for us to start showing NVIDIA the middle finger for good! Get porting people, support AMD, test ROCm, their openness and attitude receptive to public feedback and criticism has impressed me recently. At the same time, we need to push NVIDIA too: file bug reports against NVIDIA's shitty OpenCL support, against their consumer card crippling drivers and management tools, and resist their efforts to blackmail OEMs to stop selling servers with GeForce.

[1] https://www.nextplatform.com/2016/11/15/deep-learning-superc...

Beats Intel Broadwell and Skylake in many memory-bound codes and surprisingly enough it can compensate with core count and clocks for its meager SIMD units even when running strongly instruction and cache-bound codes like GROMACS.

Among the many questions remaining consumption/efficiency is certainly among the top and which accelerators will the ThunderX2 platform be paired with first? NVIDIA does not have (officially) an Aarch64 driver. AMD's OSS ROCm stack however will soon be released for Aarch64 [1].

[1] https://github.com/RadeonOpenCompute/ROCm/issues/211