Japan plans 130-petaflops supercomputer
arstechnica.com
arstechnica.com
[1] http://arstechnica.com/information-technology/2013/11/18-hou...
* Using virtual machines isn't a bad idea for cases where you have a one off calculation for a paper that you need to publish. But for some groups they need to continually model and predict stuff. For example, weather forecasting needs to be done every day.
* Hadoop MapReduce and friends are extremely slow. Virtualization slows things down as well. The person in your article was using software called Shrodinger which isn't MR.
* Often, MapReduce performance is bottlenecked by the shuffle stage. This requires a lot of data to be passed around the network. Ethernet and http connections are not ideal for this if performance is your goal (though they are good for resilience). RDMA (remote direct memory access) with Infiniband is much better (offloading lots off the cpu, less chatter, etc). There exist RMDA plugins for Hadoop but they're of ymmv quality.
* MR isn't an ideal paradigm for every calculation. MR comes from search indexing (google) and is useful in areas where you need to look at all the data and moving the data around is more intense than the calculations. If you can filter, feature extract, or otherwise reduce the amount of data being moved around, the benefits of using HPC style clusters win out. Places where you must look at all the data: Indexing, Adversary detection (insurance fraud, intruder detection), format change (mp4->webm, gz, json->bson), ingestion, backups. If you're not doing these things you might want to consider whether MR is right for you.
* Even in search indexing, deep learning is becoming en vogue and this is extremely computationally expensive. This means the cost of moving the data around is again taking a back seat.
* A large part of the benefits of Hadoop style MR was having local data. But network speeds have outpaced drive speeds so data locality is less of an issue. Coupled with deep learning, there's less need for HDFS type systems. GPFS or Object storage is fine.
Also, the page says it's only a theoretical petaflop:
"While Stowe says the Amazon cluster hit 1.21 petaflops, that's the theoretical peak speed rather than the actual performance. In the Linpack benchmark used to test supercomputer speeds, the theoretical peak is always reported, but the real-world results are what count when ranking the world's fastest machines."
There've been several points where getting a detail right has seriously improved performance. These made me wonder how well the generic map reduce approach could really deliver on performance across use cases.
>There've been several points where getting a detail right has seriously improved performance. These made me wonder how well the generic map reduce approach could really deliver on performance across use cases.
It can't. A big selling point of Big Data systems is that you can analyse unstructured data that you haven't seen before and quickly dig in. But then if you try to reproduce the benchmarks you basically have to perform a parameter sweep to get the performance numbers that people are advertising. Maybe there are some super experts who can divine the parameters for their calculations ahead of time, but I've never been able to do it.
Then on the HPC side you have people who look at the Big Data benchmarks and then take is as a challenge by writing some specific code to tackle the problem which trivially smashes the Big Data solution. Usually using a clever encoding. But this misses the point -but doesn't really since I think everyone was cheating/not-cheating anyway.
[0] http://www.nrlmry.navy.mil/coamps-web/web/home [1] http://www.wrf-model.org/index.php
And fnord123, could you please email me?
While Mass is the most outspoken on the subject, many experts insist that if the Weather Service wants to meaningfully improve its predictions, it must employ a technique called ensemble forecasting. The basic premise is either to tweak the physics equations or to make repeated changes to a model’s variables: You might bump up the temperature slightly, for example, and then run the model again. After a half-dozen or so reruns, you get a set, or “ensemble,” of forecasts that can be compared with one another. When all the forecasts in an ensemble agree, it’s a reasonably sure bet that the predictions will pan out.
Nobody I’ve spoken to doubts the superiority of ensembles. Yet they haven’t been widely adopted in the United States at the resolution required to forecast localized, or “mesoscale,” events — specifically, thunderstorms, flash floods and tornadoes — because high-resolution ensembles require more computing power than the National Weather Service can currently provide. Higher-resolution ensembles translate to greater accuracy in the same way that HDTVs are clearer than analog sets. I met with a scientist at the National Center for Atmospheric Research in Boulder who showed me a prototype mesoscale ensemble for the United States. But at the moment, he can’t exploit its full potential because the supercomputing cluster at the Weather Service simply couldn’t handle the load.
HPC systems are essentially designed for this type of computational problem, and we still want more powerful versions of it. For this reason, I find it unlikely that commodity cloud infrastructure - which was not explicitly designed for this type of problem, and it almost assuredly going to perform worse - will have much success in this area. At least in the near future.
So the distinction of "commodity" that Google made in the original map reduce paper in 2004 is gone. And the benefits of data locality are all but gone. The paper was great when it came out but it's long in the tooth since the hardware of today is different.
To answer your actual question, I've never heard of anyone running WRF on cloud infrastructure. It doesn't have to be a terrible idea. If you're trying to come up with a competing model then the agility you get might be needed. i.e. being able to roll onto new hardware as it becomes available instead of having to work with budget cycles for a hardware refresh lets you move from tesla k80 to titans whenever AWS decided to make them available.
Searching around, I see Glenn Lockwood did some benchmarks in 2013 with AWS:
https://glennklockwood.blogspot.be/2013/12/high-performance-...
This lot seem to have done some more benchmarking in 2014 (tldr 44% overhead on VMs - but take the time to read it since glockwood is always a good read): https://www.researchgate.net/publication/301407792_Performan...
I think the prevailing issue is that latency on virtualized nodes is poor so you have to work around it. Maybe fat nodes with gpgpu can overcome the issues.
If you need to talk to me privately then you can message me on reddit.
Abstract is here: http://www.eresearch.org.nz/event/eresearch-nz-2013/do-hpcs-...
For highly parallelizable and distributable applications (anything where mapreduce makes sense), commodity clouds are awesome. Just like relying on the subway in a city. You'll have a bit of a hassle when you need to go shopping, but your normal use case is such that it is worth the hassle.
Now let's say you have a code where your tasks require much tighter coupling due to communication (basically "Science codes"). In these cases, what you want is the interconnect. You want to be able to get a physical block of nodes in a very short amount of time for a somewhat long amount of time. In terms of cars, imagine that you live outside of a city or tend to do a lot of shopping or long distance driving.
And that is basically the logic here. It sounds like the Japanese rig is intended to be similar to the US's ORNL Titan in that it is primarily a capacity machine for science in general. So people with those kinds of jobs can buy time and run their large simulations with ease while still relying on stuff like AWS for postprocessing and the like. In that case, even though this is a 130 petaflop machine, it will likely only ever run one or two jobs at that scale and will instead operate as a "cloud" with a focus on large contiguous allocations.
Whereas the other form of SuperComputer are the ones meant for large jobs for a small subset of users. Those are the ones that tend to be owned directly by governments and involve a fair amount of effort to get access to. Those run a mixture of the above kinds of jobs (for the target community) as well as actual large scale simulations. As for what they are simulating: Think about what would need really detailed simulations using codes and data that probably should never be near Amazon's servers.
I've pointed this out to people in SC community, they don't really care. Either they have a problem that only runs on a SC (IE, a science code with strong scaling) or they're just not interested in trying other approaches.
Anyway, a supercomputer for deep learning makes sense. The communication patterns in deep learning can be very intense, with a lot of n2 calls, and so a good interconnect might speed up the code. However, tensorflow etc aren't designed for MPI SC architectures.
I think this will probably change: I think DNNs are going to run on supercomputer-like hardware, but not something that is a traditional supercomputer with MPI.
There is one paper showing a port of TensorFlow to MPI (WHYYY?) and the results aren't very good.
They ran it for 18 hours. So that's $44k/day for 1 petaflop. This supercomputer is 130 petaflops. So let's pretend it scales lineally.. that's $5.7m/day.
In just 30 days on AWS, you could buy the $173m supercomputer in the article.
This isn't a serious alternative.
Anime rendering? pretty rad action scenes!
Newer Intel chips will be integrating low-latency interconnects so in the future, Amazon, Google might be adopting low-latency interconnect computing.
or - how long ago would $173M have bought you the kind of power we carry around today in a pocket?
In a (simplified) nutshell: We can fit more transistors in a given unit of area because we make them smaller. That used to just mean we increased clock frequencies (make it faster) but comparatively recently (decade or two) meant we increased parallelism.
Moore's Law is expected to fail because we are now reaching the point where smaller transistors are very heavily impacted by actual limitations imposed by physics.
So while it is possible we'll have a technology shift and see similar performance gains, it won't really be Moore's Law anymore (unless we start using Pym Particles or something).
What kind of technology shift do you mean? Like a totally different computing paradigm?
But none seem all that promising and my gut is that we'll focus more on interconnects and algorithmic improvements.
But time will tell.
For those reading along, this is a fictional particle named after Hank Pym - AntMan - from Marvel comics. It's not a technology in a lab somewhere.
I don't have an estimate. My point was simply to explain to you why your logic was flawed as we are nearing the limits of what Moore's Law can give us without some pretty massive changes. This isn't a case of "Clock speeds are capped. We are doomed. Oh, wait, we can just put two slower ones on the same die" and is more "So... we are out of physical space..."
Trends are great when you are trying to make sense of data and estimate how to move forward. But they should not be used in a manner that ignores actual data.
I'd say a better estimate would be to assume density stops increasing around the point when feature size is the size of a silicon atom. I'm sure that'll be way off, but closer than estimating 21 more doublings.
I think it seem like there is a much larger anti-science sentiment then there is because these people have been given a fresh voice with social media and for the first time in a long time they can connect with each other and build echo chambers to shout at eachother in.
Some of this spilled out in the last election and provided a non-trivial number of votes for a candidate who was clearly a demogogue, instead of voting for a different demogogue who appealed less to the uneducated.
Estimates put the processing power of an iPhone 5 at at around 2.7x times that of the Cray 2[2]
[1] http://www.theregister.co.uk/2012/03/08/supercomputing_vs_ho...
[2] http://pages.experts-exchange.com/processing-power-compared/
Mainframes in general too.
I'm currently listening to Soul of a New Machine.
Mainframes are great, already using almost[0] memory safe systems programming language on the 60's with Burroughs, followed by IBM and a few other vendors.
Virtualization and containers with the 360.
Bytecode as universal binary format with JIT/AOT at kernel level, DB based file system, System/38 and AS/400.
Object based OS, AS/400.
[0] - They still have the issue of leaks and double free though, but everything else is safe Algol style with explicit unsafe blocks/modules required.
I also find it kind of amusing that IBM, a primarily consulting company, developed AS/400, given that part of its sales pitch is that integrated database requires no maintenance and you can forget entirely about your IBM i and just leave it running for a decade.
It's a neat system. I wish I had the opportunity to use one. In many ways it feels like we're still catching up to what System/38 was doing in 1979.
That would be a great read.
My favorite Cray tidbit: for fun, Seymour Cray dug tunnels underneath his home, and had a lot of his breakthroughs while doing so.
he attributed the secret of his success to "visits by elves" while he worked in the tunnel:
"While I'm digging in the tunnel, the elves will often come to me with solutions to my problem."
https://en.wikipedia.org/wiki/Seymour_Cray#Personal_life"Seymour Cray was a man of few words. I was there for three weeks before I realized he was not the janitor."
So, if this was linear, we can all expect 130 petaflop computers around 2043 for around $3,500?
Lots of caveats here though, things aren't usually as linear as all this, and this is very much a back-of-a-napkin calculation.
Given that Moore's Law is creaking I wouldn't expect pocket petaflops any time soon. I'd expect a serious outbreak of cloudy clusters everywhere, and perhaps a dynamically reconfigurable Internet 2.0 with completely transparent non-localised computation.
This might change if computing finally goes optical and/or quantum. But if we're pushing electrons around wires, current hardware is close to the physical limits. The only way to speed it up is to build a lot more of it and speed up the connections.
I'll link the url when I've scanned my bookmarks.
psedit: look at jojomonkeyboy's comment http://www.techrepublic.com/blog/classics-rock/the-80s-super... he says an i7 2600 (not an i3) has less sustained compute power.
https://en.m.wikipedia.org/wiki/Cray_X-MP "The Cray-2, a completely new design, was introduced 1985. A very different compact four-processor design with from 64 MW (megaword) to 512 MW (512 MB to 4 GB) of main memory, it was specified to 500 MFLOPS but was slower than the X-MP on certain calculations due to its high memory latency.
The X-MP-succeeding Cray Y-MP series was announced in 1988; it also had a new design, replacing the 16-gate ECL gate arrays with a more compact VLSI gate array with larger circuit boards. It was a major improvement of the X-MP supporting up to eight processors."
Note latency is huge for these systems meaning for most workloads modern cellphones absolutely crush them.
If you look at Apple's recent offerings, it would seem they think that time is more or less here.
I certainly almost never use the full computing power of anything I'm using – the limiting factors aren't hardware any more but the software running on it.
VR might just about squeeze through now, but given the hardware limitations - and the fact that people look really dorky using it - I'm not expecting it to drive a new explosion of user interest.
We really need some completely new tech to drive a new wave of innovation. The obvious candidates are optical/quantum and perhaps direct neural interfacing. Both are still science fiction, but that may change by 2030.
More extreme technologies may also be possible, but they're beyond speculative.
For now it may be useful to remember that technology rarely develops linearly, so speculating about future CPUs is like speculating about the future of transatlantic cruise liners, while ignoring the fact that someone somewhere is working on heavier than air flight.
I would really like to be excited by the prospect of neural interfacing, but all I can imagine is people catching computer viruses.
I'm not sure how to do the calculation to find the minimum amount of power required for 130 petaflops, but it seems like it's well within the capacity of today's mobile batteries.
What we need to advance is specialized Deep Learning cores that are compact, fast and low power, so they don't kill the battery. On the other hand, for general apps it's much less necessary to make the CPU faster.
Right now, I guess, 1 GFLOP is about 0.5-1W of power. This means that 130PFLOPS computer will draw 75-130MW, definitely even more.
It can be a problem.
[1] https://www.top500.org/news/china-tops-supercomputer-ranking...
all algorithms will become O(1) and even the shittiest code will be instantaneous. which is great for me.
great for me maybe bad for encryption and governments and idk.