Hacker News Reader

Comment by marshray | Hacker News Reader

Amdhal "The speedup of a program using multiple processors in parallel computing is limited by the time needed for the sequential fraction of the program" was talking about single programs which at that time were generally batch processing applications. Just like today, developers tended to start with single-threaded code and then looked for ways to parallelize it.

When he says "speedup...is limited", he means that in the mathematical sense. I.e., if you had infinite processors the program would not take zero time to complete.

Adding cores is still great when you have many threads and processes competing for CPU time. If your workload is blocking on network or disk, it means you have enough cores. Anyone who would expect additional cores to significantly speed up a single, single-threaded, application is simply in error.

If this is a myth, I don't know anyone who believes it.

View on HN

Anyone who would expect additional cores to significantly speed up a single, single-threaded, application is simply in error. </ snip> If this is a myth, I don't know anyone who believes it.

9 women can't create an infant in a month?

Ah yes, The Mythical Woman-Month.

I stand corrected. :-)

Once the pipeline is warmed up...

The point is that even the most parallel programs tend to have bottlenecks where only one thread can proceed. As you add more cores, these bottlenecks increase in relative importance to the parallel parts, which are easily dealt with by your additional parallel capacity.

Did anyone (who writes parallel code or purchases CPUs professionally) really not know this?

I'm not trying to pick on the article, I like it overall. I took an interest in the Anand article when I learned you could build a 64-core box for about $4000.

This is the article it's in response to http://www.anandtech.com/show/5057/the-bulldozer-aftermath-d... the crux of which is:

Some, including sources inside AMD, have blamed Global Foundries for not delivering higher clocked SKUs. Sure, the clock speed targets for Interlagos were probably closer to 3GHz instead of 2.3GHz. But that does not explain why the extra integer cores do not deliver. We were promised up to 50% higher performance thanks to the 33% extra cores, but we got 20% at the most.

So the real questions seem to be:

Is Interlagos' perfomance disappointing on pure-parallel benchmarks as well, or just those with a noticeable serial bottleneck?

If there is poor scalability, what is the cause?

and perhaps most importantly: Will my kind of workload actually benefit from Interlagos' more cores?

I think that probably what's happening is that even the pure parallel benchmarks are not scaling linearly relative to the previous generation of chips, the reason is a combination of architectural changes in the core itself and memory bus contention, and the target market for these chips is VPS hosts running hundreds of VM images.

I agree with everything you're saying, I just think the 'obvious point' is one that's papered over in the public discourse a lot because it's not exciting.

It is interesting why they don't scale linearly and I agree that it's probably a combination of memory bus contention and/or the chip itself being a little slower.

The whole bulldozer thing seemed like a marketing stunt rather than thoughtful pursuit of higher performance to me. Given the frequency and cost of cache miss rates, Intel HT seems to get the same work done on fewer watts.

I really don't think AMD would have bet their next generation of chips on something they believed was a marketing stunt.

Once the upcoming chip changes get explained to the Marketing Department, however, maybe it kinda starts to sound like Marketing.

Intel HT is often disabled on HPC clusters due to performance problems. Also many people refer to the IL as a 8 core CPU due to the problems mentioned above. Much like not using the HT, many codes disable half the AMD chip.

Yeah, but it's really only on highly-optimized cache-coherent vector/matrix math that it's a problem. On just about every other piece of software, it works great.

I feel like AMD saw the benchmarks available for matrix math and went for it, rather than thinking about what real-world performance profiles looked like.

"the most parallel programs" are those that run on supercomputers / huge clusters, and they don't have these bottlenecks...

But for the average parallel programs, you are (currently) right, I guess.

You've still got a driver program that coordinates the massively parallel stages, most likely, and a less-scalable dispatch process where not all nodes are yet doing work.

Yes, there definitely are programs where this holds true (as I have said), but there are others where it does not. A simple, but perfectly reasonable example is brute-force search for cracking a password (if you got the hash of the password). Your dispatch is simple: send every node 1) the hash, 2) the number of nodes, 3) it's own number. With 2) and 3), each node can easily figure out where it starts. You need no more communication until one of the nodes finds the password.

See https://en.wikipedia.org/wiki/Embarrassingly_parallel

Again, you said "most parallel", and not "average parallel", and this is what I have commented on.

Keep in mind, https://en.wikipedia.org/wiki/Amdahl%27s_law was posed at the very beginning of parallel computing. Seymour Cray later became famous both for his fast uniprocessor supercomputer and his quip "If you were plowing a field, which would you rather use: Two strong oxen or 1024 chickens?"

For most embarrassingly parallel problems, a Hadoop or BOINC-like cluster made from commodity hardware may always be cheapest. But "real" supercomputers and clusters will spend the most money to reduce bottlenecks precisely because they want to run those programs that are the most limited by them.

I know, I actually read the paper like a year ago. Actually, what was turned later into Amdahl's law is more or less a side-note in the paper :)

The parent said "most parallel programs", which I translated to embarrassingly parallel problems.

> But "real" supercomputers and clusters will spend the most money to reduce bottlenecks precisely because they want to run those programs that are the most limited by them.

I agree, but from the talks I have heard most time (= money) is spent trying to work around these bottlenecks as good as possible in software (i.e. incremental algorithms, pipelining, ...) to keep the machines busy. Then again, I hear a lot more talks about software than hardware, so you may be very right. If you have any particular example in mind, I would love to read about it!

My impression (formed from growing up as the annoying kid in the datacenter back when they let annoying kids hang out in the datacenter) is that buying new supercomputer hardware takes takes a lot of money and a really long time. Showing a return for the big investment might take much longer than the life of a typical PC.

Once the a computer is actually in place (especially in a University setting), the interesting work begins, trying to write the best code for it and even improve the existing algorithms.

It seems like there are computer purchases motivated by "this machine will run our existing code faster" and those motivated by "this machine will allow us to write code for it that will prove something faster". A supercomputer seems, almost by definition, the latter.

> even the most parallel programs tend to have bottlenecks

There's a big class of "embarrassingly parallel" tasks, which usually are only limited by how quickly you can move your data to where it needs to be.

What makes something EP is that there is very little data (relative to the CPU time) that needs to be moved.

For example, to compute a nice image of points near the Mandelbrot set, you only need to move the requested parameters (4 floats) to the CPU, let it churn for a long time, and retrieve the resulting pixels (3 bytes each).

I think EP describes a problem, whereas how much data you have to move around is dependent on your implementation. For example, SETI@home is an EP problem that could be run on a multicore machine completely in parallel without moving data. The fact that it is run on many computers over the internet doesn't change that it's EP. In fact, it is exactly because it is EP that it can be run @home.

I agree SETI@Home is very EP. What did I say to suggest it was not?

I'm not arguing that it's EP, I'm using it as an counterexample to your point.

You can solve SETI by moving data around or without moving data around, but that doesn't change the inherent fact that it is a EP algorithm.

But do we agree that if SETI required moving so much data around that it was clearly difficult to keep the CPUs fully utilized then it would not be an EP problem?

To me, something is EP if it has no or few dependencies between data, regardless of how much the data does or doesn't need to be moved around.

Yeah, that's probably more important. But they're related and the amount of data is not completely irrelevant.

Say instead of the Mandelbrot set we're talking about generating large vectors of random numbers. This task should obviously have no data dependencies and there are algorithms that can generate random numbers much faster than an external CPU bus can transfer them to memory.

If raw bandwidth limitations mean that our CPUs see only 10% utilization is this still a "fully EP" problem? Because it should easily achieve full CPU utilization on most any machine, couldn't we say that Mandlebrot is more EP than the RNG problem?

I would say that the RNG problem is EP in that each data item can be processed completely in parallel with the others. The fact that there is a memory bottleneck doesn't really say anything about the nature of the problem but rather the hardware its running on.

BTW, thanks for the really interesting discussion!

Reply on news.ycombinator.com