Introduction to Parallel Computing | Hacker News Reader

Introduction to Parallel Computing

computing.llnl.gov

41 pointsr11t13 comments

This is an excellent resource for learning parallel programming, on several levels. Don't miss these related tutorials from the same source:

* Pthreads: https://computing.llnl.gov/tutorials/pthreads/

* OpenMP: https://computing.llnl.gov/tutorials/openMP/

* MPI: https://computing.llnl.gov/tutorials/mpi/

I liked the way the content is presented. I had Parallel Computing as one of my elective this semester, and the subject isn't kool any more when you dive into it. Parallel Computing is tough IMHO and it takes some brainstorming to solve even the basic problems like computing prime numbers or sparse matrix multiplication. Also we use OpenCL, which itself is quite a complicated deal. ( TBB, OpenMP are simpler ). The tutorial makes the basics clear as well as the post have an aesthetic feel to it. All in all - a nice read.

Stupid question (I know embarrassingly little about GPGPU programming). Is OpenCL mostly used with ATI cards? How common is OpenCL? It seems to me that almost everyone is using CUDA.

Also, isn't copying everything to the GPGPU memory and then performing tasks there (and when it is completed copying it to main memory) a bit stupid? What happens in a cluster setup when two or more hosts work on the same dataset? Won't this result in a needless amount of copying? Double memory requirements?

OpenCL is different from CUDA in two important points:

1) OpenCL is an industry standard supported by many vendors (although not all support it fully currently, since it's a relatively new standard). OTOH, CUDA is a proprietary system developed by NVIDIA for its GPUs. CUDA is much more mature since it has been in use for quite some time now.

2) While CUDA addresses GPUs, the aim of OpenCL is more general. It's a "heterogeneous" language/platform, aimed by design at various computing devices: CPUs, GPUs, special CPUs like Cell, accelerators (FPGA, etc.), DSPs and so on

CUDA actually is open, but ATI chooses not to implement CUDA support in their drivers.

APIs sorted by complexity (from most hard to learn to easier to learn):

  * OpenCL / CUDA
  * Pthreads
  * MPI
  * TBB
  * OpenMP

I consider message passing programs harder to learn and develop scalable applications for than shared memory. While the errors in a shared memory program (this includes Pthreads, TBB, OpenMP - any programming model that uses multithreading in a shared address space) can be more subtle, you don't have to have as deep an understanding of the runtime behavior of your algorithms. It's easier to rely on shared memory in hardware than to use explicit send/receive on all of your data.

I disagree, in MPI you don't deal with locks and race conditions. It's like saying, that Erlang programming is harder than multi-threading in C/C++. I'll take Erlang over Pthreads any day.

MPI applications can be a little bit harder algorithmically, but a concept of Collective Communications (i.e. send/recv/scatter/gather/bcast/reduce) is easier to understand than threading and locking.

The whole concept of Shared/Distributed memory models doesn't mutter in this case.

There's two issues: debugging and scaling. I agree that multithreading at the Pthread level is harder to debug. But I think it's easier to design a program that will scale using Pthreads than MPI because with one you rely on the hardware to distribute data, and with the other you have to do all of that yourself.

According to this logic MPI for Shared Memory will be easier to understand than OpenMP for clusters.

You'll have to explain that, I don't follow. I haven't brought up OpenMP since we're both in agreement it's easier to develop for than MPI.

My point, in case it is not clear: having to explicitly send and receive data and achieve scalability is inherently more difficult than relying on shared memory. Getting a working program may not be difficult relying only on message passing. But getting a working program that scales requires a deep understanding of the data needs of your algorithms.

Of course, message passing programs have the potential for higher performance because they're not limited to a single node. But realizing that potential is harder. And I'm not including debugging, where I agree that shared memory is rife with problems.

the website is a little hokey but I found the tutorials/mini courses on MPI and OpenMP on http://ci-tutor.ncsa.illinois.edu/browse.php very useful

here's a link to the current course I'm taking, in case anyone is interested in working through some assignments. http://dna.cs.byu.edu/CS484/schedule/schedule-winter2010

It uses the llnl tutorials for most of the reading.

Reply on news.ycombinator.com