Threads, processes and concurrency in Python: some thoughts
artima.com
artima.com
My big problem with event loops is that while your framework may be all asynchronous, your code will not be. Try to make a call to MySQL and you will block the entire process. On the other hand, in this case your thread will just be put in a sleep mode and another thread will take over. Yes, there is overhead, but depending what you are doing, threads can be a great tool.
Also the conclusion is really weird; concurrency will be hidden away from programmers even more than it is now, but it will take the form of message passing and event-loop programming? That's hardly hidden, that's fundamentally rewriting how you write programs to work with concurrency. It seems to contradict his entire thesis. The details may be hidden but they already are; semaphores, for instance, are already an abstraction.
Really? I see lots of change, I look at problems and solutions differently and care if a piece of technology can use multicores. Just because you keep your head in the sand doesn't mean multicore isn't changing the world and leaving you behind.
Is this the standard/accepted/correct definition ? Or he just made it up ?
My understanding is that parallelism is concerned with actual execution of the code in parallel on the machine. Whereas concurrency is concerned with the semantics of the language, i.e. whether it is possible in the language to define multiple "tasks" that need not be run in serial order.
Whether those tasks actually run in parallel or not, is a concern of parallelism, not of concurrency.
Parallelism implies concurrency; a parallel program has threads of execution (which may not actually be shared-memory software threads) that are executing simultaneously.
The author's definition of parallelism is wrong - at least, it's different from the one myself and everyone else I know in the high performance computing and systems research community use. Many interesting problems are not embarrassingly parallel, and figuring out how, exactly, such problems can be solved efficiently in parallel is hard. (And efficiency is a solution requirement, since it's not worth the parallelization effort if you don't achieve it.)
I think the point the author is trying to arrive at is parallelism does not necessarily mean shared-state across threads. Which is true, and I share his sentiment that we should probably not reach for shared-state threads to solve our parallel problems.
"Concurrency should not be confused with parallelism
Concurrency is a language concept and parallelism is a
hardware concept. Two parts are parallel if they execute
simultaneously on multiple processors. Concurrency and
parallelism are orthogonal: it is possible to run
concurrent programs on a single processor (using
preemptive scheduling and time slices) and to run
sequential programs on multiple processors (by
parallelizing the calculations)."
-- page 25 of Programming Paradigms for Dummies: What Every Programmer Must Know a book chapter by Peter Van Roy http://www.info.ucl.ac.be/~pvr/VanRoyChapter.pdf [Pdf]Here is SO's take http://stackoverflow.com/questions/1050222/concurrency-vs-pa...
and from GHC Mutterings http://ghcmutterings.wordpress.com/2009/10/06/parallelism-co...
You write parallel software (use parallel algorithms), when you want to exploit hardware parallelism (usually multiple CPUs) to do one thing. While you write concurrent software to do multiple things at once, that require same resources (CPUs, Memory, etc.). There is theory of parallel algorithms that is concerned with extending of big-O notation to multiple processors and performance effects of inter-processor communication, while concurrent algorithm theory deals with synchronization problems.
In essence one can illustrate the difference by looking at synchronization primitives used: while concurrent code mostly uses locks, parallel code uses barriers. And on many parallel supercomputers of eighties, synchronization of parallel code was implicit, because all processors had common clock and thus you could do all synchronization by simply knowing that what given processor is doing when.
This is relevant to this "threads are useless" debate by the fact, that threads are meant for concurrency and not parallelism, because they implicitly share resources (memory) that is mostly not required to be shared. Large part of complexity of modern multiprocessor hardware (and operating systems) comes form effort to detect which part of memory is shared and which is used only by one processor (or thread). Because sharing whole memory array incurs non-trivial contention and precludes you from doing any meaningful caching.
Ideally definitions shouldnt only be essentially the same (for some values of essence), but obviously/unraguably the same. Or else we discuss definitions :-)
The point I was trying to make is simple. By conventional terminology: Concurrency deals with semantics, parallelism deals with execution. I dont see how
parallelism as splitting a computation into many independent tasks which interact very little,
and concurrency as modifying things from different threads/processes/tasklets/whatever without
incurring in hairy bugs.
translates into the standard definition. What the author describes as parallelism could easily be concurrency if the split happens at the semantic level.BTW concurrency needn't have any explicit synchronization primitives at all and needn't have anything to do with synchronized clocks. Semantics does not care about execution or clocks it cares only about meaning. Since parallelism deals with execution, there clocks could become relevant.
Consider list comprehension in Python. Its a concurrency primitive. But whether the elements of the list are computed simultaneously, depends on how the Python code is executed and hence an issue of parallelism.
"Concurrent code" and "parallel code" terminology can also be confusing. If a piece of code describes/defines things you want to compute, then it can only be concurrent and not parallel. But if the code also describes how it is going to be executed in the hardware then it could be called parallel. So I am not confident of the view
threads are meant for concurrency and not parallelism
I have no difficulty in accepting that threads are not always the right abstraction. Shared mutable state is a significant problem.But forwarding that argument by conflating conventional terminology as was done in the article hardly helps.
The article is very clear about this:
"I claim that essentially nothing will change for the average programmer with the advent of multicore machines."
He does not deny that people writing web and database servers do need to care about multicore CPUs. And I guess it is mostly true, so many daily tasks are I/O bound that the only concurrency most programmers will see are separate GUI/worker threads.
Of the tasks where parallelism does help a lot, a substantial subset is embarrassingly parallel. E.g. we parse a lot of large corpora (Wikipedia-size and larger). Parallelization is simply obtained by chunking a corpus, and giving every CPU in the cluster a chunk. Then there are number crunching applications with data dependencies where just a few OpenMP pragmas can easily parallelize programs. Of course, there are remaining hard tasks which require more hand waving.
And even as a web developer, I often write more than just something I throw at the web server. Many web applications require cleanup scripts, batch processing of messages or static content, or perform some kind of statistical analysis on their data models. Given enough data, these are all areas where multiple cores can be relevant.
But may be hackers weren't the target audience in the first place.
There is no reason to believe that HLLs have to be particularly inefficient (for example the scheme compiler stalin can hold its own and often run faster than an equivalent number crunching code written in c). I am not arguing that Python should be that high level language, but that the current state of "divide" (c for programming servers python for apps) is a little artificial and is a result of a lack of options. Not grappling with parallelism only makes the scene a little more entrenched.
I find X10, Fortress and Chapel quite exciting as they are trying to straddle this gap.
I think we'll need to add models of parallelism to high level languages, and most programmers will have to deal with it. But, that's not a given. This author is on the other side: maybe the majority of programs will be able to remain sequential, and we can allow the exploitation of multiple cores to happen mostly at the OS level through the scheduler.
Time requirements are constantly getting smaller, and computing requirements are getting larger. Serially generating data, even just for web pages, won't work forever (and I'm willing to hazard a guess that for the major players, it doesn't work now).
Multiprocessing, while a great advancement for Python, is not what I would consider lightweight. It's not a problem for the dual- and quad-core computing era, where you have a few number of spawned processes. However, if you go to a 32 or 64 core era, and the ability of multiprocessing to scale will be dramatically affected by the memory and processing overhead required.
We're fine now with the tools we have. However, if these tools do not evolve, we won't be fine in the future.
^ Disclaimer: parallelism/concurrency is far from new though.