Passing the torch of NumPy and moving on to Blaze
technicaldiscovery.blogspot.ca
technicaldiscovery.blogspot.ca
As someone working to build tools in the same space as Continuum (and perhaps as a competitor), having your competitors (Continuum) be intelligent, nice, interesting folks who really understand the problem domain is pretty darn great.
Point being: the numerical computing / data analysis landscape is going to be seing a lot of great tools emerge and/or mature over the next year, and I have no doubt that 30-50% of them will be coming from Continuum Analytics. [edit: to the substantial enrichment of high level tools for extending numerical within python / and likely generally!]
I can only hope that I execute my tool building work at WellPosed well enough that I can call them a competitor for years to come!
Real threads will never come to Python. MPI is a real pain unless you are running very large computations. This will only become more true as time progresses. Am I missing something?
Many numpy primitives are already parallel since it basically just hands off to your BLAS library. Beyond that there is numexpr which is really good at doing parallel evaluation of large array expressions. If your problem isn't solved by any of these, there are other powerful solutions like IPython, Parallel Python and even multiprocessing from the standard library
If you need even more performance, cython has some support for semi-automated parallelization, and if all else fails drop down to C and use OpenMP or whatever else you like.
So while concurrency is a problem in python, numeric parallelism is an area where many good solutions exist.
Python people have their heads in the sand! If we have a hundred core processors, running Python on a single core is not going to be a tractable solution to any problem. Your BLAS may be parallel, but any time you go back into the Python driver code suddenly it's a massive bottleneck.
Distributed Python is a messy hack that wastes all the amazingly tuned shared memory support in the processor. Writing C extensions goes against the whole point of using Python.
"if all else fails" The problem with Python is that the moment you want to do something in parallel, which in the next decade will be everyone, "all else fails" is your starting point!
"all else fails" is the starting point of pretty much everyone doing real work. What's your alternative here? Most of the time, specialized libraries will be both more convenient and more efficient than rolling your own with fine-grained concurrency/parallelism in Java or PyPy.
Now imagine that you speed up the language by 2x but have to forego the use of efficient C code. Now, the program would spend 39sec. outside the "kernel" stuff and 110sec. in the stuff that used to be a C library but had to be reimplemented.
Then again, even if you consider a "one language" design such as Cython (where you can write code that's between Python and C, both convenience-wise and performance-wise), performance-sensitive code looks markedly different than straighforwardly writing down a program.
This is why the "two language" design survives, even while you see very usable work in pure-C++ or even pure-Java.
If you're willing to try another language altogether, Julia [http://julialang.org] is a general purpose language with enough performance and expressiveness to be an effective abstraction layer for numerical programming – you never have to dip into C for speed, scale or control. In developing the language, we haven't allowed ourselves to resort to C – instead, we've worked at making Julia itself fast enough to implement things like I/O, Dicts, Strings, BitArrays (packed 8 bits-per-byte boolean arrays), etc. – all in pure Julia code while getting C-like performance.
For some workloads, the latter is actually a good advice, but for my typical use case that does not help. These would be tight'ish loop wrapped around a fork-join. Shared memory handling can be quite clunky in numpy, and if you want to do message passing, the overheads bleed off any advantage that parallalelism ought to have given you. I dont mind the message passing abstraction, just that the overhead for doing it in python/numpy is too much. About the former, one major motivation to use numpy et. al. was to not use C with its explicit indexing over arrays. Its both verbose and error prone.
It is never pleasant to drop into a different language, though it is much much better than how bad it could be, thanks to Swig, Cython, Weave. Contrary to common wisdom I prefer Weave because of its much succincter syntax. In Cython I am back to writing C again but with a different syntax. This is not a criticism of Cython, its an excellent tool and it is much much more pleasant to parallelize from Cython than from Numpy/Python.
Julia looks pretty good. I have one suggestion: The best way to get speed out of Julia is not to write vectorized expressions but to writeout explicit loops. Thats a little unfortunate because though vectorization constructs evolved out of the necessity to avoid loops (which was slow in the older languages), it did have an excellent byproduct of succinct code. Ideally I would like to retain that.
And then I discovered that it too has no reasonable shared memory parallelism story, just the same manual distribution of arrays plus multiprocessing that exists in Python.
I will speculate that Julia's authors have the same attitude as many in the Python community -- namely, that there are small jobs, which can be run in one process, and large jobs, which need to be massively parallelized, and nothing in between. But in reality there are many scientific tasks that are medium-sized, for which OpenMP-style solution is the best fit. Tasks which might take days can be reduced to hours. With new developments like Xeon Phi, that ratio might further improve.
Also many problems require a lot of heterogeneous shared state, and it is tedious to manually distribute each element in this shared state. Finally there are many problems, such as natural language processing, that are only partially numerical. For these problems, distributing arrays is only part of the solution.
I'm following Numba and Blaze with interest and honestly consider Julia the most exciting new language out there. But until they reach a point where they are useable for me I'll keep using python and the incredibly powerful, if slightly kludgy, solutions it offers.