travisoliphant

Bokeh can be used with both Python 2.7 and Python 3. Additional dependencies are needed to work with Python 2.7. In fact, the "write call-back functions in Python" capability uses a Python to JS compiler called Flexx -- http://flexx.readthedocs.org/en/latest/flexjs. Flexx is currently Python 3 only (though 2.7 support is coming for it).

There are a lot of smart people that like Python. There are a lot of smart people that like R too. Bokeh is designed to be useful from both R and Python (and other languages as well).

Here is a recent presentation showing Bokeh (from Python) along with data-shading to visualize billions of points quickly: http://www.slideshare.net/continuumio/visualizing-billions-o...

Here is another presentation showing integration of Bokeh with R to visualize geographical data from another group: http://ryanhafen.com/blog/rbokeh-gmap

0.11 and above releases of Bokeh contain a server that provides a very nice application model that can be synchronized between browser and server for allowing data scientists to build interactive plotting-based applications in the browser. Here are some simple examples of that: http://demo.bokehplots.com/

You might take a look at Bokeh (http://bokeh.pydata.org) and either the PyData stack or R (Bokeh can be used from R as well: https://github.com/bokeh/rbokeh). Bokeh inside a Jupyter notebook with widgets and/or emerging "Bokeh Apps" is a powerful application stack. Anaconda is a single download that can help you get started with all the tools (including R): http://continuum.io/downloads

It still requires some coding but it is very powerful. There are a lot of examples in the Bokeh gallery and in examples directory: https://github.com/bokeh/bokeh/tree/master/examples .

There are several devs on Bokeh mailing list eager to help and the company behind Bokeh (http://continuum.io) can provide more significant help if you need it.

Yes, LLVM is a great approach for doing code-gen from arbitrary specialized VMs. And the Python interface to it makes it easy to experiment. We no longer use llvmpy for Numba (we use a simpler interface llvmlite) and so llvmpy could use a maintainer.

The numba code-base implements quite a bit of this. We actually moved away from the AST approach and went back to the byte-code approach because the AST approach quickly becomes unwieldy as the number of Visitors that you apply grows. Compile times are also slower.

The author is definitely helping people learn about LLVM and how it can be used with Python --- which is great, because this is exactly what Numba is: http://numba.pydata.org. But, please don't start another "Numba". Just come help us improve the current one.

This is a great tutorial about first-generation Numba. The author learned a lot about LLVM and llvmpy while working with several of our devs. If you are interested in the "Further work" in his article, come join the Numba project.

This is an interesting idea.

However, their timelines are too short. It makes me feel like they don't really understand what it's going to take.

A good start would be the C++ library libdynd which was designed to potentially be used from Javascript.

NumPy has much more than vectors and matrices. Does Javascript have operator overloading?

The performance numbers they show are fairly cherry-picked and are not really showing a broad-based comparison of Numba's capability. In addition, Numba is still under active development and the version they used is a few versions older than released. Still it's good to have use-cases and examples which drive Numba improvements.

Interesting post. It seems that everyone wants to write their own jit rather than cooperate with the on-going open source efforts. 1) Support for recursion is in a PR for Numba that has just not yet been merged. 2) optional simplification of expressions using SymPy has been contemplated and is easy to add, though it's not clear if it's the interface people will want as it slows down compilation. 3) Numba has support for some array broadcasting via vectorize and is getting more in the coming 6 months.

A C++ target for Numba would also be of interest if anyone wants to participate in the development. Numba 0.15.1 was just released this week: https://github.com/numba/numba/tree/0.15.1 http://numba.pydata.org

conda install numba

The link to the "non-store" download page which we tell everyone about is: http://continuum.io/downloads.

Yes, we are a company and do sell things, but our commitment to open source is backed by many, many years of significant contributions to the community. We give as much away as we can, and sell to those who need more.

While you certainly buy more from us, Anaconda itself will always be free as long as we can continue to make it.

This is a nice post, and the Python packaging community is making strides in the right direction.

However, it's not there yet. Armin could bring himself to say this in this post: "It's there, it kinda works, and it's probably worth your time."

Conda is not perfect (it needs more people building conda binaries as well as a few pull requests to add https improvements) --- but I will confidently say "it is there, it does work now, and it is worth your time"

This is especially true with a package repository like exists at http://repo.continuum.io and the many personal repositories at http://binstar.org.

People that have used conda for deployment (especially of complex C-dependencies in the NumPy and PyData stack) are saving time and effort today.

You can also try conda's environments which are a system-level concept and is an even lighter-weight approach to some of the problems people are using Docker to solve. If you just need an independent software environment, conda gives you that while sharing files that can be shared. http://conda.pydata.org

Yes, thank you!. You have the correct link.

A lot of people in the PyData ecosystem using NumPy and Pandas are using conda and conda packcages (http://conda.pydata.org). Conda packages are easy to build and many exist at repo.continuum.io and on channels at http://binstar.org

Conda (http://conda.pydata.org) is a more general packaging format than wheels and we use it to distribute the Scientific Python Stack (including complex C-dependencies) with ease. It is more general than wheels and solves this fundamental problem now with all the benefits without waiting for something in the future. See these blog-posts for more information: http://www.continuum.io/blog/conda_packaging and http://technicaldiscovery.blogpost.com

In particular look at the `conda package` command for an easy way to bundle up any non-conda packages across a homogeneous environment.

Using conda to build and deploy binary dependencies. Create system-level virtual-environments with ease. Anaconda has jump-started providing binary packages for several platforms.

Getting a conda package from can be as simple as

`conda build --build-recipe <pypi-name>`

I'm looking forward to hearing from Brian Granger (IPython) and Peter.

And by free, he also means free to re-distribute (attribution license). It comes with a very nice package manager (equivalent to brew, yum, apt-get, etc --- but with integrated virtual environments). This package manager BSD-licensed and free to use outside Anaconda as well.

This is 100% true. Hadoop gets way too much attention given the other useful solutions that exist out there. I have known people to use Disco successfully on several hundred node cluster. You can also interact Disco with IPython parallel much more easily.

This is why we include it in Linux versions of Anaconda.

It's hard to see how you could have read the article given your comment. This is certainly not a copy of some original presentation. The article may not be motivated well and certainly can be critiqued on many levels.

But it describes a specific reason why using a high-level language to directly program the GPU can be extremely useful --- you can easily build, test, and iterate on execution order to improve performance. Hardware is changing, and we need better tools to write code for it.

The author uses CUDA Python, but you could do similar things with PyCUDA --- it's the emphasis on the scheduling that is the relevant point.

But, the upshot is that Numba produces code that is either faster or roughly the same speed as C or Cython.

Yes, there are several. See slide 15 of this talk http://www.slideshare.net/teoliphant/numba-siam-2013 and also the github repo: https://github.com/teoliphant/speed

Note that the array-expressions previously in numbapro only have been moved to numba (our plan for all premium products is to move features into open source as we get funds to support that).

Numba's original purpose was not to speed up concise, optimal NumPy code. The fact that it can actually do that is an extra bonus. Numba's purpose was and still remains best at allowing you to still write the "hard-to vectorize" algorithms that require for-loops without having to use C/C++ or Fortran.

If you never are resorting to using C/C++ or Fortran, then Numba is not necessarily going to be helpful for you.

I don't think we are trying to mislead. We are simply showing examples that illustrate what Numba can be used for. Of course your code does not have many for-loops in it because you never could write code like that and have it be fast in Python. The big-deal about Numba is that now you can. You don't have to rely on a C/C++ library that either write or get someone else to write for you. You can just use Python.

I agree with you about the "unrolling". I don't recommend adding for-loops when good vectorized expressions work. Keep using them as we will be able to do a better job of parallelizing that code in the future. However, when you can't vectorize and for-loops are all you have, Numba is great.

We have spent some time on optimizing NumPy expressions like you describe and some array expressions can be translated down to equivalent LLVM loops. But most of the effort in that direction has been happening in the Blaze project which is generalizing NumPy so it can deal with much larger data but still produce optimized code from those vector expressions. This actually requires a re-thinking of how NumPy is implemented from the ground up.

Right now Numba is still pre 1.0 and so there will be issues for specific cases. The best way to get them solved is to provide your test-case so that we can grow the test suite.

Right now, I agree that Cython and pure C++ is still the best approach if you have a case which Numba does not yet quite solve. However, I expect this to change over the next 6-12 months as Numba is getting a lot of attention both inside of Continuum and externally at several industrial-strength locations.

Our goal is that Numba can replace (almost all) Cython and raw C-extensions or even Go-extensions faster than other language approach will be able to build a run-time and ecosystem the equivalent of what is available with Python.

The roadmap (which includes Go-like channels and go-routines) as well as compiled class-support is here: http://numba.pydata.org/numba-doc/dev/roadmap.html

The project is open-source and contributions and contributors are welcome to join.

Numba is trying to let you make a different choice for different sections of the same program so that you can do different styles of programming in code meant to be a library than you might do in other sections of code meant to be more "ocassional programmer" facing.

Some of you might find the roadmap particularly interesting. It's an open-source project eager for input from others...

http://numba.pydata.org/numba-doc/dev/roadmap.html

Memoryviews are the step-child of NumPy arrays. The buffer-protocol was the real intent and the memory view the forgotten "example" until a few brave and noble Python devs rescued it from obscurity. Memory-views are not that useful to one who will always have NumPy installed.

I'd love to see how Numba helps you. Give it a try: http://numba.pydata.org

Slide deck: https://speakerdeck.com/pyconslides/numba-a-dynamic-python-c...

IPython notebook examples: https://www.wakari.io/nb/travis/numba https://www.wakari.io/nb/travis/Using_Numba

This is exactly what Numba is designed to help with. And it already works extremely well for such cases. Numba lets you have ease of use (un-typed containers) and speed (with typed-containers) in the same run-time and the same syntax. Here is are some IPython notebooks which illustrate:

https://www.wakari.io/nb/travis/Using_Numba https://www.wakari.io/nb/travis/numba

And the slides from PyCon Talk: https://speakerdeck.com/pyconslides/numba-a-dynamic-python-c...

I wish this were true, I really do, but it's just not true. Maybe for python-only packages. But, for the myriad of python extensions necessary for Scientific Computing with Python, pip and virtualenv are not enough.

We think conda is a useful tool, and have freed it for use by everyone.

Perhaps something like pip + a better virtualenv + wheel would be close, but we needed something that works now.

Cython is helpful, but you have to spell out a lot of type-information that is not specifically necessary. You might also try Numba --- easiest way to get it is via Anaconda CE or Wakari. Both at http://continuum.io