Neato! I'm surprised they went with a JIT compiler over a full-on compiler, but that might just be me not understanding:
a) Compilers
b) How a JIT compiler would apply to this situation
My lab-mate Jan Gosmann recently did something similar for our spiking neural network software Nengo [1]. Although it isn't Deep Learning, it also builds a computational graph of operations. He ended up optimising the layout of the operations in memory to increase the efficiency of Numpy operations and reduce the amount of time spent in Python. He's in the process of writing a paper about it.