Hacker News Reader

Comment by garyrob | Hacker News Reader

" I mean, faster than r or Python (for which I could write fast code but that'd mean I'd have to change the way it is written)"

You might benefit from numba. I've used it to speed my Python up enormously and completely painlessly just by adding decorators to critical functions. It's why I'm not considering moving to Julia.

View on HN

There are so many ways to write R that can actually be very fast. R is a very different language as a whole and it has addressed a lot of problems these last 10 years. I might be biased but I really do like the functional side of R and how logical the libraries from Hadley Wickham have been designed.

Python still doesn't feel like a natural fix for data science work. I am guessing it is more bias opinion but why based on 0 for this domain????

> There are so many ways to write R that can actually be very fast.

But it means that you have to work with arrays. For me it often means breaking the flow of my (code) explanation to migrate to other data structures. Sure it is then fast, but it gets less readable and harder to update.

Now, I'm a programmer at heart, so I think in the "functional" paradigm, not the array/signal one. There's some "impedance" I guess :-)

I actually prefer the 0 indexing of python. Systems code (c/c++ etc.) already uses 0 based indexing. So it is nice that when you do data science the convention stays the same.

But Fortran, R, Matlab and other tools in the domain use 1 indexing...

Does it actually matter at this point? I write off by one errors all the time in JavaScript and I usually spot them immediately. I’ve written Julia and R in the past and I found it really easy to switch between contexts, and if I forget, the errors are such that it takes only a few seconds to spot and fix them.

Is it actually completely painless?

YMMV, but for me I found that there were many unsupported parts of Numpy I had to work around which meant I was effectively doing a full rewrite.

Especially assignments using boolean masks and working with multi-dimensional arrays in general is really tough

I think Numba has some pain points. I have also encountered issues with multi-dimensional arrays.

The beauty of Julia is that relatively naive Ruby-like code is already quite quick. And if you implement inner loops in an imperative way, with an eye towards not generating excessive allocations, it can approach C++ speed while still being nice high-level code that is close to mathematics or business logic.

Besides, the other strong point of Julia is composability. The ecosystem is made up by lots of small libraries that can interact in ways the original designers did not expect or plan for. In contrast, Python has exceptional libraries, but they tend to be big monoliths.

The problem with Julia right now is that some libraries are not sufficiently mature. For example, there's no mature native replacement for XLA or PyTorch. I know about Flux, but it's nowhere close if you wanna create, say, a large transformer. Or say you are working with GLMs. GLM.jl is nowhere close to R.

Some other Julia libraries represent the state of the art, though. I just can't wait to get all foundations complete! It's a really promising space for probabilistic and differentiable programming.

It just happened wasn't using Numpy except in the most basic possible ways. I had hand-written algorithms in python. That's where it shines. So, yes YMMV! It worked really well for me.

Numba is awesome. It solves a lot of problems. Julia is like numba but you are able to use any libraries inside the loops. Think FFT, scipy, etc

For simple things only. For example, the jit class in Numba is still experimental, and quite limited. So if you need to write non-trivial class, even if you jit each method, dropping to Python can be a problem (say with lots of instances.)

I have tried to write an application targeting HPC and in itself is not very complicated (probably ~2000 lines or in that order.) But I did things like using the Python language as the metaprogramming language for Numba (basically higher-order function where you jit inside.)

All in all my experience of Numba tells me that if I am designing the same package now I'd write it in Julia where jit is "first class" and you don't need to constantly thing about the boundary between Numba and Python.

I’ve become a big fan of Numba. No more awkward, human-unreadable vectorized code.

IMO, numba loses pretty much all of the advantages of python. The code is fast, but you lose the ability to organize your data the way you would in regular python, and most of the python ecosystem can't be called from within numba functions. If I wanted that developer experience, I would just use C.

I don't think that's true. You lose some functionality, but since you can call Numba on select critical methods, it's not so bad. You can often sequester your performance-critical logic into some Numba-decorated methods, and your business elements that call out to the rest of your ecosystem are not decorated with Numba.

Or to put it another way-- if I'm using vectorized code in Numpy I can't deal with the external ecosystem from within my vectorized code either.

> The code is fast, but you lose the ability to organize your data the way you would in regular python,

Can you clarify what you mean by this? I lose the ability to organize my data the way I'd like if I'm vectorizing my code too.

I wasn't comparing numba to numpy. I was comparing to python code where you don't care about performance. The main reason I don't find numba appealing is that Julia gives you numba like performance while allowing you to use structs (think classes) to organize stuff.

This doesn't really make sense. If you don't care about performance then you have no use for Numba.

Also, the comment you replied to was explicitly comparing Numba to vectorized Python, so you should not abandon that comparison in your reply without saying so.

I think the point is that one wants to write code that is similar to regular non-performance sensitive python code, with classes and everything, and still have it be fast.

> Also, the comment you replied to was explicitly comparing Numba to vectorized Python, so you should not abandon that comparison in your reply without saying so.

The comment you replied to explicitly says "regular python".

Not true at all for me, because in my situation there are some functions which have tight loops that take 99% or more of the execution time. Put the numba decorator in front of them and the code is 100+X faster. Clearly it's not that simple for everyone. But for some people, it can be. So it doesn't make sense to call it a panacea, and it also doesn't make sense to say it doesn't work. It works for some people, not for others.

Reply on news.ycombinator.com