Hacker News Reader

Comment by adamnemecek | Hacker News Reader

Julia is next gen. The whole experience is so much more pleasant than anything in the space. The interop is also crazy. How is it possible that you can call Matlab, Python, cpp, fortran, Rust from one language?

I legit think that even if you are using say pytorch. You are better off writing your code in julia and using the python interop.

View on HN

Using pytorch/keras/TF from Julia is possible, and even pleasant: https://twitter.com/aureliengeron/status/1277751121440698368

Though I much prefer Flux.jl

In my experience Flux.jl has a ways to go before it reaches Pytorch performance. Apparently this has something to do with some very efficient GPU kernels that Pytorch uses - I wonder if it's possible for Flux to "borrow" those GPU kernels? It's possible to write GPU kernels in Julia so hopefully this will help close the gap.

The short answer is yes, it's possible and easy. The longer answer is, that's a bit of an apples to oranges comparison. Pytorch is calling optimized c++ kernels, whereas Flux is completely generic and can do GPU and CPU codegen (including AMD). Flux can do things Pytorch just doesn't have CUDA code for, It's more generic and composable and it's getting there with a much smaller team/budget. That means it has more potential, with the drawback that it isn't quite as fast, for now, in some of the common use cases the pytorch has been hard coded/ hyperoptimized for.

In other more complex cases beyond big fat matmuls and such, it's much faster, even comparing to torchscript (sciml, neural ODEs etc).

Still, the codegen is getting there with the funded work on the GPU compiler and memory allocation and there eventually won't be any sort of tradeoff.

In the meantime, Yes, there is a package at allows calling pytorch kernels without modifying your code. This is also only possible with such finesse due to Julia's power and composability. See here for the package: https://fluxml.ai/2020/06/29/acclerating-flux-torch.html

Flux already has the ability to borrow Torch kernels. This was released fairly recently.

https://github.com/FluxML/Torch.jl https://fluxml.ai/2020/06/29/acclerating-flux-torch.html

Also the gap between native GPU codegen in Julia and the hand-optimized kernels in PyTorch is lesser than you'd imagine.

In fact, see this talk at JuliaCon, which compares Julia on a 1000 GPUs against a CUDA/C implementation, and finds them to be extremely close.

https://www.youtube.com/watch?v=vPsfZUqI4_0 https://github.com/eth-cscs/ImplicitGlobalGrid.jl

Is this just wrapping the torch kernels or the entire torch API? I was under the impression it was the latter.

I feel I should point out that if you need performance and you're mostly only interested in classic PyTorch or TensorFlow style machine learning (i.e. you don't need Flux.jl's characteristic flexibility), then you should really check out KNet.jl [1]. KNet is generally around the same flexibility and performance as something like PyTorch. It doesn't get nearly as much attention as Flux because it's less novel, but it really should be looked at more.

[1] https://github.com/denizyuret/Knet.jl

Knet.jl is pretty awesome. The main challenge has been compiling its cuda kernel library. Deniz has helped anyone who ran into issues.

However, we are integrating a lot of the underlying infrastructure to the point that both Knet and Flux will only differ in API approach.

It really depends on where you're looking. We tested neural ODE compatible solver benchmarks from 3 ODEs and about 1 million ODEs and saw an advantage for Julia of >30x across the board [1]. Then showed the spiral neural ODE from the original paper train 100x faster in Julia [2]. So it should at least be qualified as, PyTorch has some good kernels, and it has some things it doesn't optimize close to as much as Julia.

[1] https://gist.github.com/ChrisRackauckas/cc6ac746e2dfd285c28e... [2] https://gist.github.com/ChrisRackauckas/4a4d526c15cc4170ce37...

Yeah, the quality of the language is pretty incredible. Especially compared to Python, which has been a disaster for as long as I can remember.

I think the ecosystem could be a bit better, but that's more of a personal opinion rather than a serious concern as, like you alluded to, you the interop story is insanely good.

The ecosystem is moving very rapidly though. What do you think is missing?

Right now, the DataFrame package doesn't support row indices like Pandas does. Row indices are especially useful time series data, and there's another package called TimeSeries that support a dataframe-like type called a TimeArray that supports dates as row indices.

TimeArray is like a DataFrame, but with a lot less features. You can't really do that much with them, and they only have limited support in an auxiliary package for resampling.

It would be nice if a new DataFrame library could be created that was inspired by Pandas.

In general, I think Julia suffers from the problem of too many small packages. For example, there's 3 libraries I have to use for time series: TimeSeries.jl, TimeSeries Resample.jl,and TimeSeriesIO.jl. None of the libraries are particularly large and should all be merged into one package.

Part of the reason for the dirth of small packages is because, unlike Python, Julia's strong support for function polymorphism allows for seamless extension of existing functionality with new packages. Unfortunately, I think this has created a cultural problem in which new packages are created that only slightly modify existing functionality.

One of the discussions on Juliacon was exactly this! How to index various things. This was one of the documents produced during the discussion : https://docs.google.com/document/d/1imBX3k0EEejauWVyXONZDRj8...

Please don't be "inspired" by pandas.

No, you're right. Pandas totally sucks. But the one good thing it got right is support for row/multi-index.

I think Python is less a disaster than it is a tool being used for something it was not designed to do.

It's a pretty good scripting language, and it's a "good enough" interface that implementers of C packages can expose powerful features, and that can only go so far. User's can't easily compose the powerful features that library designers didn't intent.

Julia, on the other hand, was designed for the explicit purpose of high-performance scientific computing, with a syntax informed by the tools people already knew (python, MATLAB). It's no wonder it's so much easier to use.

Reply on news.ycombinator.com