ND4J: Scientific Computing on the JVM | Hacker News Reader

ND4J: Scientific Computing on the JVM

github.com

Hopefully this can pave the way for deep learning and data analysis for Scala/Java. It's really frustrating having to use a subpar and inexpressive language like Python because of the huge data science network effect.

Or if Julia could get more libraries (it's maybe 25% there, which isn't bad), that would be pretty great as well.

Not to mention it's a nightmare to scale...

How is java expressive when you don't have operator overloads?

Java libraries can easily be used from any JVM language, including many that have operator overloads and much more (Scala, Kotlin, Groovy, Eta etc.)

> inexpressive language like Python?

Could you elaborate on this please?

Crippled lambdas, no pattern matching, no static typing (also no HKT), no macros, no threads, no actors, no currying, no practical FP, the list goes on and on..

Moreover, the performance is awful, everything is boxed, and there’s no support for tail call eliminating.

Compared to Scala, OCaml, Common Lisp, or Racket, Python doesn’t even come close in terms of power and speed as the aforementioned languages.

I use python everyday and I find it very hard to work with. I do quant/ml/data science work with it and python has the best libraries of any language in those fields. But there’s no real reason why that should be the case besides maybe that python is so lacking in features that it takes 30 minutes to learn.

> It's really frustrating having to use a subpar and inexpressive language like Python because of the huge data science network effect.

I'm afraid that the "network effect" is gonna continue for the years to come. NDArray is not the only thing you need for data analysis. DataFrame, statistical functions, and data visualization packages like matplotlib and ggplot are all indispensable components in a data analysis workflow. It has taken thousands of contributors a decade to build a solid ecosystem for data science in Python or in R. It's just unlikely for Java to become a mainstream data science language overnight.

I'd say if you are looking for a statically typed language for deep learning, C++ is probably a better bet than Java for now.

PyTorch and Tensorflow both have C++ frontends now, which lowers the barrier of use for me.

Being able to test quickly in Python without compiling is really nice; while I prefer C++ for final products, prototyping PyTorch in Python feels like a better use of my time.

There’s a fuller ecosystem; I needed a PCA implementation. Instead of just using scipy, I had to choose an algorithm and use LAPACK myself.

I don’t really want Java as a data science language, but Scala would be a perfect fit.

There’s already a nice data frame library that I helped develop at my old company called Saddle. Now we just need some quant finance libraries (like the excellent zip line, pyfolio, and alpha lens), some industrial NNs, optimization libs (like cvxpy), and something like scipy.

Is this ever going to happen? Probably not, but a man can dream..

I had to work on some "data pipelines" built with Python & Spark. Terrible experience. All the different data structures for tabular data without a common interface. Oodles of ETL code for transmuting data structures to apply different algorithms. Foraging the docs to figure out even the inconsistent smallest tidbits.

Madness.

I am still aghast. How does anyone get any work done using those tools?

https://www.datacamp.com/community/tutorials/apache-spark-py...

Why didn't they implement RowSet interface?

Or create new tabular data holding interface?

Any way. Whenever someone enthuses about Python, Spark, big data, whatever, I now assume they're insane.

The whole notebook (Jupyter) idea is awesome. Are the fans conflating different tools, equating Spark with notebooks?

Spark is written in Scala, not Python. I've never used the Python interface, but I've extensively used Spark from Scala and while it's a little annoying to work with sometimes, for "real" Big Data (data set > 1TB maybe), its probably the best tool around.

Have you looked at Thrill? [0] It claims to basically be a better, C++ Spark.

[0] https://github.com/thrill

This moved into the deeplearning4j monorepo: https://github.com/deeplearning4j/deeplearning4j/blob/master...

I'm eagerly awaiting their sparse matrix support. It's unbelievable that the entire JVM doesn't have a single comprehensive, production quality sparse matrix library [0]. This is one of the big things keeping my machine learning in Python.

Can't you use JNI with the usual C/FORTRAN-libs? That's AFIAK what python does [1]. Or are you asking for a nicer wrapper?

[1] https://markus-beuckelmann.de/blog/boosting-numpy-blas.html

Sure, but there's not a single mature library that's done so.

Check out Neanderthal[0], it seems to have support for some subset of sparse matrices and it's faster than ND4J[1] to boot.

[0]: https://neanderthal.uncomplicate.org [1]: https://dragan.rocks/articles/18/Neanderthal-vs-ND4J-vol1

Yeah, Neanderthal is great (I'm a Clojure a user). It's got support for structured sparse matrices (like Toeplitz) the last I checked, but not general CSC/CSR matrices.

I'm by no means a expert in the field, but I've used https://github.com/scalanlp/breeze for this....

Breeze runs out of memory for even surprisingly small arrays.

William Kahan, the father of IEEE floattig point arithmetic and one of the main references on numerical errors, has some bad thing to say about doing floatting point computations on the JVM : http://people.eecs.berkeley.edu/~wkahan/JAVAhurt.pdf

An nd array library could be great for machine learning but it is probably not the place where you want to do numerical simulations.

Honest question, how much is this about Java and not the JVM and its from "July 30, 2004", is it even relevant now 15 years later ?

Everything except operator overloading also applies to the JVM... But equally well to python. Basically, the complaint is you don't get any of the hardware fp stuff like traps and signals and all of the math is slightly unportable because it gives different answers on different machines, but these are well-known issues.

There is also the fact that intermediate operation between floats are done in float precision and not double precision (as would be the case with C). It might be the behaviour one would expect but it can lead to serious degradations of a numerical computation.

This PDF is not easily readable and it is from 20 years ago. Did the situation evolve ? Is it still relevant ?

Here is a previous HN thread on the subject : https://news.ycombinator.com/item?id=6585828

I believe the main pain points still apply.

Reply on news.ycombinator.com