Hacker News Reader

Comment by mruts | Hacker News Reader

Hopefully this can pave the way for deep learning and data analysis for Scala/Java. It's really frustrating having to use a subpar and inexpressive language like Python because of the huge data science network effect.

Or if Julia could get more libraries (it's maybe 25% there, which isn't bad), that would be pretty great as well.

View on HN

> It's really frustrating having to use a subpar and inexpressive language like Python because of the huge data science network effect.

I'm afraid that the "network effect" is gonna continue for the years to come. NDArray is not the only thing you need for data analysis. DataFrame, statistical functions, and data visualization packages like matplotlib and ggplot are all indispensable components in a data analysis workflow. It has taken thousands of contributors a decade to build a solid ecosystem for data science in Python or in R. It's just unlikely for Java to become a mainstream data science language overnight.

I'd say if you are looking for a statically typed language for deep learning, C++ is probably a better bet than Java for now.

PyTorch and Tensorflow both have C++ frontends now, which lowers the barrier of use for me.

Being able to test quickly in Python without compiling is really nice; while I prefer C++ for final products, prototyping PyTorch in Python feels like a better use of my time.

There’s a fuller ecosystem; I needed a PCA implementation. Instead of just using scipy, I had to choose an algorithm and use LAPACK myself.

I don’t really want Java as a data science language, but Scala would be a perfect fit.

There’s already a nice data frame library that I helped develop at my old company called Saddle. Now we just need some quant finance libraries (like the excellent zip line, pyfolio, and alpha lens), some industrial NNs, optimization libs (like cvxpy), and something like scipy.

Is this ever going to happen? Probably not, but a man can dream..

Not to mention it's a nightmare to scale...

I had to work on some "data pipelines" built with Python & Spark. Terrible experience. All the different data structures for tabular data without a common interface. Oodles of ETL code for transmuting data structures to apply different algorithms. Foraging the docs to figure out even the inconsistent smallest tidbits.

Madness.

I am still aghast. How does anyone get any work done using those tools?

https://www.datacamp.com/community/tutorials/apache-spark-py...

Why didn't they implement RowSet interface?

Or create new tabular data holding interface?

Any way. Whenever someone enthuses about Python, Spark, big data, whatever, I now assume they're insane.

The whole notebook (Jupyter) idea is awesome. Are the fans conflating different tools, equating Spark with notebooks?

Spark is written in Scala, not Python. I've never used the Python interface, but I've extensively used Spark from Scala and while it's a little annoying to work with sometimes, for "real" Big Data (data set > 1TB maybe), its probably the best tool around.

Have you looked at Thrill? [0] It claims to basically be a better, C++ Spark.

[0] https://github.com/thrill

> inexpressive language like Python?

Could you elaborate on this please?

Crippled lambdas, no pattern matching, no static typing (also no HKT), no macros, no threads, no actors, no currying, no practical FP, the list goes on and on..

Moreover, the performance is awful, everything is boxed, and there’s no support for tail call eliminating.

Compared to Scala, OCaml, Common Lisp, or Racket, Python doesn’t even come close in terms of power and speed as the aforementioned languages.

I use python everyday and I find it very hard to work with. I do quant/ml/data science work with it and python has the best libraries of any language in those fields. But there’s no real reason why that should be the case besides maybe that python is so lacking in features that it takes 30 minutes to learn.

How is java expressive when you don't have operator overloads?

Java libraries can easily be used from any JVM language, including many that have operator overloads and much more (Scala, Kotlin, Groovy, Eta etc.)

Reply on news.ycombinator.com