Hacker News Reader

I agree in principle, but this is exactly why we've moved our data pipeline from Python to Haskell.

The Python ecosystem has this concept of what is "pythonic", which, while it results in more code around the internet looking familiar, it largely resolves to "write it out yourself by hand". The result is that there are often no library functions for these things, even in libraries where you'd expect them to be. Need to coalesce an Optional, or need a switch statement? Write out the conditional. Need zipWith, flatMap, or mapMaybe? Write out the list comprehension. Need a combinator, or something curried? Write it out.

We spent an obnoxious amount of time fighting with bugs from this, or weird edge cases in pandas and numpy, that would show up halfway through training, or not at all until we measured results. Heisenbugs from generators were also a huge issue. We ultimately decided the situation wasn't tenable for the quick iteration cycles we needed, and moved all the data munging to Haskell. Almost everything we need for data munging is in base, usually in prelude. If it isn't, it's in lens. Changes and refactors are quick, and no more surprises halfway through training.

I like using zip/map/reduce in my data munge phases as much as anybody and I think python can become a ball of mud too fast sometimes.

But your statement is just so out there to me: > it largely resolves to "write it out yourself by hand"

me: Oh, I always felt the ecosystem was thorough between numpy, scipy, and bindings to most any database or C code like zookeeper, openCV, redis, etc. but I guess you needed something in a more specialized domain?

> Need to coalesce an Optional, or need a switch statement? Write out the conditional. Need zipWith, flatMap, or mapMaybe? Write out the list comprehension. Need a combinator, or something curried? Write it out.

...

I tried to be charitable, and maybe coming from a language with all of these things these are glaring omissions, but I had trouble no rolling my eyes.

Whatever itertools doesn't have can be done in 2-3 lines. A little repetitive? sure. But so onerous it factors into language decisions? bizarre.

It's not about it being hard to write them out; everything that needs to get written out is surface area for bugs. One missed edge case could just mean that a node of your input layer gets zeroed for some of your rows where it shouldn't, and all you see is no correlation where you expect some. The little bugs that creep in writing this simple boilerplate kill days of work.

One person writes unzip as zip(*arr), another writes it as two separate assignments with the list comprehension written out. It's a tiny bit of code; they both look like perfectly fine unzipping code, and they pass tests, but if you pass one of those code blocks a generator instead of a list - say, someone swaps a list comprehension for map - half your data disappears. No errors, just a node that shows no correlation.

Since the results are all getting serialized anyway, the amount of work to just have that part of the pipeline in a language with some guard rails against that kind of thing is pretty minimal.

I agree that this is very annoying (and time consuming!) time for debugging, as it has implicit assumption about data (e.g. that some column has more than one value).

I was tempted a few time to write pipelines in Python which make such sanity-checks.

In any case - thanks for sharing your example. By any chance, can you show an example for such Haskell pipeline? (Especially if there is some non-trivial statistics, or ML training.)

I am a data scientist (and use both R and Python regularly). I would take the plot in your link with a grain of salt - RapidMiner is not frequently used (at all) with any of my friends / coworkers, though maybe I'm just in a bubble.

I like R a lot the more I use it. Tidyverse libraries (especially purrr + dplyr + ggplot2) make R a joy to use. I would argue that R is ahead of Python in terms of libraries for everything but deep learning (which is, after all, only part of ML).

couldn't agree more; I think working in R is a vastly different experience now than it was 5+ years ago, and has changed (for the better) much more quickly than Python has. I'm primarily thinking of the tidyverse here; the three you mention are so much more intuitive than loops + pandas + matplotlib.

With Max Kuhn's new tidymodels stuff, I think R has a real shot at providing a nice alternative to sci-kit learn, though there's a lot of catching-up to do. For the deep learning stuff, Keras in R is about as nice as Keras in python, but I'm not holding my breath for pytorch-like workflows in R.

MLR and caret do a lot in the 'unified API for models' approach, though (IMO) they're not as streamlined / mature as sklearn, especially if trying to do modeling using sparse matrices (they mostly expect dense matrices as input).

I haven't played with MLR; how do you like it, esp. compared to caret?

I never really caught on to caret as it felt rigid and clunky and non-idiomatic, but I've been using Max's newer rsample (setup CV) + recipes (like sklearn pipeline) + yardstick (metrics) packages to good effect lately. parsnip, which will handle the core model-fitting, seems promising but is too early to use yet.

I don't expect sparse matrix support to get any better, as the core model functions would have to be rewritten entirely to avoid rehydrating them, which AFAIK nobody is seriously working on :(

I've only used it a little in side projects (xgboost + mlr); I think it works better than caret, though (not as brittle). Most of my day-to-day is text data (which mlr isn't really well suited for).

What originally drew me to it was a blog post [1] about using their model-based optimization framework for tuning hyperparameters. It's a lot more sophisticated than anything I've seen for Python, including hyperopt / hyperband.

[1]: http://mlr-org.github.io/How-to-win-a-drone-in-20-lines-of-R...

> Most of my day-to-day is text data

Ah the concern about sparse matrices makes even more sense now! I love that tidytext can produce sparse tf-idf matrices...but hardly any models can use them :(

I am a data scientist as well (ML/DL). This thing with RapidMiner surprised me a bit (I don't know what it is THB), but I guess it may be a totally different strand of analytics or stuff.

> would argue that R is ahead of Python in terms of libraries for everything but deep learning

Certainly there are more implementations of various statistics. Still I jump to R because there is something which isn't implemented in Python (some decision tree models, and things related to Item Response Theory - in my case). And just for ggplot2. :)

Yet, with "being ahead" it is tricky. For example, scikit-learn have a limited collection of algorithms, but once it covers something, it is with the same API and typically well-implemented.

I had not heard of RapidMiner either. I looked up its website on Alexa: https://www.alexa.com/siteinfo/rapidminer.com

It looks like most users are in China.

R, language-wise, is awful

Yes and no. Data frames are built-in to the standard library, which also happens to contain a lot of stats functions that are data-frame-aware. The formula API is really convenient. The language distinguishes between NA and NaN. Vectors and matrices are the core data structure. The language is almost fully homoiconic and can be hacked and introspected more or less arbitrarily. The standard plotting library gives direct, near-pixel-level control and a lot of high-level convenience tools (albeit with bad defaults).

There's a lot to be said for that. If you think of R as a statistics DSL, it's an excellent tool. If you think of it as a general-purpose programming language, a high-performance computing language, or just about anything else, you will come away disappointed. That's not what it's meant for.

> Julia may be more interesting (and it offers types!) plus much more focus om performance for iterative operations (which Haskell is lacking). Though, each time I tried it, I had to go back to Python - again, due to maturity of its data science ecosystem.

I definitely agree that the Julia ecosystem is still maturing and not everything is as developed, however I've often found that the lack of maturity is less of an issue in Julia because the flexability of the language through generics, parametric types and macros makes a lot of the infrastructure needed in Python unessessary in Julia. Obviously your milage may vary in this regard though.

Same here, Julia's lack of a very broad package ecosystem is less of an issue because i) it's very easy to interface with R, Python and C; ii) types and multiple dispatch plus basic libraries and efficient code generation make it quick to develop good number crunching code.

Furthermore, the ecosystem is growing quickly, and it already has some unique math & stats packages not found elsewhere. Real gems. See https://discourse.julialang.org/t/what-package-s-are-state-o...

The best language is the one that gets the job done (or one you can use to get the job done).

I've used R for a long time and agree it is terrible (as a language) but the community has learned from other languages and gotten much better in the past few years.

I'm quite optimistic about Julia but agree the ecosystem is still maturing.