Hacker News Reader

Comment by qsort | Hacker News Reader

Numpy is certainly amazing, but there are tons of competitors in the data/scientific space, which pure "A-type" data scientists tend to prefer: R, SSPS, Matlab...

The difference is that Python doesn't entirely suck as a general-purpose language. Sure, you might have better options, but it's still reasonable to write almost anything in Python.

Other scripting languages like Ruby, JS and Lua are probably a little bit better when evaluated strictly on their merits as programming languages, but they lack the ecosystem.

In other words, it might be inelegant, slow, a mess, whatever, but the concept behind it is basically correct.

View on HN

>> R, SSPS, Matlab...

I'd immediately throw out "competitors" that cost hundreds of dollars to use. https://www.mathworks.com/pricing-licensing.html Academic pricing is $275/yr and normal is almost a thousand a year. Then you pay for modules on top.

Yes, and this is why R has become so dominant in the data and statistics spaces. R is free, and has gained so much momentum that it often has packages that no other language has. Yesterday I found out that there was just no way to easily reproduce mgcv::gam in python. There are loads of similar examples.

As someone who likes both R and Python, and had the misfortune to learn Stata; I am very glad to see R winning over these expensive dinosaurs. Maybe one day it will even displace matlab!

R even has packages to read in NanoSIMS data! [1]

There are perhaps a hundred NanoSIMS machines in the world as they're extremely expensive [2]. Yet we have an R package on CRAN to analyse their data!

[1] https://cran.r-project.org/web/packages/lans2r/vignettes/lan... [2] This 2017 reviews says there are 42 machines: https://www.osti.gov/servlets/purl/1398172

numpy exists (somewhat indirectly) because matlab cost $$$. and now numpy is a far larger use of matlab's concepts than matlab will ever be.

I haven't seen anything an "A-type" data scientist needs in R/SAS/SPSS that was intrinsic to the language and couldn't be ported to Python. I don't want a "data/scientific" language, or a "web" language, or a "UI" language- I want one language that explicitly supports all the usecases well-enough.

> I haven't seen anything an "A-type" data scientist needs in R/SAS/SPSS that was intrinsic to the language

Not intrinsic to the language but in terms of the ecosystem, for a purely A-type data science workload R is significantly better. dplyr + ggplot vs. pandas + matplotlib isn't even remotely close.

Now, obviously in the real world nothing is ever purely model work, which is why Python more than makes up for the difference.

> I don't want a "data/scientific" language, or a "web" language, or a "UI" language- I want one language that explicitly supports all the usecases well-enough.

Me neither. *taps sign*

>> The difference is that Python doesn't entirely suck as a general-purpose language.

R is great when your data is already together in one place and homogenous/processed. In that instance the workflow using dplyr/ggplot is better. R the language is hot garbage though, and the libraries available for doing random things that aren't working with certain data structures, doing math or plotting are generally much worse than their python equivalents. That is the reason python dominates data engineering, and why it's pushing R out everywhere but academia.

R will still have a solid place in industrial pharma for a long time (even as python growths, R growth will continue. R is pretty tightly intertwined with the process of processing and submitting data from clinical trials to the FDA, much to SAS's chagrin.

Personally I think we have to accept that R, Python, Java, C++, and Go are going to be with us for the rest of our lives. I would expect PHP, Ruby, and Perl to go away much faster. Rust is still in question.

Rails is probably going to keep Ruby relevant for a lot longer than Perl. I might go as far as to predict that Ruby will be the new COBOL or FORTRAN - widely used but neither cheap nor easy to find someone to modify it.

Are you from the future? R is making progress to replacing SAS in pharma, but has a long ways to go. Here is an article from last year[0] patting themselves on the back for making a R submission to the FDA. There are oodles of existing processes where data has to be formatted just so because that is how SAS does it. Nobody wants to rip up and re-validate that code until they must.

[0] https://www.r-consortium.org/blog/2022/03/16/update-successf...

The trope about SAS being required is pretty old... people trotted that out when I last worked in pharma ~15 years ago. The FDA specifically released an article saying that R is perfectly acceptable for submission https://blog.revolutionanalytics.com/2012/06/fda-r-ok.html

Please don't spread the SAS FUD. I work in Pharma and talk to the statisticians all the time.

I currently work in pharma, and I can point to teams of SAS programmers. There are definitely R efforts, and I have no doubt it is the eventual future, but it is not yet here.

If there is anything in R that is better, it could be ported. It's kind of different from language intrinsics that can't be easily ported. In fact I think there's a great argument for writing a standardized data processing and visual representation layer in C++, and then making it usable from all the languages. It would be nice if this was true for the notebook layer.

It's the domain knowledge that's difficult. Like the example up thread about the lack of generalised additive models. The version in R was written originally by the developers of the method. There's no Python version because no one with sufficient expertise has ported it.

Don't get me wrong, I know mostly write Python because it's a better general purpose language but there are big big tradeoffs.

Matlab has a more interesting dispatch model than python though. (Mathematica has even better dispatch but I haven't used it in almost 20 years at this point) Julia's model is even better, but I just never have time to dig into it deeply enough to get to the zen of Julia. One day when I have time to understand Julia I'd like to, but it's hard to compete with the reliability of python just always working.

What is dispatch in this context? Function call polymorphism?

Aside: I wasn't a big fan of the Mathematica language so I wrote "PyML" a looooong time ago- it turned unevaluated Python expressions into Mathematica, transmitted them via MathLink to be evaluated the Mathematica kernel, and then the results were send back and converted to a Python expression. This was long before sympy. It never went anywhere (I found ways to simply not depend on Mathematica).

Yes, function call polymorphism. Especially combined with Mathematica's piecewise syntax. I didn't really use it for numerics, it was more for Lisp/Prolog type things. I was always impressed at how short and readable the Mathematica code ended up being. I used to use Mathematica quite a bit to generate C++ functions.

But it's a very different use case than Matlab. One of the things I like about python is it isn't too bad for sliding between lispy/functional modes for some things and more lapacky things elsewhere. Matlab has a better syntax for the lapack parts but trying to do anything functional gets really annoying and very, very slow. But still I would say Mathematica has the best pure math syntax for those sorts of things. I don't have any experience with sympy, I should probably check it out.

Mathematica is nothing less than a work of art. I dedicated a fair amount of time to become proficient, but it never solved the use case I had (find derivatives for complicated functions- this was before autodiff was common).

Unfortunately, Wolfram himself has made a number of decisions that greatly limited its applicability and so it will always be niche.

Well, why didn't the free alternatives to Matlab like Octave or Scilab win out then ?

I think people were not ready to switch from some heavy established software. Also by that time, costless was seen as less qualitative (certainly because organisations are happy with licences and are always looking for support contract ?) Coming later, when the market(?) is ready, made R more successful.

I think people used to compare to established standard and detractors are always there to point out missing stuffs, giving impression that Octave or Scilab are behind. Coming with a radical different language, R was condemned to quickly succeed or perish, and anyway avoid direct and frontal comparisons.

But R is itself a reimplementation of S.

Same reason people use Excel and not LibreOffice. Until you can promise 100% compatibility, users are going to run into something that breaks vs the original platform. For real business processes, the licensing cost is not worth sweating about the problem for parity.

>I haven't seen anything an "A-type" data scientist needs in R/SAS/SPSS that was intrinsic to the language and couldn't be ported to Python.

I don't know what "A-Type" is so I might be misunderstanding you.

For SAS at least in my Org what keeps it entrenched is Enterprise Guide and the associated environment. Having a drag and drop GUI to do the data extraction and manipulation and the analytical tasks makes it very easy for non experts to be incredibly productive. The people that use it here are Engineers (Non-software variety) and Finance accountant types. These people would not be productive writing python code (or R) but still need to use something heavier then excel for data analysis.

Over the years we have chipped away at parts of SAS with things like Power BI and Azure ML Studio but I don't see python playing in the same space.

> and couldn't be ported to Python

“Couldn’t be” is not the same as “has been”. If R has a package that Python doesn’t, I’m not going to port it to Python, I’m just going to use R.

Anyone who used Matlab to process text before 2015 wouldn’t consider it anywhere closer as a competitor to Python

Same goes for R/SASS

Python has no competitor in terms of the whole package, it is a full featured language while being particularly good at data

This is a good point.

I've done a fair bit of R and there is no doubt that as a pure data science tool it's usually quicker to produce something than Python.

But I prefer Python because inevitably any data science project ends up having other "bits" that aren't pure data science, and Python's super string general purpose computing libraries are hard to beat.

Some industries are stuck on matlab (jet propulsion, for example) but in general people from sciences are either Python or R, and SAS/SPSS are only really used in places like health care or pharma where there's a lot of regulation in place, because the software is designed around that regulation, unlike Python and R.

It is also still quite dominant in neuroscientific fields, as there is a lot of software and code there from the past, but it is changing little by little.

>Python doesn't entirely suck as a general-purpose language.

I always get caught off guard by comments like these. In my mind Python doesn't at all suck as a general purpose language. Only real argument is execution speed, but most people aren't actually writing code where Python's "slowness" matters and if you actually need that speed you can write that part in C and still use it from your Python application.

Matlab costs $$. Matlab was ahead in perf for a long time, not sure now. E.g. they used to JIT code, etc.

Comparing python to R, python feels like a much more mature language. R always seemed only useful for small scale projects, e.g. writing some script to process data with < 20 lines of code. Python codebases just scale much better.

It also has the nice feature of being good at multiple things. If you get a model running in Python, and decide you want to be able to run the code via an API, put it behind Flask and ta-da!, now it's a network service.

It's not the best way to write analysis code, and it's not the best way to write a web service, but it's probably the best way to write analysis code that runs on a web service.

I picked web services as a random example, not one I'm particularly passionate about. But as a general trend, if you want to combine 2 unrelated domains, Python probably has you covered better or more easily than just about any other language.

> If you get a model running in Python, and decide you want to be able to run the code via an API, put it behind Flask and ta-da!, now it's a network service.

You could say the same about a model (or any function) running in R. Want to instantly convert it to a web service? Use plumbr or RestRserve packages for that. Want to make a quick web application around it? Use the shiny package for that.

Just because Python is good at X, doesn't mean other languages are necessarily worse at it.

I would just add that for scientific scripting Julia is usually nicer to program in than the combo of numpy and Python. Julia is fast enough on its own that you can do the matrix calculations directly and it supports real mathematic matrix operators. It’s also a great type based design that makes using packages magical.

I think Julia has a hard uphill battle against the "Python is the second best language for anything" effect. Julia looks pretty cool to me, but I already know Python. I'll probably learn more about it in my spare time, but I'm also never going to be able to convince my coworkers to try it out, because they also know Python and that's good enough for them. And thus Python's dominance continues.

I was able to convince my coworkers to stop working on a Python project (a simulator for a scientific instrument) once we all realised that we would have not been able to keep using plain Python because of performance. When the alternatives where (1) to force everybody to learn either C or Fortran, or (2) to rewrite everything in Julia, they asked me to provide examples of both solutions. After I showed them a few different approaches (pybind, cbind, f2py...), there was full consensus for Julia. We moved what we implemented so far (it was not to much, luckily), and so far nobody has regretted the choice.

The problem in using two languages is that you do not have just to learn a new language besides Python. You have also to figure out how to properly interface them. (f2py was considered the easiest solution, but we feared that the 1- vs 0-based array indexing would have likely caused many bugs.)

Did you evaluate Cython? I'm not anti-Julia, but I like that my Cython code is useable out of the box from Python, with no wrapping, and then users can continue to use their Jupiter + Python scripting workflows with performant bespoke modules complemented by the full Python ecosystem.

Someday I'll do a project in Julia. But for some such projects, Rust seems fully guaranteed to be performant while Julia might or might not be, so I might still lean towards Rust (unless one of the high quality packages of Julia removes a lot of development time, which is a decent possibility).

I used it for another project but was not impressed. When I tested it, the documentation was scarce, and the deployment is harder than Julia because it needs a working C compiler alongside the CPython interpreter.

Very true, it isn’t usually the best designed language that wins in these cases. The language just needs to be good enough like CPP and JS were. Python is nicer to write than either and with [mojo](https://www.modular.com/mojo) possibly rising in popularity, Julia will lose some of its benefits. The momentum is behind Python which is the ultimate factor.

> which pure "A-type" data scientists tend to prefer: R, SSPS, Matlab

I very much disagree and I would say it's the opposite. The only competitor for data-scientists is R, especially if you're doing stats-heavy analysis (opposed to an ML-heavy analysis). SSPS is in my experience used mainly by someone without a data-scientist background, e.g. psychology, similarly to matlab, where the only users I know are engineers.

I would bet that the overwhelming amount of data-scientists use python, even if it's just to set up the data-pipeline. It's just the better general programming language and data-scientist are not analysing all the time but have to get the data in shape and available first.

Reply on news.ycombinator.com