(Yes, I understand imperative languages are faster, but why not just keep the imperative parts for the "inner loops", nicely isolated, and a functional core for everything else?)
(Yes, I understand imperative languages are faster, but why not just keep the imperative parts for the "inner loops", nicely isolated, and a functional core for everything else?)
R isn't exactly what I'd call your typical imperative language. It's more like the love child of APL and Scheme, with random bits thrown in by a bunch of statisticians.
Loops are almost never used in R, since pretty much everything is a vector (or some sort of data structure made of vectors), you simply apply functions to vectors and it maps over the whole thing.
Of course, the C, C++ and Fortran bits are 'imperative', which is kind of the structure you describe anyway (functional, vectorized R language which calls down to a bunch of C/Fortran functions).
I don't think this all of the difficulty is intrinsic to FP, but FP language designers don't usually care about newbie-friendliness. The closest thing to a user-friendly FP language I know about is Scala, and it, too, has lots and lots of arbitrary symbols and hideously complicated function signatures.
Remember, the typical scientist has no patience for monads, currying, inscrutable value names or functions-as-arguments and compiler complaints about function signatures are more likely to instill deep hatred than anything else.
I know it's an extreme example, but a statistician will never try to understand what something like "def map[B, That](f: A => B)(implicit bf: CanBuildFrom[Repr, B, That]): That" means, he'll just move on to a language that's easier.
The drawcard of R is the breadth of specialised statistical packages. (Often with inner loops written in C for performance.)
BUT R is not slow when it comes to parallel processing or using Revolution Anayltics also has speed ups if you need it. R also has dplyr is speedy and the data.tables is even faster. I think the original Julia speed claims were a little biased to Julia and well there is plenty of awesome things about R, but "slow" isn't a far statement. There is a reason why R has grown so much.
Interesting argument for R usage: https://matloff.wordpress.com/2014/05/21/r-beats-python-r-be...
(This comes up often, and I'm not sure why I'm compelled to reply today, but there it is.)
As for why this can matter: a big part of my job is designing and prototyping new statistical estimators. "Prototyping" means running them through lots of simulations to explore their properties in small samples. My two options in R are:
1. Program up the estimator in pure R, in which case the simulations can take days to weeks to complete.
2. Program up the estimator in C and write an R interface, in which case the simulations will run 10 to 100 times faster. (Depending on the exact operations used.) The code will take much longer to adjust if I need to change the estimator, which happens frequently, and debugging will take longer.
3. Program up the estimator and simulations in pure C, making it still faster but much more brittle.
This is a fairly iterative process -- I'll typically discover errors in the math as I run the simulations or after I've run them, or the simulations will reveal unappealing properties that need to be fixed. But once I have "good" estimator and "good" simulations, I write it up in a paper and am done with it. (Gross simplification of my actual job, but accurate enough.)
So the distinction between "R's speed" and "C's speed" very much matters, and if R were actually as fast as C it would make my life much easier.
Of course, for people who are "coding in R" by running data analysis in a script the distinction doesn't matter. But I already said that in the comment you're replying to.
Julia as a language is faster, but I'm not sure it's actually faster than R to process data if you're using R as most people do...