A comparison of programming languages in economics
marginalrevolution.com
marginalrevolution.com
The R code, for instance, is obviously unidiomatic. If you have a four levels deep nested loop in R, you're doing it wrong, and it's taking every ounce of willpower to keep from spending the rest of the afternoon trying to fix up their code and make it faster.
1. https://github.com/jesusfv/Comparison-Programming-Languages-...
It's worth knowing, for economists, how well economists can write code in different languages. But I'm sure I, too, would write better C++ than R, since I've not written any R at all. They would need to ask economists skilled in each language to write the code to get a better answer, and there would still be a lot of noise in the signal.
EDIT last sentence was gibbgersh
They should have crowd-sourced the code for this study with a Top Coder-like contest.
I was really impressed with the speed difference they noted between Python and R and the fact that Numba wasn't too much slower than C++ implementations. However, given the state of the R code I became wary of accepting the benchmarks at face value.
I haven't made the jump from R to Python for statistical applications yet. So, this article gave me something to think about. In terms of programmer happiness, I'm willing to live with a little slower performance in Python and not have to spend hours in C++.
The flipside is that it only really results in a speed up for pure numerical code - if you have to use any Python objects, then you lose everything.
Second, to make the comparison as unbiased as possible, we coded the same algorithm in each language without adapting it to the peculiarities of each language (which could reflect more about our knowledge of each language than of its objective virtues).
They have invited others to fork the code on github and write the fastest possible for each language.
I will be interested to see whether people can whittle down the respective benchmarks.
They wrote the code for each language based on prior knowledge about each language, and/or from likely googling how-to's... far from performant code with any language.
A mild understanding of each language could yield dramatic performance improvements over not understanding it at all.
To make the comparison "as unbiased as possible" would be to code the same algorithm but adapt it to the peculiarities of each language implementation.
Compiling dynamic languages is not easy: their semantics of having everything dynamic and changeable at runtime resists it. Python is internally compiled into a pretty compact bytecode, but something like a method call is much more involved business in Python than even in C++, and is many times slower.
WRT fundamentalism: most sane open source communities are pretty pragmatic; Python's in particular.
In fairness to the authors, though, it seems to me they were asking the equally interesting and rather different question 'which language gives the fastest programs in the hands of someone who is an expert in something other than programming?'
To my knowledge, value function iteration isn't even used much in this area of research, so the claim that the stochastic growth model is a "workhorse" is a bit misleading - the stochastic growth model gets used, but it is not solved very often using the methodology in this paper.
If this criticism seems harsh, compare the authors' R code to Hadley Wickham's advice at http://adv-r.had.co.nz/Profiling.html -- or just search Google for "speed-up loops in R".
--
[1] https://github.com/jesusfv/Comparison-Programming-Languages-...
One of they authors addresses this in the comments on the OP:
> As we explain in the paper, this was a “design” parameter. We did not want to “adapt” the algorithm to each language to improve speed, because it will open the door to too many “uncontrolled variables” (how good a programmer in that language you are and so on). The only partial exception was with Mathematica for reasons explained in the paper.
> Yes, we know how to code better and prettier R code with the plyr package and related tools. Again, that was not what we wanted to do in this exercise. But we will be thrilled to see alternative versions on github. That is why we posted all our code there.
> My personal guess is that, given how large our capital grid is, plyr will go less far than some people would have forecast, but I’d be happy to be proven wrong.
And if zissou is correct below in his assertion that the "vast majority of economists are horrible programmers", a comparison of what a naive user would write may be the most useful.
They also executed their test code on poorly performing operating systems not designed to execute high performance code, ie. they ran their tests on consumer desktop general purpose operating systems instead of a more specialized system designed for the task... or at the very least a more specialized OS such as a flavor of BSD or Linux without the desktop and other generalized packages.
I don't really consider this a well done test, and the results are only surface-level... certainly nothing you should base a serious decision on.
(Well, unless I am just wrong on the feats that JIT'ed code can perform nowdays.)
Compare to C++ where closures can be used with virtually no penalty and aggressively optimized. Even C with its trick of passing an extra *void for environment comes out shining performance-wise in comparison.
Edit: There's a good discussion of the problems that JVMs have in this area at http://www.azulsystems.com/blog/cliff/2011-04-04-fixing-the-....
"The Problem is a hard one to solve; (i.e. it’s not the case that we’ll wave our wands and do a quick fix & rebuild HotSpot and the problem will go away). John Rose, Doug Lea and I all agree that it’s a Major Problem facing JVMs with long and far reaching implications."
Maybe progress will be made soon though, with more pressure caused by increasing use of closures in the new streams library and with anonymous function syntax available now.
JVM inlining is unmatched by static compilers now and is going to only improve once techniques described in the Azul paper are used. Sure, JVM has a ton of other problems compared to C++, but inlining is not one of them.
BTW: I agree with the point that JVM optimizations are not enough and you often have to know what you're doing and optimize by hand appropriately.
Now, can this be improved? Of course. Scala, for example, does a good job of not requiring primitive arrays. This is done with compile time tricks, though, not runtime (JIT) ones.
But, generally, I'm not interested in whether "for" loops run faster in language X or language Y. I am interested in whether some languages encourage code styles that happen to be easily optimized (I'm also interested in whether some languages encourage code styles that are easily maintained, easily extensible, or easily run into a ditch).
This paper is meaningful to somebody, but it's clearly written by people who spend most of their thinking about things other than programming. That's not an insult; it's a simple fact that a lot of programs are written by people who are experts in other fields. I happen to work with a former physicist who has experience writing programs to run on one of the nation's largest supercomputers. The depressing thing (to me) is that while he knows how to write numerical programs that run close to a computer's theoretical maximum speed, that information is only well-known among a small group of people (namely people who have to show that if they're allowed to use a supercomputer, they won't make it run wasteful code). I wish it were shared more broadly.
No way. It may be native C, but it's not native C you've hand-coded yourself.
>> As we explain in the paper, this was a “design” parameter. We did not want to “adapt” the algorithm to each language to improve speed, because it will open the door to too many “uncontrolled variables” (how good a programmer in that language you are and so on). The only partial exception was with Mathematica for reasons explained in the paper.
Sure, but this makes the paper less useful for answering the question, "which programming language should I use?" Or, "which programming language should I learn better?"
But, yes, most economists are pretty bad programmers, so "which language should I use poorly?" is probably a useful question to answer.
This is rather half-baked reasoning. The only ways I can think of to control for programmer skill or at least account for it is to
1. Use "idiomatic" code for all languages.
2. Use "optimized" code for all languages.
3. Use "real" code for all languages with a large and detailed sample set.
1. when you improve the code, do not move any elements outside the benchmarked language, i.e., don't use mex files in Matlab, c++ in R etc.
2. maybe create two or three different versions of your code - single processor, multiple cpus, gpu and run it on a single machine with a predefined set up.
This will remove at least some of the criticism thrown at the economists and has an additional benefit: researchers trying to find the best-suited language for their problem can not only use speed but also expressiveness, aesthetics, code length, boilerplate and ease of use to guide their decision.
It seems that every time this gets brought up online a sea of infamous Fortran-programming-economists always tell you how fast Fortran is compared to everything else, signing it with "Fortran or GTFO". I've met hundreds/thousands of economists and econ grad students at schools ranked from 1 to 200, and I don't recall a single one that actually used Fortran (I'm sure there is a non-zero quantity -- I probably just haven't cared to talk with them b/c they are most likely macro theorists).
The fact of the matter is that in economics graduate school, your professors could care less what language you use. It isn't an algorithm competition, it's a story telling competition. But the type of story telling economists do is no less noble than writing elegant algos. Writing a story based on economic mechanisms and behaviors and supporting it with quality data, sound statistics, and logical/exact theories is no simple task.
You have any advice/opinions on economics PhDs?
I guess my biggest concern is the viability of pursuing a PhD in economics without having a previous degree in econ. I have a BS and MS in environmental science and an MPA in energy policy/economics. I think I have stronger scientific chops than many economists, but I don't know how valuable that actually is.
Also, it seems to me that the general market for energy economists is very strong, and will remain that way for quite some time. Do you think that's accurate?
I love plowing through data and developing methods and analysis to try and draw out relationships and show why things are what they are. I know that sounds incredibly vague, but as far as I can tell economics is the field that most closely aligns with this in the context of energy and the environment.
It's hard to say without knowing more about your background, but many have done it before. A degree in econ is useful but by no means necessary.
You should probably worry more about your math training (have you taken real analysis or another course with real proofs?) than your econ training. I had three calculus classes and linear algebra. The first week of classes the professors were talking about quasiconcavity and the Bellman equation. I was lost. I worked 80-90 hours a week the first year in order to catch up.
"Also, it seems to me that the general market for energy economists is very strong, and will remain that way for quite some time. Do you think that's accurate?"
I have no specific data, but yeah, the market for economists in most fields is strong. Our program is not highly ranked, yet almost all of our PhD's are able to get tenure track jobs with reasonable teaching loads. To my knowledge environmental/energy policy is a hot field right now, particularly if you can bring in grants.
"as far as I can tell economics is the field that most closely aligns with this in the context of energy and the environment"
You should also consider PhD programs in agricultural economics or environmental economics. Highly-ranked economics programs often have a theoretical/mathematical focus that doesn't fit with what you want to do. I know from experience that energy isn't fully respected in economics departments.
I actually have a friend in Cornell's PhD economics program right now, and he's warned me about math (and he majored in math) and that Real Analysis in particular is crucial. That's what most concerns me, as I've leaned on my physics-PhD significant other in the past for some math-related help.
That's a really good point about field-specific economics programs, I hadn't really considered that, but it makes a lot of sense given my interests and current abilities. It seemed to me that energy was weirdly neglected when looking through programs (even though it has so many neat peculiarities and inefficiencies), so it's good to know I'm not the only person who has noticed that.
Thankfully, I'm not looking at it for the immediate future as my current position will last a few years (we have boatloads of funding and bipartisan support), and I'm at possibly the easiest national lab to get a staff position and advance without a PhD (Oak Ridge). Down the line though, I know I love academia and teaching, so I'll probably go back, and your advice is super helpful to think about in the interim, thanks again!
As an economist whose hobby is the study of programming languages, you can imagine the frustration that I feel. What I have found works best is to make grad students use basic functional programming techniques. Everything they do is just a few lines long. As opposed to a 250-line heap of garbage with three nested for loops.
Get over it! It's a support tool for them and until they demand that you take their code as a perfect example, developed have no need to complain.
"They do the minimal they need and nothing more."
In other words, they work on their program until it runs to completion without throwing an error message.
I have to wonder how it would be received if two computer scientists wrote an article judging the relative merits of a bunch of economic models.
http://economics.sas.upenn.edu/~jesusfv/comparison_languages...
> although one needs to be careful with the choice of compiler [for C++ and Fortran]
"Our fourth result is that Java imposes a speed penalty of 110 to 169 percent. Given the similarity between Java and C++ syntax, there does not seem to be an obvious advantage for choosing Java unless portability across platforms or the wide availability of Java programmers is an important factor."
I would also recommend Python to new programmers, partly for the syntax.
The paper is probably misusing the term syntax though. C++ and java really are quite similar in many ways, far beyond just syntax, java just letting the programmer forget about some annoying things like memory management and portability. If most of what they're doing in the language is numeric analysis, then there probably isn't much of a difference at all, and they will probably get slightly better performance from C++.
As you noted, there are a plethora of reasons to choose either Java or C++...
I know, I'm also horrified of how economists use excel. But it does tend to be their go-to tool, and they are all comfortable in it. It'd be nice to show them just how poor the performance is in excel, to tempt them into learning scripting languages if they have an excel workbook that will barely refresh.
1. The algorithm is just a bunch of arrays and loops in one giant block. This is the easiest possible case for compilers that optimize through specialization. Even a bad specializing compiler will be good at this but will blow up when faced with a big piece of code with lots of functions and data structures.
2. The runtimes are only ~2s, which is not long enough for the JIT-compiled languages to "warm up". I suspect Java in particular would have had the exact same runtime after warming up, because in an easy example like this Hotspot will generate the exact same instructions as the C program.
EDIT: I stand corrected. I downloaded his examples off Github and ran them the same way as the paper. I get 1.87s for C, and for Java... also exactly 1.87s.
Your code probably got JIT'ed... which essentially makes your java code into native C... So... it's no surprise they perform around or exactly the same (given the JVM is allowed time to warm up, discover "hot spots", and then perform the JITing)
In any event... the "myth" that Java is slow is bogus :)
In terms of econometrics, Python is going to surpass closed source software packages like SPSS, SAS, eViews, etc soon simply because the rate at which econometric procedures are being implemented in Python is growing steadily (e.g. via a statsmodels/pandas/numpy/scipy based stack). I don't think Python will ever pass R in this respect though as R has a large share of the statisticians helping add the implementations relative to Python.
[1] https://github.com/jesusfv/Comparison-Programming-Languages-...