Comment by adeelk93 | Hacker News Reader

Comment by adeelk93 | Hacker News Reader

Parent Full thread

Sure, I thought this was a common experience, and I’d love to hear how you get around it.

The tidyverse is absolutely full of breaking changes. Two big ones off of the top of my head: dropping underscore functions like mutate_, and turning common column names into new function names. So in different organizations you might see two patterns:

* Monolith - everyone uses a set of R packages from 2017 or whatever (frozen in time either through MRAN or Nexus)

* Wild dockerization - one team’s internal tooling is incompatible with the other team’s, so working across teams you gotta rely on Docker a lot

Why? Cause there’s not really a good way to pin versions besides shrinkwrapping, at least in my experience. We tried Drake, but Drake itself had so many breaking changes that it gained a really bad rep (from what I hear from colleagues, haven’t used it myself). Tried renv but it was destroying some of our analyst’s setups (like, you just get a bomb icon in Rstudio). Didn’t troubleshoot it too much, kinda just gave up.

I don’t like forcing a specific vintage of packages, but I like everything else less.

For ad-hoc data science work R is always the first (and last) tool I reach for. It's just so fast and reliable to develop in, and with tidyverse, such a joy!! When working on python projects, I've seen very experienced python people (gold for the python tag on stack overflow) get stuck on some frustrating and mundane dependency installation issue and spend an entire day with attempt after attempt to install and get on with their analysis.

By contrast, R has the occasional but rare installation issue, and almost all the common ones are solvable in a few minutes with a re-install.

I think for large software projects where dependencies could be an issue, especially long term, R might not be the best choice.. for now anyway.

I feel some of your pain on the tidyverse front, but for that reason I stick mostly with fairly common functions and avoid exotic new ones. Even if I do venture into new ones, I comment them so I know precisely what's going on so in the event of a refactor a few years' down the track I'll be fairly safe. With R there's usually 10 ways to do the same thing, so you're never completely stuck, you'll just get an error which typically has some sensible way to handle.

To spin a positive light on the deprecations, most are replaced with something nicer, so although you have the pain of updating old code, it forces you to be aware of the new approach, which is a silver lining of sorts.

R is my go-to for ad-hoc data science as well. And you’re right, deprecations are generally replaced with something nicer. From the perspective of a language used for ad-hoc analysis, these are all great features. Hadn’t thought of it that way.

Which further emphasizes the idea of: R might not be the right tool for production processes. I used to fight that concept, but in the past year or so I’ve come to embrace it.

It's not that difficult to fix package versions for any prod deployment, as long as you deploy the projects as R packages. You can fix the versions in the ``DESCRIPTION`` files, which are easy to manage through, e.g., Docker and ``devtools::install_version``. Essentially, not a huge difference from ``requirements.txt`` that we're used to.

DESCRIPTION files don't actually solve this problem - if you're trying to peg a package version, and it inevitably ends up an outdated version in the future, you'll be greeted with that error "Package 'xxx' is not available". Why? That version has been archived. I'm sure you've run into this error at some point.

devtools I guess is a solution (write some code to iterate over package requirements), but let me ask you this - how do you protect against breaking changes in devtools?

I have yet to encounter something more reliable than MRAN snapshots https://mran.microsoft.com/documents/rro/reproducibility

I think what you mention about breaking packages is not exclusive to R, that's just the result of community driven, open software development. You have to take care of reproducibility in any programming environment, but this doesn't have to do with the entire package eco system. Especially the tidyverse is very exclusive in that, since it is very fast paced and constantly evolving with breaking a lot of old code.

Many popular packages like data.table rarely break after years of version upgrades. But relying on that in any kind of programming language is asking for trouble...

We're using renv since about a year at our workplace and haven't had any major issues occur. Maybe try packrat? Or try conda with R. Otherwise using docker should eliminate most reproducibility issues

Breaking changes in packages is not exclusive to R, you’re right. But dependency issues kinda is?

Let’s say two competent analysts wrote some code 5 years ago and you need to reproduce it today, one in R and the other in Python. The Python code probably includes a `pip freeze` or some equivalent, but the author of the R code had no equivalent. There’s no good agreed upon solution. And I don’t think I could reasonably fault that R analyst.

Packrat seems to be falling out of favor, I should read more about conda with R.

(The way I’d solve this problem in R is with an MRAN snapshot from 5 years back in a Docker image, but that’s basically accepting I won’t be able to integrate it with more recently written R)

I think renv (rather than packrat) is the current go-to for package management, although I haven't used it yet.

Ok, this is not what I thought you meant. I don't know how anyone can solve this problem in a constantly growing and evolving open-source ecosystem though.

Well the very first step would be: native R support for installing specific package versions, and sharing that with others. There’s too many third party solutions that come and go over the years. And there’s no guarantee that what you select today will be understood, or work, tomorrow.

There is `renv` for that

Which is a third party solution. A few years ago you may have said packrat. A few years from now you may say some other thing.

The idea behind a native R solution is that it is far less likely to be deprecated.

Reply on news.ycombinator.com