Tidyverse 2.0.0
tidyverse.org
tidyverse.org
This release feels like a nice incremental improvement rather than a revolution. Adding lubridate as a core package will save me a few keystrokes. I also did not know about the conflicts package, but will certainly be using it in the future.
I'm a bit surprised that other "better" programming languages haven't done more to replicate the Tidyverse. The Tidyverse is a masterclass in metaprogramming, and has a pretty big following. While R has some warts as a language, I'm not sure I've seen anything comparable to Tidyverse in terms of scope and influence. I see Julia trying with the Dataframes ecosystem, but it's got a ways to go before that feels cohesive like Tidyverse.
If I could wave a magic wand, I'd get Hadley and the RStudio crew really excited about Julia or Racket. R does have warts, and I dream of the day when something better is as good.
Look at the big picture, not simply the delta from the prior .x.y release.
The Tidyverse feels like a product that evolves over time, rather than a traditional approach that jumps versions. Major changes sometimes come in seemingly minor releases. I feel that 2.0 simply means the evolution has gone on enough to put a retrospective marker in the sand.
For something geared more to data science and exploration, rather than production use, I’m good with the evolutionary approach.
Then there are Grammar of Graphics (ggplot was Tidyverse's first star) style plotting libraries that Julia has been building. I'm probably most excited about Algebra of Graphics (https://github.com/MakieOrg/AlgebraOfGraphics.jl/) as part of the Makie Plots ecosystem. It does still feel a bit like Julia community can't decide between following Matplotlib or R's Grid/Ggplot approach.
The seeds of a Tidyverse for Julia are there, but it'll take some time to achieve the consistency and maturity of the original Tidyverse.
In particular, as you mention, plotting is one of the evolving parts of the ecosystem. Plots.jl is fine, Makie is powerful but very DIY, and AoG is slick but unwieldy. ggplot2 is far from perfect, but it works so well due to its maturity and integration with the rest of the Tidyverse.
In my ideal world, there would be a DataFrames.jl wrapper to provide nice (not just nicer like the two DFM.jl packages) syntax, and a powerful high-level plotting package (Makie is powerful but syntax is low level, Plots is mid on both) which is heavily integrated with the wrapper package.
Admittedly, I'm not a data scientist (anymore) so I don't follow the new developments in the dataviz scene much. If something like this exists then I would love to find it.
I wonder what my ideal syntax would look like anyway. Maybe something close to Tidyverse but with symbols as column names `:col_name` for ambiguity reasons.
If people are curious, DataFrames.jl has done a fair bit of work in consolidating a list of other packages that complement DataFrames.jl:
https://dataframes.juliadata.org/latest/#DataFrames.jl-and-t...
I do have to say that DataFrames.jl itself is a pretty impressive bit of work, and Bogumił Kamiński deserves no small amount of credit. If Julia ends up stealing significant mindshare from R / Python data science communities it'll be because of of this work.
Also for the Julia curious, I'll say that I find the ergonomics and sensibilities of the DataFrames.jl ecosystem much more in line with R and the Tidyverse than Python/Pandas.
From my perspective, however, DataFrames.jl's power is what makes it quite unergonomic for me. As an example, take the `args => transformations => result` syntax for doing pretty much anything in DataFrames. It versatile, but the lack of rank polymorphism in Julia i.e. broadcasting/mapping has to be explicit (which is usually a good thing given that type polymorphism is Julia's whole schtick) means that the transformation syntax feels cumbersome.
It's not that I want everything rowwise by default, an option provided by DataFramesMacros.jl, it's that I want things to be rank polymorphic when it makes sense. Base R got this right, hell S got this right, and so the Tidyverse inherited it and it makes the package so much more ergonomic than it would otherwise be.
I cannot overstate how impressive DataFrames.jl is, but I have to caveat this with "but I really try to avoid using it if possible". It's a shame, but I just think R's laissez-faire hackability, which in many cases results in spaghetti code, works really well in the tabular programming world where ergonomics are king and performance is easy.
Have you come across [Tidier.jl](https://github.com/TidierOrg/Tidier.jl) yet? It's a relatively recent package (to my understanding), but developing at a pretty rapid pace, and tries to be more ergonomic with its syntax.
Python has a very weak metaprogramming story so it is hard if not impossible to replicate something like Tidyverse in that.
If Ruby had succeeded in data science instead of Python it would almost certainly be with Tidyverse-like abstractions.
I have colleagues who write R code all day, and they don't know the difference between RStudio the IDE and R the language. They can't perform basic tasks such as defining a function, or adding a column without the tidyverse.
It is just frustrating to me when they RShiny all the things, and I have to pickup the pieces because the spaghetti has collapsed under its own weight.
RStudio has somewhat indefatigably looked for pain points that R users had back in the olden days. And yes, I remember R back when lattice plots were the hottest game in town. Not only has the Tidyverse made R infinitely more pleasant to work with, the community around Tidyverse has made the R community infinitely less insufferable for newcomers. The obtuse responses to beginners on the old R mailing lists were legendary.
My preferred R IDE is emacs, and I'm not upset with what RStudio has done. When I interact with devtools, I don't feel that I'm wanting for functionality. This is in spite of the fact that RStudio has heavily integrated devtools functionality into their IDE, and could arguably have made devtools less easy to work with using generic tooling.
I understand the mediocrity of many RShiny applications in terms of how they are written. But, another way of looking at all those dashboards and interactive applications, is that they are AMAZING prototypes written by scientists, grad students, journalists, etc., who would have never been able to build them some other way. If we forced everyone to write these apps in Rust, of course the average quality would be higher, but we'd also have way fewer of them. Democratization of technology means that its users will be more... medium.
It has definitely become the best available MVP generator. I just wish I liked R more. Python still have nothing even close to how easily a novice can spin up a Shiny application.
If I tell someone they should use Streamlit/Dash/Shiny for Python, how do they get started? The beginner onboarding experience is horrible for someone without an existing programming background. Download Python, maybe instruct them to setup a "virtual environment", use this "pip" thing to install these packages, point them at a way to edit code, and try to give them a way to launch it. God help them if want to share their work. Huge amount of existing resources assume Linux by default or that people have any idea how to use Docker.
While I hate...basically everything about the R ecosystem, the beginner experience is leagues better. Download RStudio. It will setup R, install packages, and has a big button that will launch the Shiny application. Straightforward guidance on how to get it hosted on an existing provider.
Your colleagues are probably a symptom of how low the barrier to entry has been made by the tidyverse moreso than indicative of a total capture of the language.
The difference between "collection of packages" and "framework" is that "collection of packages" implies interoperable via API conventions, but otherwise standalone packages, which could easily be used in isolation alongside base R.
Whereas "framework" generally means tightly coupled components, in a "do things my way or the highway" fashion.
Technically speaking, it's not "impossible" to mix tidyverse with base R, but in general if you wanted to mix and match tidy vs base R syntax, you'd find yourself working "against" the tidy api rather than alongside it most of the time.
R is not my main language, but every time I come across a task and I need to make a decision between R vs tidyR, the two feel like completely different languages.
I also have some beef against some of the decisions/conventions used (e.g. abusing metaprogramming to pass variables and use the variable's label as a value, completely ignoring the variable's actual value, instead of passing it as a string or as an actual variable ... seriously confusing when you first come across it).
Using that definition how would lubridate be part of the tidyverse "framework"?
> Technically speaking, it's not "impossible" to mix tidyverse with base R, but in general if you wanted to mix and match tidy vs base R syntax, you'd find yourself working "against" the tidy api rather than alongside it most of the time.
I don't know, wrapping readxl::read_excel with a as.data.frame or as.data.table call works beautifully for me.
I load magrittr and ggplot and toss the rest out. Right to left reading order is all I really want from that world. Data tables are such a superior convention to tibbles that it's not even funny (I need a tibbles like I need another hole in the head).
The other glaring rhino in the room is the relationship between RStudio and the Hadleyverse.
I find the relationship and degree of influence the tidy verse and Hadleyite way of doing things to be a dark cloud over the R community.
I believe that Hadley's work is the driving reason R remains a competitive alternative to Python for data analysis.
I agree that nonstandard evaluation can be tricky, but the nice notation you get is worth it IMHO, and at this point you can switch to standard evaluation in a number of convenient ways. My favorite way is to just use base R temporarily, since outside of modeling functions it mostly works by standard eval.
And I think that's good advice in general. The best way to use the Tidyverse is to use it for what it's good at. And for things it's not so good at, use something else for a bit before coming back to it.
Not a fan of the nature in which tidyverse promotes itself. They used to be more aggressive - i.e. advertising their books in package load messages [1]. But these new changes still feel like an ad.
To me, as an outsider, tidyverse seems to mainly be about "dplyr". The main thing holding all of it together otherwise is just the fact that majority of their functions return a "tibble" instead of a data.frame, forcing you to load all the jungle of the rest of their stuff whenever you use one of their packages.
[1]: https://win-vector.com/2019/08/30/it-is-time-for-cran-to-ban...
Overview
The tidyverse is a set of packages that work in harmony because they share common data representations and API design. The tidyverse package is designed to make it easy to install and load core packages from the tidyverse in a single command.There is "the tidyverse" which is a collection of packages like dplyr and tidyr for data manipulation and ggplot2 for visualization.
Then there's "the tidyverse package" which allows you to import them all at once with "library(tidyverse)". You don't have to use the tidyverse package, you can load individual libraries as you need them.
Also worth noting that one of the common organizing principles between all packages in the tidyverse is the concept of "tidy data" (see specifically H Wickham's paper), and the use of intercompatible combinators.
Individual packages can be used in isolation, if that’s what you need. However, they complement each other nicely, in that they share data representations and similar conventions while handling different facets of data analysis. Thus, people often import the whole thing in one shot as the tidyverse and go from there. For example, you might read the data in with readr, clean it up with tidyr or dplyr, analyze it with tidymodels, and plot the results with ggplot.
The python equivalent would be a hypothetical meta-package that tightly integrates pandas, seaborn, and sklearn.
I don't think that lubridate has any relationship with anything else, for example, apart from being created by the same people.
Still using R because I find python cannot do exploratory data analysis of visualisation as well as R.
However, there's a ggplot clone called plotnine in Python which seems to be reasonably comprehensive.
Python is a much saner language than R (I am not talking of packages here), but it's still super clunky by design (less so than R). It can most certainly do all the things that R does, although its plotting libraries are either lacking in their design (matplotlib) or maturity (the other GoG-inspired libraries).
When you continue on your journey to learn more languages, at some point you realize that when you have a well-designed, simple and consistent language, you really don't need very many packages to get the job done fast. I had this a-ha moment with Clojure (although I find working with the JVM annoying).
That's like saying that cat turds smell much better than dog turds.
Maybe it's technically true, but let's not split hairs here.
Regardless of how true your comment might be, this can seem a bit hostile or condescending to people who don’t have a “cleaner” reference point for language design.
Or, in the “fog of war” of competing alternatives, it may even appear unreasonable to experienced programmers: we’re so used to thinking that everything is a tradeoff that over time we can forget that some things really can be “just better” than other things. Most of the time when we hear such claims, it comes from a place of subjective preference or over-generalization, so our brains pattern match strongly against that.
And at the end of the day, as a practical matter, the best camera is the one you carry with you, and the best tool is the one you can use.
But the second I’m doing anything outside of those specific tasks above, Python outclasses R to an equal degree.
The worst thing I can say about tidyverse is that I find the stack traces increasingly gnarly. The information you need to debug is there, there's just a lot of filler. I have some faith that they know and they're working on it.
The pipe operator as well as treating csv tables as SQL was intuitive for me as a software engineer.
One issue I've noticed is the increasing frequency of warnings, such as when performing left or inner joins. These warnings can be a bit overwhelming at times.
However, it's worth noting that you can suppress these warnings by using the "multiple='all'" parameter, which is easy to fumble when you’re exploring data, why not simply "all=TRUE”? It seems like a lack of ergonomics and considering user experience.
"There are multiple OOP systems to choose from. In this book, I’ll focus on the three that I believe are most important: S3, R6, and S4. S3 and S4 are provided by base R. R6 is provided by the R6 package, and is similar to the Reference Classes, or RC for short, from base R.
"There is disagreement about the relative importance of the OOP systems. I think S3 is most important, followed by R6, then S4. Others believe that S4 is most important, followed by RC, and that S3 should be avoided. This means that different R communities use different systems."
https://rconsortium.github.io/OOP-WG/
"The S7 package is a new OOP system designed to be a successor to S3 and S4."
S7 does look nice though, from the README overview at least.
maybe this?