I'd say the learning curve for making a sexy plot is a lot shorter with tidyverse, but overall, relying on it handicaps you versus spending the half hour longer to do the same thing with base graphics (or a base-like package).
Well, yes, but that is part of the reason why it is so popular. R may be written by world-leading language design experts - but it doesn't show!
> ...relying on it handicaps you versus spending...
There is almost no reason to drop out of tidyverse if the problem domain is visualising data. Most people could go an entire career as an analyst using just ggplot + tidyverse.
If something needs to be plotted and ggplot won't work it probably makes sense to drop out of R and go straight to TikZ or OpenGL.
or D3.js