One complication is the fact that Babel results are often cached, so with a reliable cache, Babel performance barely matters at all. However, every caching approach I've seen has reliability issues in practice. Certainly a tool is more pleasant to use without a cache, since "maybe I have a bad cache" is never a worry when running into strange behavior.
It also depends a lot on your scale, how you compile your code, and what operations you want to be fast. When switching to Sucrase (my own Babel alternative project, mentioned elsewhere) on a large codebase, webpack startup went from 40 seconds to 20 seconds, though incremental builds weren't any different. Running tests (in node with a require hook) became much faster, from minutes to seconds when uncached and about a 2x speedup when Babel pulled from cache. In both of these cases, the remaining time is now in other things: webpack processing the files, node running the imports, etc. So there's a lot more besides Babel, but Babel still is a non-trivial component in the cases I've run into.