It depends. The problem isn't so much what you might miss out on if all you are doing is writing a library, the problem is what limitations it places on consumers of your library.

The most obvious Babel plugins are JSX and Flow. If your library is using TypeScript then it cannot go through the same build process* as any downstream project that is using any Babel plugin. That means you need to publish your library as ES5 and the client loses most the advantages of your library being in TypeScript in the first place.

Also, Babel appears to be doing a slightly better job of keeping up with the draft ES standards, so if you want to use cutting-edge JS features (like async/await) that are on the standards track, you may need to wait a bit longer for TS adoption to get there.

I also think it's unlikely TypeScript is going to win out in the long run, so if you commit to TS now you're probably going to be rewriting in a few years. It's better to stick to standard JS where possible. The history of ES4 strongly suggests that TS is never going to become an ECMA standard.

* You can, at least in theory, do a two-stage transpile that goes through both Babel and TS, but IMO, nothing good is likely to come from such a setup.