The most obvious Babel plugins are JSX and Flow. If your library is using TypeScript then it cannot go through the same build process* as any downstream project that is using any Babel plugin. That means you need to publish your library as ES5 and the client loses most the advantages of your library being in TypeScript in the first place.
Also, Babel appears to be doing a slightly better job of keeping up with the draft ES standards, so if you want to use cutting-edge JS features (like async/await) that are on the standards track, you may need to wait a bit longer for TS adoption to get there.
I also think it's unlikely TypeScript is going to win out in the long run, so if you commit to TS now you're probably going to be rewriting in a few years. It's better to stick to standard JS where possible. The history of ES4 strongly suggests that TS is never going to become an ECMA standard.
* You can, at least in theory, do a two-stage transpile that goes through both Babel and TS, but IMO, nothing good is likely to come from such a setup.
[1] A hand-maintained definitions file: https://github.com/facebook/immutable-js/blob/master/type-de... . Sure, it's nice as documentation, but that is a lot of redundant effort if you already have separate documentation, and types and documentation in the original source code too. [2] https://github.com/AgentME/contain-by-screen is one simple example.
With Typescript, you can't do this, because the code has to pass through the Typescript compiler first, which doesn't support (most of) these features (e.g. it does support decorators IIRC) so will throw an error when it finds an unfamiliar language construct, and it doesn't have a plugin architecture like Babel, so you can't add them in that way.
One example is the spread/rest syntax which is sometimes used in React projects isn't fully supported in TS: https://github.com/Microsoft/TypeScript/issues/2103.
I should point out you can use Babel and Typescript, so the Babel processes the output from tsc (needed for using e.g. async/await at present, as tsc can't compile these down to ES5), but tsc has to run first to make the code into valid JS (on this note, I was surprised how easy it was to remove TS from a project - Babel will ignore most of the TS specific stuff such as interface and type declarations, as it already strips them out for Flow compatibility and the syntax is largely the same)