Ideally you wouldn't directly expose the compiler to make optimizations. You would create an IR of which the compiler could use and mung around with the IR instead of the compiler directly.
One possible disadvantage, you can't gain information by automatically converting code to another representation, but you could potentially lose information that would be relevant to an optimization. I don't know of any practical examples of this happening with LLVM, but I believe Java's type erasure carries a very slight penalty.
Another possible disadvantage is if you have a language designed to be compiled extremely fast, then adding an intermediate step could slow it down unacceptably.