It does not permit taking over pointers and doing a plain old RCE, sure, but it still does permit a variety of optimizations that could break the logic of your program in surprising ways.
Here are two examples of where it's clear that the UB from the underlying C is still present and the optimization passes are still able to do things that wouldn't be possible without UB, like falling through functions and deleting safety checks inserted by the Fil-C runtime:
https://news.ycombinator.com/item?id=49677463 https://news.ycombinator.com/item?id=49681273
I'm not really convinced you can take LLVM and just flip a few poison bits off and remove a couple optimization passes and know that UB isn't going to explode in your face.