For those kinds of cases, I love having as-simple-as-possible reference code checked in alongside it, even if it's just #ifdef'd out for common builds. Having a baseline makes it sooo much easier to understand and debug the optimized code.
Ideally, the slow code is available alongside the fast code and the unit tests confirm they are sufficiently equivalent. If that's not possible, the go-fast bits need to be refactored until it is possible. Don't forget that error handling is part of the equivalency testing.
Beyond unit tests, this seems like a great opportunity to leverage property-based testing, at least for pure functions; checking that both versions return the same results is an easy property to think of.