Take a simple 4-wide dot product for example, on x86_64 you'd have 3-4 different implementations (SSE2, SSE3 w/ hadd, SSE4.2 w/ dpps). But the function itself is just a few clock cycles, and calling it via function pointer will eliminate any gains and you might as well compute it with a scalar loop at that point.
This is further compounded by inhibiting compiler optimizations. You can't use the dot product function in higher level code expecting it to be inlined and further optimized (which is really the key to performance) if it's behind a dynamic dispatch.
A sufficiently smart compiler could maybe propagate the dynamic dispatch above, so that all the dot products get inlined but all code using dot product would get emitted multiple times with different dot product implementations, with the dynamic dispatch only at the top level. This has a slight risk of combinatorial explosion, but there really aren't that many combinations of supported ISAs in real hardware out there.
Another option you can use without any special compiler support is to take all your performance sensitive parts and pack them into a shared object/dll, compile multiple versions with different compiler options and choose the correct dll at runtime. Or even build the entire executable a few times and have some kind of launcher pick the correct one.