There are other reasons for not using -O3 which people already mentioned.
As a matter of fact, Gentoo specifically says in their docs that -O3 breaks some packages[2].
[1] https://stackoverflow.com/questions/57889116/different-evalu...
O3 does more aggressive inlining and generally expensive optimizations and optimizations that are not necessarily profitable.
Edit: found example where code works on -O2 but breaks on -O3, related to using abnormal amounts of stack space https://stackoverflow.com/questions/47058978/g-optimization-...
Likely because your codebase had UB in it that didn't show itself until a certain optimization level. The solution is to fix all instances of UB. See my comment above.
Forces me to use a better algorithm when it matters. Which more often than not, does not.
;-)