My current understanding is: just add -flto to CFLAGS and CXXFLAGS, it does wonders and there is no downsides(other than longer build time). correct me if I am wrong please
My understanding is that LTO itself is not breaking anything. In these cases the software that it was running on was broken in some subtle way that didn't show up before.
Linux distros like Gentoo that can try to compile lots of stuff with LTO find a lot of bugs related to this.
There are cases where it causes runtime errors as well.
Sometimes software is written incorrectly and has "undefined behavior" (you probably know this already). The thing with UB is that it may or may not cause a visible bug at runtime. Sometimes these bugs go unnoticed for a long time.
It's really common for code with UB to run without visible errors when compiled without optimizations, but when you compile it with optimizations it segfaults at runtime. LTO is adding another form of optimization and sometimes the transformations the compiler made just happen to cause already existing UB to "surface".
This is what I meant by LTO "causing" runtime issues to appear!
There are cases of this happening in the Gentoo bug tracker, and in the Linux kernel!
If anything you might want to explicitly do -fno-lto to not use LTO when you have fat objects.
It does more than optimize.