You say "if you build the cross-compiler" as though it's as simple as "if you cross the street". Have you ever tried building GCC? I've wasted days on it and in the end still failed to make it work.
See also https://en.wikipedia.org/wiki/Cross_compilers#External_links
I've hand-built GCC for half a dozen of different target architectures over the years and it's perhaps one of the most stable pieces of software when it comes to building it. I've only had one build failure over the years and that was unstable from git (even that usually works no problem).
To build GCC, follow the simple steps on the GCC manual. With a modern fast computer, it's all done in an hour or two (of mostly waiting).
If you're dealing with bare metal projects (kernel programming or micro controllers), building binutils and gcc for cross compiling from source is pretty much required. There are scripts that help and some distros have cross compilers in their package managers.
But building GCC is not difficult and it pretty much works first time, every time.