Actually, you'd be surprised! For something relatively modern, it's MSVC or GCC, you only need to pin the version. Which you can do by bruteforcing a few functions with different compiler versions / flags until they match.

For very modern stuff it might even be clang. Also easy to detect. For really old stuff (90s and older) there's a lot of diversity yes but the programs are so small it isn't a real challenge. By the time programs got so big as to make manual reversing infeasible, the compiler landscape has consolidated;)