However, unless you are a kernel developer, if you are recompiling the kernel or sound drivers, you are working against the system and making life harder on yourself.
There's a small glitch in newer versions of X.org that (until recently) didn't allow proper dual-screens on Radeon cards. I didn't remove the debian package, download the newest X.org CVS, and compile my own version. I just reverted the package and pinned it until I felt like upgrading it. Hell, there wasn't any reason to upgrade in the first place, so that was the first mistake.
If you ever have to compile something you are either not using a modern distro (Debian, Ubuntu, and, I presume, Fedora) or are using the same techniques that were required in the 1990s but are just bad habits today (I learned this the hard way).