At least when I used Gentoo, the point of compiling from source was more about customization than security. I remember having to set so many different options. It was quite granular. Now I just compile certain things from scratch and modify them as needed rather than having an entire system like Gentoo do that, but I do see the appeal to some people.