As a balding man, I've certainly been tempted to try finasteride. But the idea of keeping my hair by messing with my hormones sounded insane, so I never did it. I agree, this should not be an approved drug.
So far I've had no issue with sex drive while on the medication. However I did notice an increase in my sex drive when I first went off of the pills while my wife was trying to conceive. (However that seems to be temporary for first few days while my hormones adjusted.)
[Throwaway account since I consider my medical data personal]
I can absolutely see that happening. I'm very interested in tracing this line of questioning back to a root cause because it likely has a systemic cause, and fixing it would have major implication in the way medicine is practiced and the treatments people receive.