More evidence in favor of androgens being the cause is the ridiculous recovery that some transgender women experience in hair regrowth after starting HRT. Sometimes it's slick bald to nearly a full head of hair. There was also a guy who isn't trans on some hair loss forum who decided to take the strongest antiandrogens he could order plus some estrogen and had incredible regrowth.
Utimately it's probably multifactorial. Minoxidil helps, and that seems to be more about blood flow than androgens, though it's not totally clear why it works.