Yes, it may be just a coincidence that this is so, with the temples and the temporal lobes being physically close to one another. But intuition makes me wonder.
There is a pretty large concentration of veins/arteries - 1 that come up around the temple, so that could be related to the frequency of graying starting there.
1- https://www.gettyimages.com/detail/news-photo/superficial-ar...