It is still causal. You can't add sum of components that "eating x is overall worse than y because it is better in one thing but worse in another". It's hard to compare two axis and specially if one axis is unscientific land of hormone disturbance.
Also hormone imbalance isn't unscientific. It has been widely been studied that women will have their periods change timing when malnourished.
Anectdotally: I had my testosterone levels wiped out when switching from a diet with excess calories to a one with not enough calories and nutrients. The condition fixed itself when I started practicing a balanced diet (with exercises).