Why is that problematic? Doesn't the fact that is has very exceptions make it a rule? This basically means there is a wide set of confouding factors at play...
Vast amounts of money might be wasted by governments in Poland, Hungary & elsewhere on fertility boosting programs that likely won’t work.
And academically, our dataset of countries is relatively small to begin with, so it only takes a few exceptions to break any statistical confidence.