Reasoning must be checked against observation and experiment. It must be checked against reality.
It’s not just that humans are fallible and full of biases and failure modes. It’s that reason itself is only capable of crystalline perfection in domains where all priors can be enumerated and the system is closed, like pure math. Even there we know of mathematical constructs that exhibit phenomena like emergence and computational irreducibility where the future state can’t be guessed without fully evaluating the function; where “leaps” are provably impossible.
The craziest stuff is some of the longtermism stuff. They are literally writing sci fi (and cliche sci fi at that) and then reasoning from it as if it were real and using those conclusions to guide present day moral and political thinking. Absolute lunacy. We can’t predict the next 50 years let alone the next 5000.
Sometimes I think one of the evolutionary counter pressures that has likely prevented human IQ from being driven entirely up and to the right is that high IQ increases propensity for delusional thinking. I’ve met quite a few incredibly brilliant idiots. The smarter you are the more elaborate a prison you can construct for your own brain. The major innovation of the discipline of science (and it is a discipline) was to put forward a method to avoid this by taking a step, confirming, and only then taking the next step. It’s pretty simple but it requires restraint even when there’s a shiny thing that looks so “truthy” and cool.