Humans can explain their reasoning when you ask them. The best we have with NNs is a fancy version of first-order sensitivity analysis.
If you make them believe that they choose another decision, they will create another explanation on the fly.
Slow thinking - such as reasoning - is more expensive so we only use it when we are faced with a new situation for which we have not 'cached' the answer. It entails the creation of a mental model and imagining outcomes. The exact mix of the two modes is tuned by evolution.
If all we had was the slow system we'd be much less efficient at survival.