I think it's just because many single Magit commands run multiple git commands behind the scenes in order to collect all the info. So any slowdown in how long it takes to run a single command is multiplied proportionally.
I'm suspicious of this explanation because the views that gather information from multiple commands can generally be configured down to a bare minimum that uses very few (sometimes just one) command. I assume they would have tested this before considering it inexplicably slower.
I experience the same slowness on Windows by using bare magit without any customisation. Can you point to some resources about customisations that could make it fast?
Couple of references that helped me (though for transparency this was a few years ago):