Huge thing is mindset- one thing the Total War series got extremely right was shock value and the ability to win battles by routing opponents. Getting charged with a horse would be terrifying (speaking from semi-experience, I've been charged by both bull moose and cattle). Nevermind the animal also having someone on top of it armed to the teeth, I can easily understand running away in terror.

From my amateur research, this is one of the reasons the Romans were so effective- they weren't always better armed than their opponents, but their culture and training promoted better cohesion. Whereas in the scenario of a medieval peasant facing a knight, even negating the shock value of being charged by mounted troops the knight likely just has far more experience in battle and is less likely to rout. Every peasant revolt I've read about seemed to end with the nobles riding the fleeing peasants down and stabbing them in the backs. Makes one reconsider the modern connotation of the word 'noble', although maybe it was a bit of a euphemism even back then. I know chivalry was much more of an ideal than a creed people actually followed.