French Revolution versus Red October, Mao's Cultural Revolution, gulags and concentration camps for every communist country. In 90% of these "elite trimmings" society is pushed back decades at least. Unruly mobs do not move the world forward.
But I wouldn't argue with you that the majority of these 'elite trimmings' set the society back. The question is after the set back does it move forward faster than it was or in a better direction (Often the answer to this would be hugely reliant on your personal values).