Exactly so. It is quite difficult to come up with a wealthy non-state actor who's wreaked the same sort of havoc as a Hitler or Mao. Let alone even relatively minor players on the international stage who're responsible for billions to trillions in actual damages to people's living standards.

The typical complaint against billionaires is stuff like their yacht is too big or people don't like one policy they took advantage of. It isn't really to the same scale. At their worst, wealthy non-state actors lobby powerful figures in government to act.