So if taking more or less taxes from the rich results in no meaningful difference in ordinary American lives, shouldn't the government then default to taking less? The other way would be just wasteful. And if your argument is that no, in fact the additional tax revenue from the rich is helpful because it funds various redistributive social programs, then my counterargument is that what those programs seem to be most effective at is incentivizing creating an unemployable underclass at the taxpayer's expense[0] while enriching politically connected players.
[0]: See Thomas Sowell's extensive body of research on the general welfare of black Americans before and after the Great Society programs of the 60s.