Piketty's argument, as well as the findings of the IMF report, is that what you've called relative wealth and absolute wealth are linked: greater inequality leads to lower growth rates. The entire economy suffers for the benefit of a few. Eventually, inequality might choke growth out entirely, leading to crisis.