The gains in GDP have more or less gone entirely to the top 10%, and in particular, the top 1% of the population. This makes everyone else feel like the pie isn't growing - because it isn't for them. Now that their stocks and homes are worth substantially less, and their credit is drying up, people have to spend less. And this fall in aggregate demand has disastrous consequences - not just here, but abroad as well, for all the countries that depended on exporting things to US consumers (Germany, Japan, and China).
Income inequality is of course not in itself a bad thing; it is the consequence of an economy where risk and work are rewarded. The question is extreme income inequality. Obviously this is a subjective matter. You may not think we have extreme inequality now, but when it's at the highest level it's been in 80 years, I'd venture that it is extreme. And extreme inequality often leads to falling aggregate demand and debt deflation. In short, a depression. If you think income inequality means nothing, you know nothing.