"It's money that was paid into social security with the explicit promise that you'd get it all back upon retirement"
That's not even close to true; there is no legal connection that requires you "get it all back" or that the government can't arbitrarily adjust benefits up and down.
In practical terms the program is a transfer from present workers to past workers, it doesn't fundamentally operate as a savings account.