First, let's unpack your justification a little – you actually offer two claims: 1) You aren't Facebook's customer. You're just a user. and 2) You agreed to let them change their services without your consent when you signed up for your account.
To the first: most of us may not be their customers, but there's clearly an exchange of value going in both directions. I give them my attention, demographic information, etc. which is valuable to them as it then allows them to charge their customers for access to me and my info, in all the various ways. In return, I get access to FB and all that goes with it... but I'm not just a user I'm a supplier. As such, there's still an agreement, both formal and informal, going on here.
The formal agreement is the one you allude to in your second claim – the TOS that most of us don't remember even agreeing to. That matters a lot from a legal standpoint. They're clear to do whatever they want, legally, within reason. (I'm not trying to go deep on this sub-point).
But much more important is the informal agreement between FB and their suppliers (err "users"). That's the stuff that brand equity is made up of – and the brand promise, the value to users is the ability to keep up with their friends and know that their friends can keep up with them.
FB has begun to renege on that promise – and as such, they'll lose brand equity and the value in the eyes of their users.
Call it a bait and switch or don't, but the effect is still the same.