The biggest problem, for me, is that who can see these things is not up to only those involved in the conversation. It hardly even matters what the exact effect is of the rules currently in place, since they could change at any time. The rules on who can see my email inbox don't change. (I think this was what really upset people about Google Buzz: suddenly the rules around who could see what changed, making everyone on gmail who cares about that sort of thing insecure about their own email.)
I know a lot of people. I don't have an account on FB. Absolutely zero % of the people I know have stopped talking to me even though 95% of them all have an account.
I trust FB far less than any site online.
Fun Fact: pretty much every other social network, HI5, Tagged, MySpace etc. were all started by spammers. Friendster and FB were just started by douchebags.