As far as concrete reasons go, I've had to deal with far too much fallout from being incorrectly flagged by braindead processes trawling private databases which I didn't even know I was in. Since lots of these databases share information, but not necessarily updated corrections, I still run into the same mis-flagging every few years, across utilities, courts, credit agencies, banks, letting agents, etc.
As far as Facebook goes, being a citizen of the CCTV-riddled UK makes me acutely aware of the power, and potential abuse, that facial recognition technology can bring; having images of my face tagged and fed into a database does not sit well with me.
Since I don't use Facebook, I don't even get the meagre upside of whatever services they build on top of this database (some kind of gallery, I presume).