I have phone contacts from the past decade and haven't kicked anyone off my Facebook ever. This gives me a nice large pool of people to contact when I need them and the yearly Facebook birthday wish is enough to have them remember me.
I don't take notes at all, but if I'm about to meet someone after a while, a quick look at their Facebook gives me all their current information.