Regardless, you don't have to hit the Like button, or share stuff on Facebook, or post on peoples walls. If you want your private information to be well, private then don't post it.
Yes, you can remove the item from your news feed, but it also appears in your Page | Info list as well. This can only be removed from the original site (AFAIK).
My concern is that it's getting harder to have meaningful discussions for fear of future ramifications. Abortion? Gay marriage? Stem cell research? Can I only talk about those topics offline now? Can I read online articles on those topics if I'm logged into Facebook without it being tracked?
This new feature provides "Like" buttons on third party web sites, like http://cnn.com. If you are 1) logged into FB and 2) click one of these "Like" (or "Recommend") buttons on a third-party site, that information shows up on your Facebook profile and is visible to everyone and all applications, regardless of your privacy settings. You also cannot remove such a "Like" from your profile without returning to the third-party site and un-liking the article.