It's more a question of who I block. Sometimes I have IRL friends who sign up for twitter and have no tweets yet. If I see that they also have 10000 people they are following, I don't waste time figuring out if I know them.