Sure, but how do you honestly know that? Is it based on the profile of the person you see looking through them? Some people don't look like they should be readers? Or the fact that the "good ones" -- the ones that people presumably want to read -- get taken?
I guess I'm happier not getting angry over things that I don't know for sure, I'm happier generally assuming the best of my neighbors, and I accept that the books are out of my control once I drop them off at the library.