I don't understand this argument. You already have "random" people having access to your building. You don't get to vet a single one of your neighbors, and they change all the time (simply less frequently). It's currently entirely within the realm of possibility that a drug addict or noisy college kid could move into the room next to you.
The only argument I can sympathize with is that the people may not necessarily respect your space.