Are you assuming "people in power" were tied to those situations, though? Specifically, did you check if they were following a robots.txt? I have some criticisms for how they handle robots.txt, but if that's the root cause then it paints a very different picture.