That's just something people repeat over and over again until it seems true. In reality it just provides cover for malicious actors/actions.
The CIA, NSA, DHS, etc are all much, much smarter than me, and I would use IoT to compromise targets if it were my job. So there's that data point.