Parents are routinely investigated or get police contact because of this. Of course, being America, it varies a lot from place to place (you probably won't have a problem if you're in an extremely rural place and your kids are playing around in the 40 acres that is your backyard, but in a conservative suburban subdivision you certainly can have CPS investigate just for letting your kids play in your back yard).
As for "always", things were definitely nothing like this when I grew up in the 80s. This isn't the same country I grew up in.