I have also lived in very rural areas and some very shady parts of town. I'm in Canada though, so handguns just aren't as prevalent up here (although I do have friends with restricted licenses and handguns they can shoot at the range).
I had an "interesting" youth. Growing up in rural area means I'm no stranger to seeing people at parties getting beer bottles smashed over their heads during fights or hit with ash trays at bars. Looking back at that, there was not a single situation where a person having a handgun would have made things better. It almost definitely would have made things much worse. People would have died as opposed to just need stitches at the hospital. Even in this rural area, it wasn't that difficult to talk my way out of bad situations. I was blasted out of the blue once, but it was a case of mistaken identity.
I think that if home invasions were a real threat to my family then I would really start to think about moving. There are just so many nicer places to live.