I've done this all throughout my life. From exploring abandoned buildings & sites to going into "restricted against lowlifes" areas, it's amazing the cool stuff you can see!
I've camped in ghost towns in Oregon. I've visited uncleared temples in Cambodia, covered in underbrush deep in the jungle. I've been in abandoned factories & hospitals, crashed a few gala affairs, visited beautiful beaches... you name it!
And now, living in the former DDR (East Germany), there are so many abandoned places that you can visit a new one every weekend :)