And if I was naked at the time, and if they then sold those pictures of me, I'm fairly certain it would be a crime.
If you setup a directory /admin on your webserver, and forget to password protect it, and someone accesses it, you can't really call it hacking.
My point which you gloss over is in the definition of "attack".
Whether browsing to a publicly accessible url with no security is an attack or not, is a very grey area.
Whether or not accessing anything on a webserver is illegal depends partly on your intentions while doing so. That's a basic function of most court trials--to discern your intentions. For example, see the differences between the various degrees of murder, and the way such trials often hinge on "preconceived intent."
If my intention in accessing "/admin" was to access your server without your authorization to access some protected piece of information, damage computers, or defraud you, that may indeed be a crime. See http://www.law.cornell.edu/uscode/18/1030.html . It doesn't matter whether it was password protected or not. If I went there by accident, however, it probably was not a crime, although depending on what I accessed I may wind up having to go to court to prove that my intentions were benign.