I'd argue that you generally wouldn't expect that to include IPs you manually enter into the URL bar. In every other case what that does is prevent applications from going out and doing stuff in the background, so that's going to be the assumption for browsers as well. Now that's not how the safety feature is designed, but that's how I as a user experience it working.

Also it doesn't really work does it, because Firefox can still do a DNS lookup, from a DNS server, which is just an IP.