Perhaps Google checks this... They are typically not cool with cloaking. If you want to hide your site map, just make it a non-guessable URL and submit it to Google the Webmaster CP.
It still 404s with a GoogleBot user agent.
You can do a PTR/A lookup to check if an IP is legit Googlebot or not though. It's possible they're doing that.