There is such a proper mechanism: robots.txt. Whenever I put all my users' credit card numbers on a public web server I make sure the URL is properly protected by robots.txt. If I want to be absolutely sure everything is safe I use CSS to make the numbers invisible on top of that! ;-)