Usually it's robots.txt that 'prohibits' such use, but you're right it's not enforceable.