No technical reason. Following robots.txt is purely self-enforced, crawlers don't even have to read it if they don't want to. It's just part of being a good netizen.