Is there a technical reason that other crawlers can't just follow the googlebot rules? It's great that bots seem to obey the wishes of site owners, but I'm genuinely surprised that the yandex bot doesn't just follow the Google bot rule and ignore the general disallow rule.
No technical reason. Following robots.txt is purely self-enforced, crawlers don't even have to read it if they don't want to. It's just part of being a good netizen.
Thanks. That's what I suspected - I'm just surprised that yandex is behaving like a good netizen :)