This policy was actually changed in 2017 to completely ignore robots.txt.

[0] https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...