The article mentions that they tried that and the crawler ignored it.