ParentFull threadrho138·If it respected the standard then a lack of a robits.txt implies do not crawl, which they openly state they ignoreView on HN