ParentFull threadrsbadger·Exactly - seems the safest way is to explicitly block known bots by user agent from even reaching pages you don't want indexed.View on HN