Hacker News Reader

I set up a robots.txt with one entry: a disallow for a specific html file. The html file is not listed elsewhere and thus can only be discovered by the robots.txt. The file features random stuff and an <a href=""> element with the nofollow attribute and a link target set. That link has no text contents, no size, and thus doesn't occupy pixels that a user could click on. When clicked, that link would report the http request to me.

At 2025-02-22 02:14:52, less than a day after i set up this honeypot, 66.249.68.37 (AS15169 GOOGLE) made a request on that invisible link, using user agent string "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.6943.53 Mobile Safari/537.36 (compatible; GoogleOther)"

So now i have hard evidence that google runs bots that ignore the rules from the robots.txt, but still parses urls from there, queries these resources anyways and follows links marked as "nofollow".

View on HN

$db=... $sth=$db->prepare("INSERT INTO beacon (time, addr, query, agent, referer) VALUES (FROM_UNIXTIME(?), ?, ?, ?, ?)"); $sth->execute([ $_SERVER["REQUEST_TIME"], $_SERVER["REMOTE_ADDR"], $_SERVER["QUERY_STRING"], isset($_SERVER["HTTP_USER_AGENT"])?$_SERVER["HTTP_USER_AGENT"]:NULL, isset($_SERVER["HTTP_REFERER"])?$_SERVER["HTTP_REFERER"]:NULL ]);