Hacker News Reader

Comment by pilif | Hacker News Reader

So more senseless errors in my error log and more traffic caused by robots requesting a file that doesn't exist?

While I applaud the idea, can we please, please have a meta-tag or header that points to the location of this file if it's available?

We don't need another favicon.ico nor robots.txt

View on HN

You'd rather a robot hit your frontpage— and by association, your application- than hit your 404 static page?

Instead of generating errors from 404s (a lot of noise), try generating errors from the same repeated 404s.

The robot will very likely hit my frontpage anyways, so I'd rather have them look (and not find) for a meta tag there than to produce an additional 404 error which will, depending on the content of said page, waste a considerable amount of bandwidth.

Besides, my frontpage is either heavily fragment- or just page-cached anyways - especially for anonymous user - so it can be served directly from RAM for all intents and purposes, so a hit to the frontpage is really, really cheap.

> an additional 404 error which will, depending on the content of said page, waste a considerable amount of bandwidth.

Because your 404 should be one of the heaviest pages on your site, full of graphics and surprises. Why not make a full-featured game specifically for your 404s?

thebest404pageever.com

Crawlers don't (usually) download graphics, much less from 404 pages. They aren't regular browsers.

Cheaper than a hit to a static text file? It's cached in RAM too.

We reduce the noise by only looking at 404s with referrers.

That will cause you to miss some behavior if internal URLs ever change that users bookmarked or remember.

As an example, a TV station site I follow had a /fullepisodes/ basename page which wasn't updating in sync with their show pages, apparently pulling from a different data store. After about six months of declining updates to that page, they eventually re-vamped it with a more-correct data source. I like to think it happened because I and others kept browsing directly to the easily memorable URL

Second. A meta-tag would make it easier for services to add in mass as well.

With a meta tag the data could even be hosted by a third-party service if the site owner prefers that.

It's probably better to use a <link rel="business" href="/business.txt" /> than a meta tag.

what about meta entry(s) in the robots.txt, specifying if additional things like business.txt exist (and where)

But robots.txt is just for exclusion directives, right? AFAICT it's not really used for assistance with finding resources.

That's true, but you can also specify the Sitemap from the robots.txt, which is an inclusion standard.

Reply on news.ycombinator.com