Hacker News Reader

Comment by seanwilson | Hacker News Reader

Hi, I didn't get much feedback when I posted last time so I'm giving it another try. This is aimed at helping web developers follow SEO, performance and security best practices so I'd love to know what the community thinks. Can you think of any changes that would make Checkbot more helpful? Did you notice any bugs? Thanks!

View on HN

- A regex blacklist for URL structure would be very helpful.

- Acceptable title length seems a bit short. 70 characters seems closer to what we're allowed today than the old 60 (per http://www.bigleap.com/blog/5-tips-take-advantage-googles-ne...).

Might also be worth segmenting URLs with query parameters in them into a low priority batch to check later (or skip).

(E.g., when spidering a WordPress site, the crawler wastes time on .../article/?replytocom=* URLs. URL filtering solves this, somewhat, but it might require multiple passes to identify all of the problematic query strings.)

Cool, I'm checking it out right now. Can you share what your tool provides that others don't?

The major one is that instead of manually checking pages one at a time, Checkbot lets you easily test 1,000s of pages in a few minutes to root out issues you'd normally miss. As you're doing web crawls from your own machine, you can also crawl any site you want as often as you want including localhost/development, staging and production sites. This lets you identify issues early and confirm fixes during development before problems go live.

Who’s the customer you have in mind who will want to do this for thousands of pages?

People involved in web development, SEO and marketing. It doesn't need to be in the thousands but checking large number of pages for problems has obvious productivity benefits over checking one page at a time. It also lets you identify issues that impact multiple pages like duplicate titles, descriptions and content that you'd normally miss.

Interesting. So the intent is to tackle problems before deployment. Do you plan on any devtools integrations so it can be used as part of an automatic CI/CD process? Also, does Checkbot dig into all dependancies or skip them like some others?

> Interesting. So the intent is to tackle problems before deployment. Do you plan on any devtools integrations so it can be used as part of an automatic CI/CD process?

A lot of the time, websites owners don't know there's a problem until their search results or Google Search Console updates. So I'm seeing it being used by developers to check localhost/development sites, then on staging for other problems, then on production when changes are made there. A command-line version to support CI/CD is something I'm really interested in as well.

> Also, does Checkbot dig into all dependancies or skip them like some others?

Can you expand on what you mean here?

> The major one is that instead of manually checking pages one at a time, Checkbot lets you easily test 1,000s of pages in a few minutes to root out issues you'd normally miss.

ScreamingFrog does this too, but it's still nice that your work is out there to add to the suite.

Reply on news.ycombinator.com