Hacker News Reader

Comment by lwhsiao | Hacker News Reader

I gave this a shot and was shown

> We have analysed 68 links on your website and haven't found a single issue.

I'd love to see what this actual list of links was, even if they weren't broken.

I'm partially curious because I personally use lychee [1] for this purpose, but when I run lychee (even while ignoring several classes of links), it is checking _significantly_ more links. E.g., the output looks like:

> 1401 Total (in 7s) 1361 OK 4 Errors 35 Excluded

Curious to understand what it is checking that Bernard is not.

[1]: https://github.com/lycheeverse/lychee

View on HN

What is the URL are you testing with? If you want also share the /demo URL and I'll check what is going on.

Seems like lychee also extracts links from text, Bernard does not. Only links inside HTML tags are considered, text nodes are ignored.

https://bernard.app/demo/33b26fe3-6c67-4338-a6cd-dd4e9204fbf...

I also suspect a different is when I invoke lychee, I'm running it on ALL of the *.html files output by my static site generator. Perhaps Bernard is only testing a single page?

If I only run lychee on the same single page, the numbers are closer:

> 76 Total (in 1s) 73 OK 0 Errors 2 Excluded

No, it follows every link recursively, as long as they are part of the same domain.

So, if you add example.com, the crawl starts at / and all links found therein are added to the queue, and so on. For convenience www.example.com is also considered within crawl scope (and viceversa). Everything else is treated as an external link, so tested but not parsed.

Is it possible that your links are hosted on another subdomain than the one you have entered? I'll do a deeper dive into this issue tomorrow.

Maybe not all pages are reachable from the root.

Cool!

> I'll do a deeper dive into this issue tomorrow.

FWIW, I don't think I'd go as far as calling this an issue, I suspect both Bernard and Lychee are doing different, but reasonable things.

No need to spend much time on my account, and cheers on launching a useful project!

There's why Bernard isn't finding as many links: a sample one from /blog

    <a href="https:&#x2F;&#x2F;luke.hsiao.dev&#x2F;blog&#x2F;2023-workstation-build&#x2F;">My 2023 Workstation Build</a>

I have never seen forward slashes encoded that way. These were simply not recognised. Added to the todo list.

Huh, me neither. Not an intentional encoding on my part. Looks like I have a bug to fix as well!

Reply on news.ycombinator.com