Hacker News Reader

Comment by arp242 | Hacker News Reader

There's loads of stuff that Bing doesn't index very well.

If you turn of JS there's almost no actual text on the site; only the FAQ has any serious amount of text. This probably doesn't help. Having the page open also really slows down my laptop and constantly uses ~50%-75% CPU and ~130M memory. No idea if that's a factor, but it takes quite a lot of computing resources to actually make the homepage render any text and perhaps Bing doesn't deal well with that (and even then, it's not all that much text).

Admittedly I have a somewhat slow laptop, but I can't recall the last time merely opening a webpage slowed down my machine this much, and this includes webapps, so it's really an outlier.

Either way, Microsoft specifically blocking it is the least likely explanation by far.

View on HN

JS-dependent sites are crawlable by bingbot, and Googlebot, and have been for years now. Otherwise, much of the modern web would be absent from search. Things can still happen, but it doesn't look like OP is doing anything unusual with rendering. If Google can crawl, I'd be surprised if Bing was entirely stumped (plus: meta descriptions).

> Having the page open also really slows down my laptop and constantly uses ~50%-75% CPU

OK here, for contrast: https://i.imgur.com/Qb2ux3o.png

> it takes quite a lot of computing resources to actually make the homepage render any text

On my own laptop, I see text in 100-200ms. A 3G connection on a simulated cheap mobile in Sao Paolo gets 2.3s: https://www.webpagetest.org/result/230209_AiDc5N_FKY/

These are both fine times: even a cheap device on a bad connection can get text rendering quickly. It's not a very taxing JS payload.

> JS-dependent sites are crawlable

Yes, I know; but there are limits on this, otherwise people would be mining Bitcoin via search indexers and whatnot. Like I said: I don't know if this is a factor here, but it might be. It's something that should probably be addressed either way.

The initial render time is fine. It just constantly uses CPU, according to htop and the Firefox process viewer, and it noticeably slows down everything.

at a previous company (2019-20) we did some tests on this.

The main problem with relying on js rendering is that it happened much much later than the crawl. Days at least and sometimes weeks. For us, fresh content was key, so js rendering was not possible.

Oh interesting stats, we will look into that. We actually have lots of blog posts with text so not sure about that one though.

There's a lot of stuff that Google doesn't index too, btw, that Bing do. I think what we all need to agree is that... different search index will inevitable have different results anyway.

Reply on news.ycombinator.com