Hacker News Reader

HNHacker News

Top New Best Ask Show Jobs

Comment by alpb | Hacker News Reader

Full thread

alpb·

Do you think blogs with most recent posts can be bumped up (instead of a static ordering)?

View on HN

I originally implemented it by roughly sorting via HN karma, which you can see in my comment here:

[0] https://github.com/surprisetalk/blogs.hn/blob/22a471d56504d5...

To maximize serendipity, I ended up going with random ordering.

I think there are other RSS-focus solutions (like https://ooh.directory) that are probably a better fit for sorting by recent.

Remember "folksonomy" ...? So I would like to see some kind of automated keyword extraction for each blog to allow selection of topical blogs.

OK, keywords would need to normalied (e.g.plural to singular, plus maybe a bit of abstraction) and this would be a first (tiny) step to make it a blog-centered search engine ... ;-0

I've been experimenting with techniques for figuring out the topic of a blog, and never quite gotten anything to work well that didn't involve extreme computational costs that just do not work on a search-engine scale without a serious hardware budget (zero shot classifiers did a decent job IIRC). An unga bunga technique like TF-IDF works well for a given document, but less well for a website.

Had some half-decent success finding similar blogs given a reference though.

Yes, I expect full text extraction and normalization to be computationally expensive. I did some simple minded experiments on Usenet feeds back in the 90ies and it was quite some work for the VAX it ran on.

You can get it to run fairly fast on modern hardware. Like run a text extraction, tokenization and POS-tagging workflow on a quarter billion documents on PC hardware, takes like 24-36 hours. That's doable and affordable. But ML-adjacent methods are not. Requires far too much GPU compute, have no A100s :-/

Like... an RSS feed keeping tabs on all of these blogs/their "last updated" field value?

Reply on news.ycombinator.com