Hacker News Reader

Comment by rahimnathwani | Hacker News Reader

This is really cool. I started reading HPMoR after finding it.

I'm curious: after performing NET on the corpus, how did you filter to find books only? Did you just search on Amazon's catalogue for exact matches, or was more tweaking required?

View on HN

I built a little tool to help with that, but pretty much the first result on amazon was always right - for books with more than one match, at least.

Thanks. How did you deal with, e.g. 'Steve Jobs'? I guess a generic NER tool would extract that as a single entity. Then you'd search Amazon and find an exact title match (the biography by Isaacson). But you probably don't want this in your list, as the references were to Steve Jobs the person, not Steve Jobs the book.

So did you use some manual review step to remove these?

Well, the nature of NER is that it can extract 'Steve Jobs' as either a PERSON or a BOOK entity (in my case). You can in principal extract whatever kind of entity you want, provided you have appropriate training data. The idea (loosely) is that it learns the structure of text around, say, a book. You can then label a piece of text as a book, despite not previously knowing that title. It doesn't solve the problem of two books having identical titles, however :).

(Note: I'm not an expert in NER/NLP, so please correct me if I have got something wrong!)

Aha, I see. Thanks. I thought you were using some generic pre-trained NER, but I understand now that you used either one which was pre-trained to find books, or you trained your own.

Reply on news.ycombinator.com