Hacker News Reader

Thanks. How did you deal with, e.g. 'Steve Jobs'? I guess a generic NER tool would extract that as a single entity. Then you'd search Amazon and find an exact title match (the biography by Isaacson). But you probably don't want this in your list, as the references were to Steve Jobs the person, not Steve Jobs the book.

So did you use some manual review step to remove these?

View on HN

Well, the nature of NER is that it can extract 'Steve Jobs' as either a PERSON or a BOOK entity (in my case). You can in principal extract whatever kind of entity you want, provided you have appropriate training data. The idea (loosely) is that it learns the structure of text around, say, a book. You can then label a piece of text as a book, despite not previously knowing that title. It doesn't solve the problem of two books having identical titles, however :).

(Note: I'm not an expert in NER/NLP, so please correct me if I have got something wrong!)

Aha, I see. Thanks. I thought you were using some generic pre-trained NER, but I understand now that you used either one which was pre-trained to find books, or you trained your own.

Reply on news.ycombinator.com