Hacker News Reader

I use Stanford NER to extract book titles and authors from HN comments. Previous thread: https://news.ycombinator.com/item?id=14202557

View on HN

How do you disambiguate similar/identical titles?

Did you evaluate other NER against CoreNLP? e.g. how does spacy compare?

EDIT: answering my own question -- research here [1] ranks CoreNLP as the winner among CoreNLP, NLTK, spaCy, Lingpipe but AFAICT spaCy is competitive.

[1] https://aclweb.org/anthology/W/W16/W16-2703.pdf

I found spacy's documentation lacking in details on how to train NER from scratch (at least I couldn't make sense of it). That's why I decided to stick with Stanford's NER.

Regarding disambiguation: People usually mention books with their respective authors, so it takes one query against the local database to check whether entity is indeed a particular book by a particular author. When no author is mentioned, I check parent comment (if any) to find if the given book is mentioned there. Well, and if comment turns out to be standalone then I query Goodreads / Google Books API. Given books with identical titles, these APIs return the most popular option.

Reply on news.ycombinator.com