Hacker News Reader

One thing which is missing from this implementation is the use of lemmas. Rather than treating words like "house", "houses", "housing" all as separate terms, they all get reduced to the stem "house". http://lemmatise.ijs.si/ is a good resource for this.

View on HN

He covers this in some detail, under the term stemming (which I've heard more often in this context). He even makes a library recommendation.

Lemmatizing and stemming are similar, but not the same.

Example for possible results:

Lemmatizing: indices -> index

Stemming: indices -> indi

http://en.wikipedia.org/wiki/Stemming#Lemmatisation_algorith...

Btw, in case you want a good lemmatisation implementation, there's one in NLTK that uses the WordNet database.

It's probably worth noting that in this context stemming is probably most appropriate. Lemmatizing is unlikely to give better results and in most implementations you suffer a significant performance hit.

Reply on news.ycombinator.com