Hacker News Reader

Comment by tyingq | Hacker News Reader

Popfile is worth looking at if this was interesting to you.

It is a general purpose naïve bayesian email classifier that you can integrate with almost any email system.

They took some of the concepts in the article mentioned here and expanded on them a bit.

For example, they have the idea of "pseudowords"[1] so that you're working with more than just the words in the email. Like html:td for example...it expands to the number html table cells in an email, which might help with choosing a bucket.

[1]http://getpopfile.org/docs/faq:pseudowords

View on HN

Thanks for mentioning POPFile, good to hear my old code mentioned on Hacker News. I wrote the original code for POPFile in Visual Basic(!) in 2001 (it was called AutoFile back then) as I wrote it initially for my own use without Microsoft Outlook. I got the idea of using Naive Bayes for my email classification after seeing the ifile extension to exmh(1). ifile dates to 1996.

I rewrote POPFile in Perl and made it work with generic POP3 servers (hence the name) and later with IMAP and more. It is still actively maintained by a group of people I've never met(!) with the last release in December: http://getpopfile.org/

(1) http://qwone.com/~jason/papers/ifile00.pdf

Nice, thanks for the link. I did find a few research papers and the like on this topic, but none of those seemed to have very interesting approaches/results. But this looks quite interesting. It would fun to compare Popfile to my simple approach with a publicly available dataset (such as Enron), perhaps.

CRM114 in an interesting mail sorting tool, although I don't think it has been updated in years:

http://crm114.sourceforge.net/

Orthogonal sparse bigrams are another interesting tokenization method, the authors of this paper tested it out against CRM114 with good results:

http://www.siefkes.net/ie/winnow-spam.pdf

Orthogonal sparse bigrams (OSBs) have been implemented in CRM114 since the publication of that paper.

Reply on news.ycombinator.com