Naïve Bayes for Machine Learning
blog.floydhub.com
blog.floydhub.com
float
bayes(float prior, BAYES_TEST * test, int result) {
if (result) {
return prior* test->tp / (prior * test->tp + (1-prior) * test->fp);
}
return (prior * (1-test->tp)) / ( (prior * (1-test->tp)) + (1-prior) * (1-test->fp));
}
Still, it is super easy to get up and running, will work with remarkably little data to train with (determine those false positive and true positive ratios) and runs very fast during classification, it also requires very little in terms of hardware (no GPU or anything like that).
Yes, you can improve on this, but it isn't always worth the effort or the resources, I've built more than one revenue generating tool with this.
Edit: Ah I'm guessing false positive and and true positive rate?
This makes adding tests very easy.
How can someone write that much text, and then all they end up doing is importing scikit-learn?
I can figure out how to do that myself in 2 minutes.
The real understanding of machine learning comes when you start implementing the models yourself.
Let's build a bayesian spam engine:
Start with a known-good email (not-spam) and a solicitation (spam). For each word in the not-spam document, count the number of times each word appears. That's your not-spam "corpus". Repeat for the known-spam document. Now you have a list of words+frequency (dictionary of counts). There will be many words which appear in both lists. This is normal.
Now, when you have an unclassified document, for each word in the document check both lists (spam & not-spam). In which list does this word appear more frequently? if the count for that word is higher in the spam list, then this is a "spam" word, therefore this document is +1 spam. If a not-spam word, then this document is +1 not-spam. Repeat for all words. At the end, which count (spam or not-spam) has the higher total? There's your answer. Spam or not-spam. Done.
Simple counting and comparison.
Seriously, that's the algorithm. It's that simple.
If the system miscategorizes a document, move it to the appropriate group yourself and count all the words again. (retrain). The system just "learned" from the mistake. It got "smarter".
You can easily add more categories, other than "spam" and "not spam" -- "questions from customers", "family & friends", etc. It works the same. The group with the highest total "wins" the classification.
extra credit: Now here's the interesting part. The corpus doesn't have to be words. It could be a switch set to "on" or "off." Joint angles on a robot. Ping time of a packet. Temperature from a thermistor.
And the classification doesn't need to be spam/not-spam, it could be "friend vs foe", upright/inverted, safe/not-safe, or anything else. The algorithm doesn't care, it's looking for probability of being in a classification group.
There you go. Have fun!
It's funny because when I saw that line I thought I would be pedantic and double check to see if you got the laplacian smoothing correct (because in practice it's never "that simple" for any numeric algorithm) and then realized that you don't understand how to implement Naive Bayes' at all.
For starters you aren't using probability. If you want to put all of the words in each document together into a two dictionaries of counts, then for each word in the unclassified document you want to look at the product of the probability of those words appearing in the spam corpus vs the non-spam corpus. That probability is n_word/total_words the corpus.
This is where you need to do some smoothing because if a word does not appear in the one of the corpora then you will get a probability of 0 for that class. Smoothing just adds 1 to the numerator and N_classes to the denominator. It is the equivalent of assuming a weakly informative uniform prior.
Thanks. It's been 15 years since I implemented it. Guess I failed the interview.
I never met a measure space I didn't like!
In this case, wouldn't it actually be better to just drop the denominator, because it will be the same for both (spam & not spam).
- try different models for the likelihood (the author mentioned Gaussian, Bernoulli, and Multinomial which are part of scikit but didn't bother to implement them)
- understand how the "naive" assumption makes implementation easier but less expressive (author mentions it in words but pictures are critical here, e.g. the Gaussian case)
- using log probabilities for numerical stability
- reframing the model as a linear model
It’s good to know how to create these algorithms from scratch (especially to demystify them) but at the same time you’re probably better off using sklearn and using your time instead to manage your data (unless there are significant customizations needed to fit the problem that sklearn can’t provide).
Yours was a decent example. I've seen one for weather conditions on a golf course and people's inclination to play used to predict how busy you'd be based on the weather that was really good.
Recommend downloading the PDF for better rendering of math symbols.
Edit: Now it's like.. 50/50?