Hacker News Reader

My use followed a similar arc.

One tip: look at your mail server logs for the top 100 bad addresses. For me they were a combination of incorrect first names (doug@, joe@) and incorrect roles (sales@, support@). Deliver all of that mail directly to a spam trainer. That gives me good odds the Bayesian filter will have repeatedly seen novel spam before they try my account.

I also subaddress [1] any address I give to a vendor. So when those leak to spammers, I route those to sa-learn as well. It's a real pleasure to use the key spammer advantage -- indiscriminate volume -- against them.

[1] https://tools.ietf.org/html/rfc5233

I use Gmail because I know I'm not going to be able to keep an email server secure over time. But I still have to deal with shitty newsletters and other junk. After putting up with it for too long I decided to just domain-filter people that send me newsletters. The only thing that caused me trouble was LinkedIn, where people would occasionally message me expecting a fast response. So I solved that problem by deleting my LinkedIn account. If I reset my, say, Twitter password I know where to find it.

All other email from them and countless others is useless.

I'm no l33t hax0r or anything, but when I see "I've hosted a personal mail server for 10 years", I think, "I could probably own that person."

Almost certainly false (I couldn't even think of a more technical word than "own"), but I just dunno why people take that risk. I get that the threat profile of Joe Schmoe isn't the same as Car Company, Inc., but still...

I find the "anything I don't like goes into spam" is among the biggest problems for classifying spam using rules shared across users, and has been forever. We support training the classifier in our products, via a couple of different mechanisms, but don't really make it too evident how to enable it and use it, because the result is often a lot of confusion as users break their classifier over time by putting legitimate (but no longer wanted) mail into the spam folder.

Unsurprisingly, GMail also exhibits this problem. I have to check my spam folder regularly because it gets a ton of false positives these days. Nearly always newsletters and such I've signed up for. Another category of common false positive, which is admittedly harder to cope with, is moderation notifications from my WordPress sites...it's often actually a spam post, but it seems to learn the shape of those messages and start to treat them all as spam, even the legitimate ones.

Spam filtering has always been, and continues to be, hard. It seemed, for a brief shining moment a few years ago, that spam was on the run...but, it's come roaring back in recent years, I guess as cloud-based mail services made it feasible for spammers to bounce from one provider to another, to keep sending through "known trustworthy" senders. There probably ought to be some way to punish those providers that look the other way when used for spam, but blocking them completely is likely to impact non-spammers (but might be a necessary pain to reduce spam).

I forget at times that I originally started using Gmail because of the spam filtering. My main problem wasn't getting spam, it was getting alerts on my damn phone that I got spam. I don't mind checking the spam folder, as it's like having a bunch of people at the front door waiting for me to come outside. If they all knocked, I'd never go outside.

I've more recently switched to rspamd from spamassassin. I've been using SA for more than a decade on my own server, but have been getting frustrated with its spam matching more recently. I've tweaked it and tweaked it (including doing things like telling it to trust pyzor more) and still obvious stuff is leaking through. That's even with a cron job running to train itself on email I move to a specific folder. There's certain patterns of spam that just won't go away.

CPU usage was noticeable for it, too. I started looking for compiled alternatives, and rspamd fit that need. It's faster than SA, and using less resources, and seems to be hitting better than SA filtering.

I'm a happy SpamAssassin user too, just wanted to add that I've found using sa-learn to teach it about ham/spam makes the spam detection a lot better. When spam gets through, I move it to a junk folder, and from time to time, use that folder to train SA on. The level of false positives is tiny, perhaps 1-2 emails a year.

Did the same. From Qmail/Courier to now using Postfix/Dovecot. But am not using Spamassassin. Instead, filter all TLDs except those that I white listed. Then by reverse ip, dmarc and sfp check. And lastly block domains from which I have received a new mail and marked it as spam from the client side within 24 hours.

conventional wisdom is to filter unwanted newsletters and never click the unsubscribe links. Clicking those links confirms that your email address is valid and being monitored which can make it more valuable when the company shares and sells your email address to others. Also attackers can forge newsletters from respectable companies or just made up something new entirely and the unsubscribe links can be used to send you to malicious sites that can open the door to malware infections and phishing. When someone sends you trash, you should assume it is toxic and avoid interacting with it as much as possible.

I encountered same thing, malicious spam, & logged it at https://gitlab.com/davchana/gmail-indian-spam-domains/blob/m...

When I said legit newsletters, I meant the sort you get from sites you've already signed up for. Newsletters that arrive out of the blue are not legit and you should of course never interact with those, but that's self-evident for the reasons you stated, and the whole point of me mentioning legit newsletters specifically was to point out that these shouldn't go in your spam folder because they tend to look too similar to legitimate notification emails, and you only end up skewing the filter toward false positives.

I use a similar setup. Spam is simply discarded, whereas mails from all whitelisted users will force autolearn. Any mails thrown into the spam folder will be passed on to sa-learn as spam (and submitted to SpamCop and Razor2).

So far, no users have complained about false positives being discarded, and moving mails to the spam folder usually results in similar mails being blocked in the future.

The Bayes filter is very, very effective on it's own, but as you mentioned it's not perfect. I've found that using Razor2 as a compliment is very effective.

I have the same postfix dovecot setup and would love to setup Spamassassin correctly. Do you know of any tutorials which outlines how?

I used to operate a mail server for my company, but it was just too much extra work to get sa working company-wide. It just got pinched out of the list of things I was willing to spend any time on because it is such a nuisance.

You can also use the same technique to categorize your e-mail in more categories then just spam and not spam, I for example have a category for "commercial", eg. newsletters and offers, kinda like Gmail started with some time ago. But I also have more categories, like a category for each project.

I wish I had the patience to train Bayes properly. The problem in my case is coming up with a good method of feeding it ham (not just for my own mailbox, but for every user in the company, who obviously don't all correspond with the same people or utilize their mailboxes in the same way). I end up doing a LOT of manual configuration, my local.cf is currently 178 KB (which doesn't sound that big, but it's a lot of rules to run against each and every message).

I use Roundcube for webmail, which auto-collects outgoing addresses in an address book (and you can obviously manually add addresses as well). I have a script that periodically exports the address book for each user and uses that as a post-Bayes filter; anything from a known address is automatically whitelisted, and if it would have been marked as spam by Bayes then it is also trained as ham. It works pretty well.

Hey - could you share some of your SA rules/config that are working well for you?

I tend to find that when I manually categorize spam/ham and sa-learn, spam goes to 0 for a couple of weeks, then creeps upward as trends change (?!) or as they learn (?!?!).

Cool setup! How do you check false positives? You take a look at the spam folder for legit emails?

Not the OP, but I have a similar setup, however instead of Postfix I am using OpenSMTPd, which I found much easier to configure.

Checking for false positives is a manual process. I've never had legitimate mail marked as spam though, only the other way round.

My experience with Gmail's filter is not as nice btw. At work, I had a couple of situations where important mails from a customer were flagged as Spam, which went unnoticed for two weeks.

Pretty much. I just check it occasionally. Did the same thing back when I was a Gmail user, because Gmail has false positives too from time to time.