Hacker News Reader

Comment by chrissnell | Hacker News Reader

I've run a few large vBulletin forums for years. The biggest problem with internet commenting isn't the software, it's the spam. I wish I had a system that I could train with 10 years of user registration and comment moderation history and have it do the tedious, daily pointy-clicky grind for me.

View on HN

I setup https://blogspam.net/ to offer an API to do exactly that kind of real-time spam detection for blog/forum comments.

It works. But it is pretty coarse, and doesn't support training. (It used to have a bayasian filter setup, and you could train ham/spam, but I found too many reports of false-positives so this was removed.)

I've seen plugins written for wordpress, drupal ,etc, but it never really took off the way I hoped it might. Still it does mean my own blog is spam-free.

Interesting, a few questions :

* Is it like an open source alternative to Akismet?

* Is it supported by the debian project or is it a personal project?

* What are the specs of the server that hosts this service?

Yes it is like akismet, which is open-source.

It is a personal project, although I did use to run it upon the https://debian-administration.org/ website (which I created and ran as a personal site). I'm not sure why you'd ask about a link to Debian though!

The system was recently rewritten in golang to improve performance, and it will run happily with 1Gb of RAM. It's running on a host that does a lot of other stuff, but basically it is an idle host. The biggest issue is that we store "known-bad" IPs in redis as a cache, so RAM can grow just due to that.

> The biggest issue is that we store "known-bad" IPs in redis as a cache, so RAM can grow just due to that.

How many requests do you serve per month? I see 50 million total rejected comments, but even if you serve a few million requests per month you should be fine with a traditional database like Postgres or MySQL.

I don't keep track of connections to be honest, but I know that I see a fair amount of incoming requests every minute. It goes in waves.

For transient data, which only lasts for a few hours, I'm happy enough with redis - I avoid running database servers for things like this, though I've got a couple of sites that use sqlite as a back-end.

Forcing sign-in to be "log in using email" wiped out +90% of spam for the ~300 forums I run.

I use auth0 now in a passwordless setup, and it requires either Google Auth, Windows (Hotmail) Auth or a one-time-token being emailed to the person (with a 15 minute expiry).

It has required that I pay for Sendgrid to ensure the email delivery... but it's worth it. I deal with virtually no spam nowadays and the biggest cure was this one thing.

When I moderated a mid-sized forum, pretty much 90% of our spam had a URL in the first post, whereas normal users were pretty much the opposite.

It seemed like sending those direct to moderation would have removed a lot of the work. In the end we just moderated all first posts since it wasn't that much volume and I didn't want to modify the software myself.

Is your spam more complex? I've noticed that it seems to be getting fancier.

A moderation queue for everyone's first post is among the best solutions if you can minimize approval time. But you need some impressive manpower for that, like mods willing to get notifications for each first post.

Right now I post registrations and first-posts to a #forum-activity channel on my forum's Discord.

The forum spam problem is easily solved if you ask a question during registration like: "What are the last four letters in the name of this website?" Anything that requires momentary thought from an actual user, and is specific to your website, will throw bots off.

This only mitigates spam if your forum is small enough for spammers to not care. Those questions are a mere extra parameter on the registration post request. Spam software automatically detects those and asks a human to fill it. If the question is static it's even easier than a captcha solver.

Spam is done by massive botnets that crawl the web at a rate nearly equal to Google's.

And they have their equivalents of Amazon's Mechanical TURK to learn how to bypass security questions.

https://2captcha.com/public_statistics <- this for example

You can find humans willing to work for $0.001/captcha.

I've also read that some torrent or other illegal download website display captchas that are in fact captchas from other websites, to crowdsource the captcha bypass.

Correct. There is a long list of human farming options they employ.

They frequently use that one then "pretend" they have workers.

As someone who uses stuff like that to reduce spam, it cuts spam by about ~50%.

Many of them rely on human operator intervention for forum registration when the automated method(s) fail these days.

Have you tried using reCAPTCHA? It even has an "invisible" mode: https://developers.google.com/recaptcha/docs/invisible

Reply on news.ycombinator.com