Cracking Anubis Challenges

I'm the administrator of 3 online forums, 2 of which are basically archives at this point. All 3 forums were getting absolutely hammered by bots. In fact, all 3 were taken offline by their hosts due to the heavy traffic in 2026. I installed Anubis on 2 of them and bot traffic decreased dramatically to the point where it appears to be only legitimate bots. (Unfortunately, the other forum couldn't use Anubis so easily and has remained offline since then.)

It's clear to me that Anubis won't work forever. You don't need to get as sophisticated as this blog post either as you can bypass the proof-of-work challenge with the right user agent string. I think the reason I'm not seeing bots change their behavior in response to me adding Anubis is simply that my forums are relatively small players. To the people running these scraper bots, a handful of forums adding Anubis is no big deal to them as there are tons of other sites they can scrape.

I think turning the forums that are archives into static HTML is the best path forward for those forums, but that's easier said than done. I've also been thinking about writing a new forum software where all of the content pages are static HTML, and the only dynamic pages are where dynamic features are unavoidable like when posting.

Yeah, I implement PoW on my Edge nodes as well, which connect back to my self-hosted infra over Wireguard tunnels. The PoW works very similarly to Anubis, but practically speaking, they don't really work. Whole swarths of botnets, including botnets that utilise residential proxies (thousands of comprised desktops in poorer countries) just solve the PoW(s) and get through anyway at massive scale.

The only effective way I've found to combat "unwanted traffic" (bots, botnets, crawlers, search engines, etc) is to a) make request/responses as cheap as possible (heavy caching, rate limiting, etc) and b) measure cohorts, fingerprints and behaviour and block patterns that can't possibly look like "normal human traffic" (tricky to get right, but this is to a large extent what Cloudflare is doing I believe, and I've had great success with this technique and building out an OSS Caddy module for this...)

I had recently done some work where I had rehash a bunch of insecure hashes for security reasons. It required brute forcing about 1 trillion SHA-256 hashes to achieve this. Was the first time I had actually ever used Hashcat, very cool piece of software. I later had a shower thought about Anubis and it's proof-of-work solution to tackle web scrapers. I thought... What if I cracked the hashes out of the browser? Here is my write up of what I discovered.