Hacker News Reader

Hacker News

Comment by lfodofod | Hacker News Reader

Parent Full thread

lfodofod

What are you doing that 50k daily hits is even worth investigating?

View on HN

Looking at his metrics and investigating an anomaly?

Absolute magnitude is irrelevant. A standard deviation uptick vs baseline is always interesting.

You see it’s a scraper, shrug, move on with your day.

This is just standard background noise.

Running his personal blog. I'm sure he's a lovely guy and that, and everybody likes him, and his writings are interesting, and so on - but it usually takes a bit more than this before 50,000 visitors per day stops being meaningful.

I have a few not particularly popular sites and like 60% of the nginx log entries are presumably botnets (unless someone legit is scanning from residential IPs in Colombia) doing automated vulnerability scans. I still don't get why people get so worked up about this stuff. It happens every day and there is very little you can do to stop it. I've read some incredibly heated email exchanges that all started because some network admin is mad that someone is port scanning his IP space. Just use key based authentication and update your stuff. I don't really buy the load argument either - I hosted a site that was on the front page of HN for nearly a day on a $3/month Vultr VPS with 0 downtime.

>I've read some incredibly heated email exchanges that all started because some network admin is mad that someone is port scanning his IP space

There seems to be significant overlap between sysadmins who send handwritten abuse reports and “targeted individuals”/“gangstalking victims”.

Well, we may still worry about green computing. 50K requests/day per site isn't nothing, just for scans. Is does sum up considerably.

Looks like it's DWiki, which is a python wiki. An uncached page load took 571ms, ICMP latency is 100ms. Frontend server is Apache, forcing a Connection:close on every serve (may actually be an advantage against the bot, unless it is also fetching the assets).

Whichever way you cut that, if this happens to be a single host server, you're rapidly approaching a core-day with this load once you add some logging and such on top.

>Whichever way you cut that, if this happens to be a single host server, you're rapidly approaching a core-day with this load once you add some logging and such on top.

No way a scraper running at less than 1rps is anywhere near a core-day without some horrible misconfiguration, it should be way easier to address that than to fight a war against scrapers.

I have significantly more complicated flask sites handling 100+ rps on a single core, it doesn’t require any black magic.

Reply on news.ycombinator.com