Hacker News Reader

Comment by nomilk | Hacker News Reader

My Cloudflare solver is command + w. It's 100% reliable in a headful browser and occasionally results in finding an alternative site that doesn't place its priorities (denying bots) over the user's (judicious use of time).

View on HN

15+ years of expectations around how a website can be run were upended by scrapers for LLM training.

They ignore robots.txt rules and seem to scrape as quickly and as broadly as possible. Unlike search engines, it doesn't matter to the LLM provider if the website is offline. Unlike search engines, they take steps to be anonymous and difficult to block, including using residential proxy services and other shared IP space.

I try and keep an open data repository online, but it was not designed for this. Cloudflare is hopefully a temporary fix while we redesign backend and frontend components.

The end result is going to be worse for users — slower querying, less flexibility as we primarily expose what we can precalculate, shifting CPU and bandwidth load to the user's device rather than our servers.

Unfortunately scrappers are problematic enough that for more and more organizations alternative is to stop serving traffic.

For example OpenStreetMap is using nowadays this or equivalents to prevent downtime and/or runaway costs.

Doesn't work that well if you actually need to use whatever site is protected by Cloudflare. There often is a choice, but not always, these days.

You made me press command+w :-|

Luckily command+shift+t undoes.

For me its a bit nuanced, for a small website that may be constantly attacked, they may not have much of resources to work around except putting in cf safeguards in place, and I tend to look past that.

What I do not appreciate as well is companies with larger money pool also resorting to and not caring for, for example Dell has the stupidest security check that has disappointed me (a human) enough times one time that I chose to buy a competitors product.

every fuckass website with 0.5 daily visitors has it now. russian minitrue had blocked cloudflare and the internet became completely unusable without a vpn. like you search for something and not a single link works.

Because websites with almost no visitors are hit the hardest. A hundreds bots per minute hitting your server (if you host any kind of source code) versus one real visitor every other day is a great scenario to protect against.

The Russian government being stupid and evil isn't really Cloudflare's problem to solve. Neither is the lack of abuse prevention from VPN providers necessitating the extra challenges for VPN IP blocks. And, now that botnets are the norm, residential IP addresses too.

Obviously, Cloudflare's tools aren't the solution. The web needs to be closed down behind paid or real-identity accounts for websites to ditch these protections. It's only going to get worse from here.

>Because websites with almost no visitors are hit the hardest. A hundreds bots per minute hitting your server (if you host any kind of source code) versus one real visitor every other day is a great scenario to protect against.

a $10 VPS hosting a static website can trivially serve a thousand pages per second.

I have a few fuckass websites hosted together on the cheapest VPS I could find, and the last time I got an automated alert about downtime was a decade ago.

there is simply no incentive for anyone to ddos a random website.

>The Russian government being stupid and evil isn't really Cloudflare's problem to solve.

didn't imply it was, it was just an observation that everyone uses Cloudflare now.

I do this with google's bullshit free AI training captchas. If I see a site using those, I never use it again. I've canceled paid services for this before.

Reply on news.ycombinator.com