Hacker News Reader

Comment by abrookewood | Hacker News Reader

Why on Earth not??

I imagine the commenter is worried that if it's open source, publishers can read exactly how it works and then adjust their setups to defeat it. Vs I suspect publishers already know how it works, or could figure it out from their own server instrumentation, or by inspecting client code (I haven't looked at this project but I know a few I've seen previously work mostly with JS and CSS modifications).

Of course they know. I have seen local newspapers who managed to block archive.ph's mechanism. If a bigger outlet does not use technical measures against services like wallhop, it is safe to assume that’s deliberate.

It makes sense economically since the part of the population using such services is so small and may have specific demographics that justify specific treatment just from a marketing POV.

Then you want search engines to get the whole content (okay at least before they started to summarise everything but the kitchen sink) - that will always leave a loophole.

I think you have that the wrong way around. Archive services have to explicitly bypass paywalls so most local newspapers are simply unsupported.

I elaborate: the local newspaper was perfectly archivable for years. Then it stopped.

The CMS usually allows search engines to get all content. The feature is to know human browser operators and make them pay, or know archiving/unblocking services and block them.

Based on some of my recent experience with building a scraper for purposes that aren't a news website, what has changed in the last 12 months or so is the availability of very low cost (or free, if you can run it on a 256 or 512GB system in your own office) LLM that are good enough to give it a target of something and have it build a custom scraper/paywall bypass profile on a per site basis. I'm referring specifically to things that can be run in harnesses and score well on terminalbench 4.0 and SWE LLM benchmarks.

When previously nobody would have gone through the effort to build and maintain a custom scraper against a moving target, for some small to medium size city's newspaper, now it's just one of a myriad of 'scraper profiles' you can have a near fully automated tool create.

Also it's one thing to shut one instance down.. entirely another to shut down a whole bunch (not to mention private instances)

OK, yep, makes sense.

Reply on news.ycombinator.com