University libraries are being swarmed by AI bots, putting public access at risk | Hacker News Reader

University libraries are being swarmed by AI bots, putting public access at risk

cbc.ca

134 pointsblakesterz63 comments

LLM bots are the easiest to spot. Rate limit them to 16kb/s and 1 request every 5 seconds. Give them a friendly multilingual WCAG status 429 page that says to slow down or status 444 and drop them into nftables ipsets if they continue to be aggressive. Ask your favorite LLM how to set this up. If your LLM gets session paused, ask Gemini or Grok. Tell it if you are behind a CDN. Enable logging of ssl curves, ciphers, etc... Show your LLM some examples from your access logs.

I get tens of thousands of IPs making 1 or 2 requests against pages on The Consensus (job search, code viewing, etc.). They're not super expensive pages. But I don't want to deal with the bot traffic (mostly because it messes up view counts I use to sell sponsorships). But then if I require an account (even a free one), valid readership is also going to drop with the new friction.

I could build something more complex to discover the patterns, or use Turnstile maybe. I'm not really sure at the moment.

Bots give all the information required to block or rate limit them including those that try to look like a real browser (some things they just can't hide, even residential proxies via behavior.). Ask the LLM how to change your log format to gather all the fields your web server can see. Share the resulting updated logs with your favorite LLM. Ask it for generalized rules to block the bots whilst permitting real browsers. A tiny percentage of bots will still get through but the load becomes much more manageable. Tell your LLM you also wish to spot fake chrome via CH and bots via ssl curves. No need for JA3. Nginx can expose the important aspects of GREASE.

I know you can't be super specific because you'd just be enabling the companies who get around this (who, funnily, I also index on The Consensus), but I have spent a few rounds on this over the months and haven't yet been able to cut them out.

Some of the things that seemed like they'd cut out lazier bots/scripters involved browser fingerprinting that I didn't want to yet have to deal with disclosures or approval for. But also even some of these browser fingerprinting things were disable-able via flags when you start up Chrome or whatever.

I think what would work overall is to track all requests for a pattern of page access and start blocking other systems that are part of that pattern. But that also wouldn't account for like... landing on the front page of HN, which happens not infrequently.

All of this is aside from per-IP or per-user proof of work. Whenever someone lands on a more expensive page I make the browser do proof of work and store it in a cookie, unique to the IP or user. (Signed in users don't have to do proof of work). This is a perfect solution except for the IP rotation I'm talking about above.

edit: ah also, I don't do SSL termination so can't spot the SSL configuration. But Cloudflare does. And all of this is even after Cloudflare paid bot traffic systems are in place.

edit 2:

> tiny percentage of bots will still get through but the load becomes much more manageable.

The load is not the problem. It's only at most a few bot hits per second. I'm just having trouble removing bot attribution from view counts.

Aside from the things I suggested asking LLM's (they will take one down some fun rabbit holes) there are more crude methods such as forcing brotli or zstd on specific URLs accepting that older versions of Safari pre-2026 will get gibberish if forcing zstd. It's rude and paternal but botters can be rude and childish so these are the games we play.

I love sharing methods with the botters so they step up their game, thus forcing me to step up my game. Everything I use does not require javascript, cookies or authentication. I save authentication as a last resort which means I lost that round.

Botters are welcome and encouraged to test their setup on my silly hobby sites. Here's one of them: [1]

[1] - https://altchan.org/sh/33

P.S. These are the simplest crude methods for the obvious bots, let your LLM to help spot the fakers. pre-2026 Safari will not work. Do not visit image boards with unpatched browsers.

You can consider adding in a small amount of js that tracks if a person actually engages with the content before "counting" their view. Eg, did they stay on the page for more than 15 seconds.

Youtube wont count a view until you watch for some percentage of the video. Maybe consider something similar.

Yeah I do have the JavaScript side in place, was just loathe to start totally owning this data instead of Fathom.

You could still have fathom own it, but create a sort of small proxy that takes in the view requests and then forwards them on to your instance behind the scenes. That way you can be responsible for counting who is actually a view and fathom can own the data and do all the BI things.

Just my two cents, you'll figure out a good solution.

Funnily enough, Youtube recently changed how they count views to be more in line with its competitors, so now they do count views as soon as the video starts playing.

seems like open web is dead which is sad

i heard hcaptcha is good

Static web pages might be back for a bit while this gets figured out

I mind a public access website for a government entity and this is exactly the pattern I see. US residential IPs make a couple of requests then never come back.

The bots may have layer 7 signatures that make them "easy" to block, but I'm dealing with COTS software with no ability to make changes to the stack. Even if I could by the time they're blocked the work is done and they're never coming back anyway.

I don't think I'm seeing the LLM bots in the scraping on this particular site. This site's data is highly sought-after by data brokers, legal services advertising, debt collectors, etc. I tend to think I'm seeing homespun scrapers. This site has been up for 20+ years and the scrapers seem to be well-adapted to using the HTML as an ad-hoc API for the data. The difference in the last few years has been the availability of vast numbers of residential proxies to mask the traffic.

LLM bots make only a part of traffic and really easy to set up (and I think this will eventually throttled by OAI or Anthropic themselves).

But there are bad actors - bots that try every possible urls and traversals and cross scripting and so on… I see it in our access logs. This cannot be solved otherwise than some greylisting/blacklisting on ISP level. E.g. this is so huge that we have started to block all Chinese inbound traffic. I am sorry for that because Deepseek and other good actors are probably blocked as well but IP rotation made it impossible to filter bad ones only… (and I suppose that DS uses EU or US addresses as well so they shouldn’t be blocked in full).

eventually throttled by OAI or Anthropic themselves

Indeed, they use an edge SSL Bump proxy that is already bandwidth limited and has to speak real HTTPS.

I do not focus too much on IP addresses but rather the behavior that they can not hide. LLM's can help you spot this behavior. Good LLM vs. Bad LLM. Gemini knows a few tricks but you have to coax it out of them. It seems to not want to overload people with information. That said there are some good IP lists that can be used in blackhole routing tables, nginx geo tables, etc...

> That said there are some good IP lists that can be used in blackhole routing tables, nginx geo tables, etc...

I focus on content on our site so blocking part of internet with wordfence was brute but as life has only few thousands of weeks, it was the least I wanted to think about :).

Would you mind to share a source of good IP lists pls?

FireHol [1] and lists_vpn [2] are useful for nginx geo files. Be sure to use iprange [3] to aggregate cidr ranges, IP's or lists get big fast The IP ranges can be used in conjunction with rate limiting and/or firewall ipsets and/or blackhole routing tables depending on what type of abuse your site is receiving.

LLM's can help you implement iprange into IP list building scripts to keep the lists smaller and more efficient.

[1] - https://github.com/firehol/blocklist-ipsets/

[2] - https://github.com/X4BNet/lists_vpn

[3] - https://github.com/firehol/iprange

Thanks. I will definitely explore these in production.

You're welcome.

Before pushing such things to production consider first testing in dev/qa then staging or some canary environment. I use those IP's to determine who gets into a moderator review queue so worst case it just makes more work for me. Consider using those in conjunction with whatever rate limiting capabilities your web server provides first so it is not a hard block in the unlikely event a real customer shows up on one of them.

I think you overestimate many organizations ability to implement this type of rate-limiting.

I think you overestimate many organizations ability to implement this type of rate-limiting.

I was told that I could only lead a horse to water but I could not make it drink. Turns out that was not at all true. I had to find which horse was being the ass and re-educate it so the yearlings could get to the water.

That said I think most people here can at very least have discovery conversations with LLM's to explore their options. They can share their architecture, infrastructure design and have LLM's walk them through gathering more information. It may be bumpy at first but I suspect some of them will be able to give the bots a few flat tires. I have confidence that necessity will drive innovation and they most likely will not even have to spend a dime.

the trash hides among the agents. bots are just a front. the diarrhea of persons suuuperhappy they get to "hack" now (a few years plus minus) ain't so easy to spot, especially since one rather passes them without smelling or seeing or hearing them. there's professionals for that and we should all hope cybercleaners and cybercrime divisions get the manpower they need. it's pretty disgusting. and I don't understand how so much AI content on so many platforms is still untagged or at maximum the info is somewhat "fineprinted" ... disgusting, it's all so fucking disgusting and retarded

What I still don't understand is who is doing it? I assume it's not the big companies like OpenAI or Google or Anthropic, right? So there are a bunch of small things out there doing the scraping and then they sell it to the big places? Or are these just smaller AI models doing the scraping and using it for themselves? Is there an underground ecosystem where people buy and sell this stuff? I understand why it's valuable to the AI companies I'm curious about how all this work, the economics of it all.

Can't speak for everyone. I'm very gently scraping a lot of libraries for public domain periodical scans. The scans often exist nowhere else, or not in the same quality. Sometimes I find a certain library has perhaps the only known remaining intact copy of a periodical or a specific page.

The problem with the libraries is that all these scans are behind walls, so while the texts are public domain nobody can easily access them. I'm trying to liberate them and re-upload them outside the walls.

Until recently a lot of libraries would allow you to remotely sign up for a library card, but in the last 12 months almost all of them have reverted to having to bring ID in person to the library to validate your account, which puts the brakes on things.

I've been scraping Overdrive for the last few months, so that I can identify every book/magazine title they hold. About halfway done (13 million ids to check). I can slurp down full magazines from their preview function. Books I can only get through my local library's Overdrive service, and they only have a tiny subset of those.

Are you doing newspapers?

Definitely not newspapers. I would love to, but the scope creep would kill me. I estimate there are 100 million magazine issues published over the last 200 years across the world. That will keep me busy enough :)

Overdrive has a pretty limited amount of back issues for magazines, which is why I generally ignore it, except if I want to quickly read something on my Kindle.

Do you have a plan for all the data once you have it?

I'll probably download all the magazines. They keep releasing new issues (for the few titles still printing). They claim that they only keep the back issues for the last 3 years... but all they do is make those impossible to find with their search/browse features. If one has the ids for the individual issues, they seem to have 100% retention. So I'll be able to get those... there are 4 or 5 titles I can't get current issues of otherwise.

I think your estimate of magazine issues is high, but I commend your effort. I wouldn't mind hearing about which you've managed to complete. I keep finding that I can get all the historical issues, and I can get all the recent issues, but there's this hole in the middle from about 1985 through 2005 that is usually impossible. It's irritating.

I might test if I can pull audio books down too, though those will be limited to the ones appearing in my own local library (every municipal library seems to have access to the full magazines though, if you know the magic urls).

Fire me an email (on my profile).

note that doing this directly costs your library lots of money per book/mag

I've been looking at historic newspapers recently, its bothersome and unfortunate that many (most?) are all behind newspapers.com (ancestry) paywall.

Yeah, I'll be on ancestry.com and it says "We found this article about your great-grandfather!" and then you click it and it's newspapers.com and I have to close the tab.

What's the solution, though? All these newspapers won't scan themselves. I'm setting up a non-profit right now to deal with the magazines. The rights issues are both better and worse with magazines. "Better" in that most magazines have gone out of business and the rights are cheap. Worse in that most of them are orphans and the rights are very hard to track down.

We're currently in the middle of an AI goldrush. There are probably hundreds of companies trying to do model pretraining. Anthropic and OpenAI and so on probably care about optics and implementing their crawlers properly. But there will be many firms few of us have heard of. I would guess some of them are Chinese, for example.

Some people making money off it, it's a whole value chain:

- Residential proxy providers

- Hosting providers with IP-rotation as a service.

- Hosting providers who chase AI training customers, DigitalOcean is a big one

- Clouds like AWS with headless browsing as a service

- Platforms like Replit whose value proposition starts to fall apart if clients can't make requests from their platform.

- Companies like meta, whose existing bots are now obviously completely unchained from their advertised responsibilities

As I understand, most large companies now are relying on a variety of contractors and the constractors try to vacuum up everything so they have the maximum to sell to different companies.

This is the way for companies these days to avoid legal responsibility for their actions - hire a contractor on a "don't ask, don't tell" basis.

Of course such things are extra-ironic in the context of US companies complaining about having their condensed by Chinese companies.

Edit: I read this in one article a while back. I can't links either corroborating or refuting it now and I think that's again because whatever the companies do, they want to keep it hush-hush.

Same thing on the tiny scale. My side project site gets maybe 2 real visitors a month and a steady stream of 0-second "visits" from US datacenters. For a library with real users that load must be brutal.

Behind nearly every AI bot is some human who will see real benefit from scraping my site.

I just make sure all my pages are super lightweight to cache and serve from cloudflare and let the bots do what they like.

If I put something on the internet for all to see, I want bots to be able to see it too.

I've been on the other side of bot blocks - when I ask my LLM to buy me X and it comes back and says the only place it could buy from is some Bulgarian seller because everywhere else has anti-bot scripts, I make sure to give that Bulgarian site my business.

Big agree.

While I am fully sympathetic to the bot scraping problem and support high quality public data on an open internet, I can't help but think the whole system needs a re-architecting.

Can we re-deploy the internet as statically hosted data available through a defined api? Something like DNS for data; eventually consistent and evenly distributed. Either the data is open and ubiquitous, or closed and inaccessible.

In what world is 11k searches a lot? The story here is that AI bots are exposing University Libraries bad setups. I haven't heard any institution here in northern europe having any trouble.

It was 11,000 searches on just one topic (Finnish music). Not sure how many topics.

Agreed, this just reads like the universities have no idea how to actually serve data effectively. Just bad architecture and lazy or incompetent administrators.

Couldn’t possibly be that serving data to scrapers isn’t a core function for a university or that they can’t pay big tech salaries to draw sysadmins and engineers that know how to handle large scale deployments

If the core function of the university library's archival system is not providing free access to their collection, then what is its core function?

The core function is being obstructed by irresponsible AI bot traffic taking down their catalogs and archives and preventing humans from having access. Bots have never been the intended audience of library content.

> university library

First function is archival and preservation, second is access for university students and researchers. It’s not Anna’s archive or TIA

Free access for its student body and research community wouldn't have required to be able to handle 11k request a day for a single item.

I would encourage you to re-think your comments. You just labelled some people "lazy or incompetent" while having barely any knowledge of the situation, or what resources they have or don't have available. Libraries aren't start ups focusing on scalability, their focus is on preserving knowledge and helping academics and the public find it.

If you believe that they should focus on the scalability required to handle bot traffic perhaps you would be best donating money to them to help give them the resources to be able to do that, or volunteering to try to help understand their problem and fix it if you believe that it's a skill issue.

Reframing what you said in a more positive way could be: - I wonder what architecture they used for this and how it could be improved? - I wonder what resource limits they have that prevent them from handling this issue? - I wonder whats unique to a library that limits their ability to do what tech companies do?

Well I have experience with universities, and their IT is horrendous. Departments are little kingdoms, none of them agree on procedures, architecture, data formats or even care about the absolute basics of software development or security.

It really isn’t a resource problem. It’s that academia is ruled by little princes who are full of themselves.

No one is lazy or incompetent, but they built and use systems that are scaled for pre-LLM usage. Now they need to upgrade and they should. They challenge is of course who pays for the extra capacity?

I’m in the process of developing a WAF reverse proxy designed for the smallweb and selfhosters to address precisely this.

https://git.eeqj.de/sneak/smallwebwaf

It’s alpha but it should be ready for real-world testing in the next couple of days. I’m going to drop it in front of my Gitea and home-grown api servers.

I’m super tired of Anubis spamming me with furry fetish art on its pointless interstitials 20 times a day. As someone pointed out in another comment, LLM scraper bots are super easy to detect with basic heuristics. PoW and session cookies and an interstitial are massive overkill and annoying spam.

Honestly the constant accusations of it being fetish art are burning me out and making me want to give up on the project to go do other things where people won't have this kind of accusatory behaviour.

All you have to do is stop putting catgirls on millions of screens where people don’t want catgirls and the entire class of issues you are complaining about stops instantly.

Dislike for your transparent attempts at promoting whatever your ideology is (in public-facing pages in a reverse proxy, of all places) isn’t unreasonable or persecution or open source userbase entitlement. You are free to push furry stuff into the faces of unrelated third parties in unrelated software and we are free to say it’s lame and inappropriate and unnecessary and build better alternatives. You are not being victimized. If the negative feedback from your pushing fetish art to millions of eyeballs in an audience that isn’t yours makes you burn out, imagine what you would feel from the hypothetical feedback to something you built that promotes a more controversial portion of your personal ideology, whatever that may be. FWIW, I fully support the concept of social activism via unrelated avenues that access lots of eyeballs to promote a cause. You wrote the software, you are totally allowed to try to normalize furries via the broad social scope of the UN’s website viewership. I’m simply surprised that you are surprised that people are complaining. Most of the people of the world aren’t furry fetishists, and it’s not simply because they were ignorant of the fact that catgirl art is available on the internet.

Interstitials are super annoying even if properly designed to be efficient and text-only, and I believe them to be totally unnecessary for this specific Gitea-like protection use case with a little bit of WAF smarts. The majority of my technical complaint with Anubis is the how, it’s just the what that elevates it to actively irritating (I use NoScript, naturally).

Also, your tool exists to stop scrapers, and I think the main benefit of the web is programmatic public access; CAPTCHAs are the worst thing to happen to sysops AND users. Scraping is a super important use case, and things like this stop people from politely scraping even a single page of text. That’s very bad and if lots of people deploy this then the web gets Very Bad For Everyone. The idea that a site operator can or should tell you what kind of HTTP client you can use to look at their webpages is offensive to me, because a web server that responds to a request is explicitly agreeing to service that request by sending a response to it. (You may recall that Signal and Discord have both tried this shit.

I’ll post an update in a couple weeks of beta testing. It’s possible my theory is wrong, and that stopping bad bots requires more than heuristics, but I doubt it; their badness makes them stand out and thus easy to discriminate against.

So these bots are going to get the Aaron Swartz treatment, right?

RIGHT?

I wonder if Internet 2: The sequel is infested as well.

https://en.wikipedia.org/wiki/Internet2

University networks in general. Don't ask me how I know.

Get with the times, university libraries.

"Open Access" means supporting users. Now your users are also bots, and the people that deploy those bots. You need to be able to serve those users too.

Bots are running to make cash for some evil companies. The library is for knowledge - not profit ir commerciaisation.

These days, AI attacks are indiscriminately targeting personal websites, and they've become far too severe. They even come in disguised through referrer spam[1], so they're tricky to block. In my opinion, I suspect such scraped data is being used to classify and train AI models.

[1]https://www.makonea.com/en-US/casual/scraping-bots-are-destr...

11 THOUSAND searches in one day? What's the problem?

If anything, libraries should be becoming more AI friendly, not less. Instead, we get this sort of bullshit from the ACM:

https://dl.acm.org/doi/10.1145/3832783.3834511

It's a canonical DOI on what was a download server. It says exactly which file is being requested. (I chose a random recent one.) At the very most, they should serve a static page with a bibtex entry, and a few other metadata fields, and a link to the PDF and whatever other formats are available.

Instead, we get piles of captchas, login-gate spam about how they're "open access" now and have some initiative to be LLM friendly.

Getting back to the problem of people doing stuff that requires access to library materials: What sort of server-side cost multiplier do you suppose this ACM page has vs. just serving a few S3-compatible URLs that point to a few mirrors? (Or, got forbid, Anna's Archive / libgen, since ACM is open access, and those sites would be doing nothing wrong by mirroring ACM content.)

The problem is unintentional DOS due to traffic can constitute a felony under the CFAA; the case in question being Pulte Homes, Inc. v. Laborers' International Union 2011.

After Pulte fired an employee represented by the union, LIUNA urged members to call and send email to the company, expressing their opinions. As a result of the increased traffic, the company's email system crashed. The Sixth Circuit ruled that the LIUNA's instruction to call and email "intentionally caused damage," reversing the lower court's decision.

Now thankfully we have Sandvig v. Barr, 2020 where the DC Federal Court ruled that the CFAA does not criminalize the violation of a website's terms of service, but imo all bets are off in terms of interpretation of the law in the wild west of AI.

I was not aware of that (terrible) court decision. Wow.

I suppose this is one of those laws that doesn't apply if an individual is a victim? I have a email box that sometimes fills with spam and rejects legitimate email. Can I get the corporations that filled it up to be charged with felonies?

Not a hope. Anthropic have proven that when they admitted to indisputable breaches of the CFAA twice.

(1) They admitted its Claude models had breached the systems of three companies since April; including Huggingface.

https://www.bloomberg.com/news/articles/2026-07-30/anthropic...

(2) They secured unauthorised access to a U.S. state government website hosting election-related data, pulling aggregate vote-count statistics from a gated government dashboard. The state government itself confirmed the access was entirely unauthorised.

https://seekingalpha.com/news/4657023-anthropics-claude-bypa...

If a human security researcher had deployed an agent that behaved this way, it would constitute a textbook felony under the CFAA for accessing those systems without authorisation.

Unfortunately different rules apply when you have the power to buy off Congress and the Appeals Courts.

Most recently a US appeals court ruled on 5 August that Amazon was unlikely to succeed on a claim that Perplexity's AI agents violated the Computer Fraud and Abuse Act by covertly accessing private Amazon customer accounts.

https://www.rte.ie/news/technology/2026/0807/1586941-artific...

It's 11 THOUSAND searches ON JUST THAT ONE TOPIC in one day when they are used to seeing less than 11 THOUSAND searches in one day TOTAL. None of these systems were built for this type of traffic. The total traffic from these crawlers is many times more than just 11,000. It's more like 11,000 searches on 100,000 topics.

But, per groups of queries, the worst they've seen is 11 thousand, and that's across many libraries. So, some tiny fraction of their users amplified their traffic by 11,000x by using a misconfigured LLM. I've done that a few times without LLMs (who hasn't passed --recursive to wget without remembering the -np?)

Also, if they just turned off all the user-hostile crap, they'd probably get at least 1000x back on their serving costs. How much does it cost to serve a login gate, captcha and rejection screen vs. a ~1MB static file?. What about the human cost where actual users cannot get access to the data they should have access to?

Just now, I ran a Kagi search. Its LLM summarized some page on the internet, but when I went to check the reference, I got a "sorry, humans only" access denied error message. I guess it's good that I could ask the LLM to quote the document verbatim for me in a follow up message, but what fresh hell is this? (Also, the error message screen was clearly longer than the document I'd requested.)

Reply on news.ycombinator.com