It was nice for interested guests to get an impression of what we are doing.
First the AI crawlers came in from foreign countries that could be blocked.
Then they beat down the small server by being very distributed, calling from thousands of ips one or two requests each.
We finally put a stop to it by requiring a login with a message informing people to physically show up to gain access.
Worked fine for over 15 years but AI finally killed it.
Also you're implying that the only way to crawl is to essentially DDOS a website by blasting them from thousands of IP addresses. There is no reason crawlers can't do more sites in parallel and avoid hitting individual sites so hard. There are plenty of crawlers for the last few decades that don't cause problems, these are just stories about the ones that do.