Hacker News Reader

Do the AI labs in the U.S. publish the AWS IP address sets that their crawlers use? Is that not an effective way to block those crawlers anymore?

They are often from residential IP.there are even businesses that allow you renting such IPs

Nobody serious would ever use an AWS IP lol