Regarding CommonCrawl
How many websites block CCBot
How many block GoogleBot
https://www.google.com/search?q=inurl:robots.txt+CCBot&num=1...
How many websites block CCBot
How many block GoogleBot
https://www.google.com/search?q=inurl:robots.txt+CCBot&num=1...
This claim can be verified. Archive.org has good collection of robots.txt files. CC itself has collections of robots.txt files.
If CCBot has been in most robots.txt files since 2008, it would be difficult to argue the reason has been to stop usage of crawls for training AI.