everyone needs equal access to public data. right now only big tech can download the many web pages (without thottling or being ip banned) on linkedin (microsoft), youtube (google), facebook, github (microsoft) and billions of more pages. this also leads to a gap on AI training sets to give big tech even more entrenchment. for instance, only microsoft can build that ai coding application they did because other companies can't access all of github without being throttled or ip banned (last time i checked - but i could be wrong now)[microsoft owns github]. regardless, we need some sort of bot 'bill of rights' to ensure equal access going forward. perhaps the answer is legislation or perhaps it is some massive p2p proxy net. i think it is legislation because the p2p proxy net is too hard to implement, and it would have to solve turing tests.

but perhaps web 3.0 (dweb) can just bypass all this nonsense and make its own versions of these popular services with baked-in accessibility for all.