Because there's a better way for bots to get the data via the wikipedia database dump. Sending some large zip archives is a lot cheaper than individually serving every page on Wikipedia.