How to Mirror Wikipedia
igeek.co.za
igeek.co.za
wget --mirror http://en.wikipedia.org/wiki/Main_Page
so I decided to try it first. After downloading the robots.txt, it stopped. I looked and saw the following: #
# Sorry, wget in its recursive mode is a frequent problem.
# Please read the man page and use it properly; there is a
# --wait option you can use to set the delay between hits,
# for instance.
#
User-agent: wget
Disallow: /So, mirror what you can :-)
BTW: Is it possible to mirror incrementally? Downloading the 5.2GB file every time is a big waste of bandwidth.