> The Taco Bell answer? xargs and wget. In the rare case that you saturate the network connection, add some split and rsync.

If you're going to download millions of webpages, you'll instantly saturate your network io. yes you can split and rsync, but you'll lose proper error reporting, and ability to systematically retry, dynamic scaling, machine failure recovery, along many others that a properly designed system would provide you.

It often depends on what you require and expect from your solution.