The only complicated part that I've run into when writing crawlers is accidental tarpits. It's very easy to run into a situation in which you're repeatedly requesting the same content via many different URLs.

For example, when a tracking parameter is added to any URL within the site:

http://example.com/?cid=104484&pid=12002348&ref=1294902

http://example.com/?cid=104484&pid=12002348&ref=1294904

http://example.com/?cid=104484&pid=12002348&ref=1294905

http://example.com/?cid=104484&pid=12002348&ref=1294906

You can quickly get to billions of permutations for a single site. The canonical tag solves the problem when it's there, but I still haven't seen a simple solution to the problem when it's not.