Most crawlers of any reasonable size need to use anonymizing proxies, throttling and shuffling of proxy IP addresses. All this starts to get complicated in bash.