You should check out Manning's "Introduction to Information Retrieval", it has far more detail about web crawler architecture than I can write in a post, and served as a blueprint for much of Applebot's early design decisions.
The book is freely available online at https://nlp.stanford.edu/IR-book/information-retrieval-book....