That was really a very useful reply messel. However in our case the crawler code etc is ready. The only thing we need is some powerful infrastructure to run it on.
We have certainly considered all the factors you have mentioned while implementing our crawler however unlike typical crawlers the objectives of our crawler is purely a research goal and completely different than what most have done.