I have a web scraping script written in Python that I want to make multi-threaded. It scrapes web pages from a list, and enters results into a DB. Can someone (the author maybe) show me a simple example of how to make a multi-threaded Python script?
# Assume we have functions GetUrls() that retrieves a list
# of the urls we want to get, and Download(url) which
# downloads the content of a url and sticks it in the
# database.
import multiprocessing
pool = multiprocessing.Pool(processes=100)
urls = GetUrls()
pool.map(Download, urls)
http://docs.python.org/library/multiprocessing.html
http://docs.python.org/library/threading.html
Also, if you're entering results into a database, the easiest way may be just to spawn multiple python processes from the command line.