It happened with me and many other former colleagues.
Though obviously, everyone decides for themselves when does that point come -- or if it comes at all.
It happened with me and many other former colleagues.
Though obviously, everyone decides for themselves when does that point come -- or if it comes at all.
tasks = {}
for url in URLs:
future = executor.submit(fetch_url, url)
tasks[future] = url
bothers you, this is perfectly (some would say more so even than the original) Pythonic: tasks = {executor.submit(fetch_url, url): url for url in URLs} def main():
with ThreadPoolExecutor(max_workers=len(URLs)) as executor:
for url, title in zip(URLs, executor.map(fetch_url, URLs)):
print(f"URL: {url}\nTitle: {title}")
The default value of `max_workers` since
Python 3.8 has been min(32, os.cpu_count() + 4)
You should probably avoid max_workers=len(items_to_process)
It will not save memory or CPU time when you have few items (workers are created as necessary)
and may waste memory when you have many.The amusing part is that the article calls out two groups of people into which your advice falls.
It’s not that much code, it’s about 4 lines of code, creating a “pool” and calling a wait on future objects.
This is a perfect solution for Python developers who have been perfectly happy using Django for years, and just need to scrape some API or download multiple files.
No, they shouldn’t switch to a different language the moment they need to optimize something embarrassingly parallel, they can see whether a simple solution in stdlib is enough, and probably move on.
Yes, I've written a shit ton of code in all aforementioned languages.
[1] https://www.jtolio.com/2016/03/go-channels-are-bad-and-you-s...
Is reading the official docs section on concurrency lots of research?