Hacker News Reader

Whatever works for you, but changing 20 lines of code in this to use asyncio will send your throughput through the roof.

View on HN

The limiting factor for a fast TCP scanner is not your favorite async toolkit.

Well, no, of course not. Don't be stupid. The default ThreadPoolExecutor size is the number of cores you have on your laptop. In my case, 8. So that's 8 concurrent connections at once. That's, frankly, shit.

Use asyncio (or roll your own toolkit, or just spawn 'nmap') and you get 20000 concurrent connections.

Which is better?

You're conflating concurrency with parallelism.

Your 20000 concurrent connections with asyncio are not parallel. You can have 20000 threads if you want.

The connections are actually handled by the OS in parallel, it's the callbacks to Python that are not. Why do you need a whole OS thread to handle sending a SYN/ACK?

As you well know 20000 threads is not a great way to do anything. Especially in Python.

I agree that threads in Python are not great but 20000 threads is otherwise not an issue.

Are you serious? Have you ever tried starting 20k threads?

Are you sure about that? Have you ever tried starting 20,000 threads? Try it.

Not using something to deserialize all that blocking I/O waiting on the internet is THE limiting factor.

Reply on news.ycombinator.com