Hacker News Reader

Comment by jftuga | Hacker News Reader

I wrote a fast tcp port scanner using Python 3.5 and concurrent.futures.ThreadPoolExecutor.

See lines 123-124:

https://github.com/jftuga/universe/blob/master/tcpscan.py

I have used this under Linux, OSX and Windows. It's cool to add the Thread Count field in Task Manager and then see something I wrote use so many threads! I am more of a sys admin, so this code could be better - but it seems to work very well. :-)

View on HN

quick github tip: if you click the line number on the left you can get a link directly to the line you are referring to.

extra tip: after clicking the line, press 'y' on your keyboard and you'll get a link to the file in it's state at the current commit so future commits won't break your old hyperlinks.

cool, thanks

Whatever works for you, but changing 20 lines of code in this to use asyncio will send your throughput through the roof.

The limiting factor for a fast TCP scanner is not your favorite async toolkit.

Well, no, of course not. Don't be stupid. The default ThreadPoolExecutor size is the number of cores you have on your laptop. In my case, 8. So that's 8 concurrent connections at once. That's, frankly, shit.

Use asyncio (or roll your own toolkit, or just spawn 'nmap') and you get 20000 concurrent connections.

Which is better?

You're conflating concurrency with parallelism.

Your 20000 concurrent connections with asyncio are not parallel. You can have 20000 threads if you want.

Are you sure about that? Have you ever tried starting 20,000 threads? Try it.

The connections are actually handled by the OS in parallel, it's the callbacks to Python that are not. Why do you need a whole OS thread to handle sending a SYN/ACK?

As you well know 20000 threads is not a great way to do anything. Especially in Python.

I agree that threads in Python are not great but 20000 threads is otherwise not an issue.

Are you serious? Have you ever tried starting 20k threads?

Not using something to deserialize all that blocking I/O waiting on the internet is THE limiting factor.

Reply on news.ycombinator.com