Stress Testing HTTP with Twisted Python and Treq

blog.mailgun.net

9 points·old-gregg··3 comments

Speaking on using this with multiprocessing. AFAIK Twisted does not work well with multiprocessing – the details are not on top of my head.

So the simple way to make this work on a multi-core machine is to wrap the whole thing (including imports) into a function and spawn a pool of processes BEFORE you import Twisted, initialize the reactor, etc.

As long as you mean the multiprocessing module, that's correct. Alternatively, you can use the reactor's spawnProcess method for creating processes, or Ampoule if you want a process pool (which is nice because it works remote, too). Perspective Broker and it's object-capability successor, Foolscap, are more full-blown distributed computing tools, but they would work fine for this as well if distributed requests is what you want :)

Of course, all of this mumbo jumbo is probably only required if you can't manage to saturate that network link with a single core :)

In case anyone was wondering, this is wrapped in a cooperator for a reason. At every yield, the reactor is given a chance to act. That's important, because otherwise the reporting function would only ever run after you've made all the requests, which is probably not quite what you want :)

Also, you might want to replace the automatic rescheduling with twisted.internet.task.LoopingCall.

Reply on news.ycombinator.com