Comment by plq | Hacker News Reader

Comment by plq | Hacker News Reader

Parent Full thread

My point was that green threads sort of emulate OS threads so should not be treated any differently. As Python already has nice library-level support dealing with such thread-based concurrency, there's just nothing to do for the core language team to support this use case.

> 2. Green threads don't fragment the library ecosystem.

They do. That's why you need to monkeypatch everything.

Thing is -- you got two ways to do IO in an async world: Use the async system calls nothing less than the kernel provides or use threads to use blocking system calls and emulate async IO. There's no escaping that reality irrespective of the async paradigm you are using, green threads included.

> 3. Provide a better abstraction without extra code bloat.

From what I understand, your problem has always been the GIL, not Twisted. If your business logic is not better expressed in Twisted, you should not use Twisted, period.

For some of the code I need to deal with, Twisted's callback logic fits perfectly. It makes my code more testable and easier to reason about. So that's what I'm using. For anything else, I just deferToThread and use blocking code just like normal.

This said, I'd still like to emphasize one very important point:

Here's the secret sauce of gevent: https://github.com/python-greenlet/greenlet/blob/master/plat...

A sibling comment to yours explains briefly how Windows folks have given up trying to get green threads to work even with kernel support.

I do realize the average Python programmer couldn't care less about such low level stuff. However those of us who peeked under the hood of gevent and realized how many basic assumptions it violates stays far away from it.

Green threads are the GOTO of cooperative multitasking. In case you want to switch to "structured programming" from using GOTO-based code, you need to switch to the Twisted mindset.

> Green threads are the GOTO of cooperative multitasking.

You know, in GOTO based vs. structured code, one is a mess where nobody can get things correct at the first several tries, where another is a organized piece, built observing programmers limitation.

The same does apply to bare async-io vs. green threads, but you got something missed-up there.

> > 2. Green threads don't fragment the library ecosystem.

They do. That's why you need to monkeypatch everything.

Not in the same way Twisted or async + yields does. Monkeypatching it not done in the library, that's the whole point. It is done in the start phase of the process once. If I get an IRC library which does uses sockets and spawns threads, that could work with Gevent, eventlet or just regular threads.

If I get a Twisted one then they returned deferreds an my main program doesn't know how to handle deferreds. Or alternatively I am using Twisted I have to find libraries which return deferreds. That's what I meant by fragmenting.

> se the async system calls nothing less than the kernel provides or use threads to use blocking system calls and emulate async IO.

It's the other way though? Green threads use async version of socket calls with a select/poll/epoll/kqueue hub (or reactor in Twisted world) but then they provide a blocking synchronous API to the higher level code.

That is usually the sanest abstractions. The only times I've seen callbacks work well is when callbacks are very short, think something like web simple web proxy for example.

In general callbacks in a complex program end up a mess from what I see. inlinedCallbacks or co-routines with yields help there, I've used those. But it is still suboptimal as library ecosystem is still fragmented and code is still cluttered with yield and awaits and so on.

> Green threads are the GOTO of cooperative multitasking. In case you want to switch to "structured programming" from using GOTO-based code, you need to switch to the Twisted mindset.

I think it is the opposite. A callback chain is an ad-hock, poorly implemented and obfuscated model of a blocking concurrency unit. That is a socket event starting a callback chain of cb1->cb2->cb3... is usually much better represented as a set of nicely blocking functions calls fun1->fun2->fun3. Except callbacks are scattered all over. And just because they are callbacks doesn't mean you don't locks and semaphores, you can still have data races between another callback chain started from a different socket which also calls cb1->cb2->cb3 before first one finished.

Also noticed that languages which are used in highly concurrent environments follow the same paradigm, namely Erlang. It is not a sequence of callback but rather isolated blocking concurrency units. Inside each unit calls are blocking but there can be many concurrent (and run in parallel) such concurrency units. Go does the same.

Reply on news.ycombinator.com