Ok. This is what has been a huge hangup for me.
It really seems that if you're doing asyncio, you must do EVERYTHING async, it's like asyncio takes over (infects?) the entire program.
It really seems that if you're doing asyncio, you must do EVERYTHING async, it's like asyncio takes over (infects?) the entire program.
There are bridges from async to sync and vice versa, but they must be used very sparingly, as they aren't nestable.
You could easily have, say, a thread to do all your asyncio stuff, another to do some CPU intenstive stuff (so long as it blocks the GIL) and yet another to do some blocking I/O e.g. interacting with a database with its own blocking APIs. In that case, you could spawn separate threads and exchange messages between them like usual.
Where this falls down a little is if you want asyncio to infer your whole program - or more accurately, if you want to use it in all of the task-management code at the top level of your app. In that case, you would use asyncio's thread API [1] although it's a bit clunky for when you want to maintain state bound to a specific worker thread (like SQLite database objects).
[1] https://docs.python.org/3/library/asyncio-task.html#running-...
1. You jump into a huge codebase and find a very useful function you want to use in your code (it uses asyncio)
2. Your code doesn't use asyncio, so you start converting functions to be async (or else you can't use the `await` keyword)
3. It turns out your function is called by a bunch of different users, some of which do not use the asyncio runner
4. You end up having several meetings with upstream teams asking them to use the asyncio runner
5. A single team says "no"
6. Scrap the project and just rewrite that original function you found synchronously
What you've written is true, but "infects" a smaller proportion of the code than you'd expect, in my experience. In fact, the better organised the program is, the less needs to change when converting to async. For example, reading from a connection your top-level async function would often be something like this:
async def read_and_handle_messages(connection):
while True:
next_message_bytes = await connection.get_next_message()
next_message_parsed = my_parser.parse(next_message_bytes)
my_message_handler(next_message_parsed)
I find that there's very little actual async code in async programs that I write, even with no blocking IO historical baggage. Admittedly, that's partly because I organise programs to reach that goal, but it actually works out for the best because all the async-task lifecycle management ends up in one place, which makes following overall program flow particularly easy.
The problem is that the event loop is not reentrant [1], so if your sync function is being called from an async function, things do not work. Why would you ever do this you might think? Well, asyncio might be an implementation detail of whatever environment you are using. Jupyiter or ipython for example.
[1] there are... hacks to make this sort of work of course.
The python motto.
This works well if the callback is just there to notify you about a result (which is the most common case in fairness) but doesn't really work if the callback is meant to synchronously determine and return a result that is then used within the function that's calling the callback.
But it feels wrong having to spawn a thread and event loop just for that.
Modern frameworks should be orthogonal and compositional. Even orthogonality (when you use multiple frameworks, each framework solves a problem along a different "axis" and does not interact with any of the other axes) is optional.
Compositionality means I should be able to combine several systems without one affecting the other unnecessarily. async does not fulfill that requirement.
JavaScript has an extremely nice "out" (I'm sure other languages do too):
async function() {
await async_thing()
do_sync_stuff()
}
function() {
async_thing().then(do_sync_stuff)
}
(If you really, really need to do this - triple check - use queues & worker threads to decouple the sync parts and keep them out of your event loop thread, as a sort of sync-async impedence match.)