Django Async: What's new and what's next?
deepsource.io
deepsource.io
There is just so many layers of magic in Django, that it was becoming impossible for us to improve the performance to an acceptable level. We isolated the problems to serialization / deserialization. Going from DB -> Python object -> JSON response was taking far more time than anything else, and just moving over to FastAPI has gotten us a ~5x improvement in response time.
I am excited to see where Django async goes though. Its something I had been looking forward to for a while now.
-- Boring code - Business logic, CRUD, management, security, ...: django
-- Perf: JWT services on another stack (GPU, Arrow streaming, ...)
So stuff is either Boring Code or Performance Code. Async is great b/c now Boring Code can now simply await Performance Code :) Boring Code gets predictability & general ecosystem, and Performance Code does wilder stuff where we don't worry about non-perf ecosystem stuff, just perf ecosystem oddballs. We've been systematically dropping node from our backend, where we tried to have it all, and IMO too much lift for most teams.
Have any interest in expanding this into a blog post? I've been working on a similar post. Maybe we can compare notes. I'm at michael at testdriven dot io, if interested.
For the DB requests Are you writing sql directly, using a different ORM, or something like sqlalchemy core that makes sql pythonic without being an ORM?
And it uses SQLAlchemy under the hood. Can use all of it. But if you want full async all the way down, can just use core and something like encode/databases for the DB access.
You can also skip the object serialization from the ORM and work with python dicts directly to significantly improve serialization performance from the database.
I have also been experimenting with encode/databases for async DB access. It still uses the SA core functions, which is nice, but that means it does not do the nice relationships stuff that SA has built in when using it to handle everything. At least not that I have found. However it does allow for things like gets without relationships, updates of single records, and stuff like that quite nicely.
https://fastapi.tiangolo.com/tutorial/sql-databases/#create-...
I am using asyncpg[1] (much more performant and provides close mapping to PostgreSQL, making it much easier to use its advanced features) through raw SQL statements without problems.
This is somewhat inaccurate. They use SQLAlchemy in the tutorial, but FastAPI is in no way tied to SQLAlchemy.
Edit: Although, now that I think a little more its not that surprising. Our initial tests did literally just define FastAPI schemas on top of our existing DB. The co-mingling while actually running is an interesting concept though.
I’d define the class, add it as the return value, I was manually instantiating the class and returning that, but for it didn’t like that and would constantly throw errors about it. I think it was PyDantic which was the root cause there.
The Depends functionality refused to inject my classes as well, but I was probably doing something wrong there...
Dropping back to Starlette was good because it gave me everything I needed and got out of my way. I’ve still got everything fully typed and passing MyPy.
If your views involve heavy-lifting calculations or long-running network calls to be done as part of the request path, it’s a great use case for using async views.
When doing a long computation the CPU is not idle so there is no free compute power to use for something else.
Finally, when doing IO calls in Python so GIL is usually released so the kernel can already schedule another thread while waiting for IO, so it is not sure that converting to async will yield improvement and should be benchmarked if you plan on converting an existing program.
This is true, but scheduling another thread through the kernel can have higher overhead since it requires context switches. Running multiple threads also has other potential issues with lock contention; how problematic they are will depend on the use case.
The potential advantage of scheduling another thread is, of course, that it can do CPU bound work; but in Python, unfortunately, doing that means the GIL doesn't get released so that thread will prevent any further network I/O while it's running, the same as would happen in an async framework if a worker did a lot of CPU work. So Python doesn't really let you realize the advantages of threads in this context.
I don't think that's true, the GIL get released for many computing intensive or IO bound tasks in Python, for example when reading from a socket the GIL gets released at https://github.com/python/cpython/blob/e822e37946f27c09953bb...
I/O bound, yes, since that requires system calls, and system calls (reading from a socket is an example of a system call) release the GIL.
Computing intensive, no. Code that is doing a CPU intensive computation but makes no system calls will never release the GIL.
https://docs.python.org/3/c-api/init.html#releasing-the-gil-...
Yes, you're right, my statement was too broad.
Any code that does not involve Python objects can release the GIL, no matter whether it makes system call or not.
For example, NumPy the most popular scientific computation package in Python, on which many other popular packages like Pandas are based, releases the GIL when doing operation on matrix. This is documented at https://numpy.org/doc/stable/reference/internals.code-explan...:
> If NPY_ALLOW_THREADS is defined during compilation, then as long as no object arrays are involved, the Python Global Interpreter Lock (GIL) is released prior to calling the loops. It is re-acquired if necessary to handle error conditions.
To do so it uses the same macro used by the socket module when doing system calls: https://github.com/numpy/numpy/blob/18a6e3e505ee416ddfc617f3...
And does not involve running Python bytecode. Yes, numpy and other packages that provide C extensions do this when they are doing computations that don't require running Python bytecode.
> no matter whether it makes system call or not
Yes, you're right, my statement was too broad.
- https://docs.python.org/3/library/concurrent.futures.html
- https://docs.python.org/3/library/asyncio-eventloop.html#asy...
Using cython:
with nogil:
# whatever you need to do, as long as it
# doesn't touch a python object
compared to FastAPI[1] I really don't want to use it, I only miss the ORM since in FastAPI it looks like you have to manually write the code to insert stuff[2].
[0] https://realpython.com/getting-started-with-django-channels/
[1] https://fastapi.tiangolo.com/advanced/websockets/#create-a-w...
[2] https://fastapi.tiangolo.com/tutorial/sql-databases/#create-...
Django is great for CRUD apps, MVPs and such. And I've used it with success for larger platforms, but it doesn't take long for me to want something closer to the metal whenever I need custom work. FastAPI has filled that need wonderfully well.
I also miss the ORM though… SQLAlchemy is a pain.
FWIW I'd have just used Pusher.io and triggered via django signals but I didn't want clients to be able to query the list of channels, or have to think about costs.
This was more of a Friday hack for fun rather than a core product requirement for profit. But I feel like using a separately deployed and scaled service, agnostic of business logic, was the right move.
For most things you're probably better off "doing the work" in a celery task, regardless if it is IO bound or CPU bound. Then use web sockets just for your status updates/progress bar, instead of having your front end poll on a timer.
I'm consistently surprised that there are not awesome web frameworks in JavaScript similar to Django
Also, I still haven’t seen how the async addition will work with Class Based Views. Also, Django Rest Framework is still considering spending time for support. Until these two use cases are viable many users won’t benefit.
Nooo, not at all. Your tasks should be I/O bound, not CPU bound to take advantage of asyncio. Maybe the async server using multiple threads with multiple event loops, but don't ever do a CPU-heavy task in an event loop because you just invalidated using asyncio completely.
reference: http://calpaterson.com/async-python-is-not-faster.html
It's been discussed before but the outcome of that discussion (in the link you brought up) was divided. Highly highly divided. There was no conclusion and it is not clear whether the benchmark was flawed.
The discussion is also littered with people who don't understand why async is fast for only certain types of things and slow for others. It's also littered with assumptions that the test focused on compute rather than IO which is very very evidently not the case.
the point is that it's not meant to be any faster than a parallel pool of processes that perform the same heavy IO without blocking all requesting clients. asyncio is about packing as many concurrent socket interactions into a single process as possible, hence optimising for throughput by giving up the speed that gets eaten up by task context-switching. Hence the flaw in the becnhmark. The benchmark was run on the same machine where Postgres was operating. The benchmark used different number of processes for sync and async workloads, the connection pool was not setup to prevent blocking when a coroutine tries to acquire a connection from the pool when the pool is exhausted (for benchmark purposes it should not have upper bound limit and to be pre-populated with already established connections).
Wrong. Workers amounts are the same. See the chart with benchmark results. http://calpaterson.com/async-python-is-not-faster.html
>The benchmark was run on the same machine where Postgres was operating.
This wouldn't affect the variance between sync and async results very much because both frameworks were run on the same machine.
>the connection pool was not setup to prevent blocking when a coroutine tries to acquire a connection from the pool when the pool is exhausted.
Real world connection pools have an upper bound limit. I don't see why setting an upper bound limit to be closer to reality is not a good test.
Also you're completely wrong about the connection pool blocking when it is exhausted. See source code:
https://github.com/calpaterson/python-web-perf/blob/master/a...
If all connections are exhausted then the system still yields to compute and incoming requests.
>(for benchmark purposes it should not have upper bound limit and to be pre-populated with already established connections).
Disagree. The real world sets an upper bound. There's nothing wrong with simulating this in a test.
I see that aiohttp has 5, uwsgi has 16, and gunicorn has 12, 14, and 16 depending on a web-framework, is this your definition of the same?
The author says:
> The rule I used for deciding on what the optimal number of worker processes was is simple: for each framework I started at a single worker and increased the worker count successively until performance got worse.
That's not how benchmark is supposed to be conducted, one doesn't fit workers number to the result that one finds "optimal", one should use the same amount of workers and find bottlenecks and either eliminate them or explain why they cannot be eliminated without affecting benchmark invariants.
> this wouldn't affect the variance between sync and async results very much because both frameworks were run on the same machine.
it will affect the variance. Firstly, because the db will spawn processes on the same machine, pgbouncer will spawn processes on the same machine, they all will compete for the same CPU and the order of preemptive context switches affects individual benchmark runs differently. On top of that, there are periodic and expensive WAL checkpoints, and fsync that competes with a benchmark for the kernel system call interruptions and context switches, and the multi-process workers setup may be affected dramatically. If you don't believe that external processes affect the numbers to the extent they become incomparable, try to serf the Internet with your web-browser randomly while running a benchmark.
> Real world connection pools have an upper bound limit. I don't see why setting an upper bound limit to be closer to reality is not a good test.
Because benchmarks are not real-world workloads, they are designed to show unbound performance of the implementation detail that is selected for a test, where external resources are non-exhaustable for the purpose of avoiding side-effects external to the functionality that is being tested.
> Also you're completely wrong about the connection pool blocking when it is exhausted. See source code: > If all connections are exhausted then the system still yields to compute and incoming requests.
I didn't say that it wouldn't yield, I said that the coroutine will be blocked at the point where it tries to acquire a non-existing connection from the pool, which affects the benchmark. Now, instead of one blocking context switch at a network socket call that queries Postgres, the coroutine will yield AND WAIT twice - at the exhausted connection pool, and at the network socket call after the connection is acquired. This is exactly the reason why resources should be unbound, and why DB should be on a separate machine (unbound spawning of connection processes upon request), and why the number of OS workers should be the same in all benchmarks, because the sync version will also block twice, and the consequence of blocking there will be much more dramatic and different than in the case of async, WHICH IS THE POINT of a proper benchmark - https://github.com/calpaterson/python-web-perf/blob/master/s...
The real issue, as the article notes, is latency variation. Because async frameworks rely on cooperative multitasking, there is no way for the event loop to preempt a worker that is taking too long in order to maintain reasonable latency for other requests.
There is one thing I wonder about with this article, though. The article says each worker is making a database query. How is that being done? If it's being done over a network, that worker should yield back to the event loop while it's waiting for the network I/O to complete. If it's being done via a database on the local machine, and the communication with that database is not being done by something like Unix sockets, but by direct calls into a database library, then that's obviously going to cause latency problems because the worker can't yield during the database call. The obvious way to fix that is to have the local database server exposed via socket instead of direct library calls.
No it's not pure python. It's a combination. The underlying event loop uses libuv, a C library that's makes up the underlying core of nodejs. The marker of "Uvicorn" is an indicator of this as "Uvicorn" uses uvlib.
Overall the benchmark is testing a bit of both. The event loop runs on C but it has to execute a bit of python code when handling the request.
>If it's being done via a database on the local machine, and the communication with that database is not being done by something like Unix sockets, but by direct calls into a database library, then that's obviously going to cause latency problems because the worker can't yield during the database call.
I am almost positive it is being done with some form of non blocking sockets. The only other way to do this without sockets is to write to file and read from file.
There is no "direct library calls" as the database server exists as a separate process to the server process. Here's what occurs:
1. Server makes a socket connection to database.
2. Server sends a request to database
3. database receives request, reads from database file.
4. database sends information back to server.
Database libraries in Python that support this (as opposed to blocking, synchronous sockets, which are of course common) are pretty thin on the ground. That's why I would have liked to see more details in the article about exactly how the benchmark was doing the database queries.
> There is no "direct library calls" as the database server exists as a separate process to the server process.
Yes, you're right, I wasn't being very clear. The key question is, as above, whether nonblocking sockets are being used or not.
https://www.postgresql.org/docs/12/libpq-async.html
If you're compute-bound, then Python async (which uses cooperative scheduling similar to green threads) isn't going to help you. You get parallelism, but not concurrency from this progamming model; only one logical thread of execution is running on the CPU at a time (per-process), so this can only slow you down if you are CPU-constrained.
The standard usecase of a sync API backed by a local DB with low request latency is typically going to be compute-bound.
This is covered in the Django async docs (https://docs.djangoproject.com/en/3.1/topics/async/) and also in green threading libraries like gevent (http://www.gevent.org/intro.html#cooperative-multitasking).
The case where async workers are interesting is for I/O-bound workloads. Say you're building an API gateway, or your monolithic API starts to need to call out to other API services, particularly external ones like Google Maps API. In this case, the worst-case result is that the proxied HTTP request times out; this could block your Django API's work thread for many seconds.
In the async / green-threaded model, this case is fine; you have a green thread/async function call per request, and if that gthread is blocked on an upstream I/O operation, the event loop will just start working on a different API call until the OS gives a response on the network socket.
Essentially, there's no reason to use Django async if you're doing a traditional monolithic DB-backed application. It's going to give you benefits in usecases where the standard sync model struggles.
(Note, there's an argument that you might want green threads even in a normal monolith, to guard against cases like "developer accidentally wrote a chunky DB query that takes 60 seconds to run for some inputs", but most DB engines don't support one-DB-connection-per-HTTP-connection. There was a bunch of discussion on this topic a few years ago, with the SQLAlchemy author arguing that async is not useful for DB connections: https://techspot.zzzeek.org/2015/02/15/asynchronous-python-a... although asyncio support was added: https://docs.sqlalchemy.org/en/14/orm/extensions/asyncio.htm...)
> This article doesn't evaluate the case that you actually want ASGI for, so I don't think it's very useful.
> The standard usecase of a sync API backed by a local DB with low request latency is typically going to be compute-bound.
(Note, I'm specifically talking about Django here)
> Essentially, there's no reason to use Django async if you're doing a traditional monolithic DB-backed application. It's going to give you benefits in usecases where the standard sync model struggles.
My claim is that these benchmarks are not looking at the use-case that Django async is intended to solve. It's not about increasing throughput to your local DB, and so it's not surprising that you don't see an improvement in benchmarks testing that case. Django's async is intended to enable API-gateways and other long-running requests where the upstream latency can be long enough to starve your API worker threads.
Regarding compute-bound vs. I/O-bound, I'm sure YMMV, but my APM tracing for a mature non-trivial production Django API shows that waiting on the DB accounts for about 25% of the total request time across all my endpoints.
Serialization/deserialization takes an embarrassing amount of time in Django, see https://news.ycombinator.com/item?id=24161828 for example. This framework is optimized for developer productivity, not for performance.
> This is covered in the Django async docs
Where is this mentioned in the linked docs? I see four mentions of the ORM in the linked pages, including where it says they are working on async ORM, but I see nothing about performance for the ORM typically being compute bound.
If you heard of Gunicorn, Uvicorn is the version of Gunicorn with libUV, hence the name.
For some web applications, it might actually be faster (in meaningful aggregate volume) to service a complete request on the calling thread rather than deferring to the thread pool periodically throughout execution. I think the break over point between sync and async comes down to how much I/O (database) work is involved in satisfying the request. If each request only hits the database 1-2 times on average incurring a few milliseconds of added latency, making sync all the way down is might be better than with any amount of added context switching. If each request may take 100-1000 milliseconds to complete overall due to various long-running I/O operations, then async is certainly one good approach for maximizing the number of possible concurrent requests.
In most of my applications (C#/.NET Core) I default to async/await for backend service methods, because 9/10 times I am going to the database multiple times for something and I cannot always guarantee that it will return quickly under heavy load. For other items, I explicitly go wide on parallelizable CPU-bound tasks. All of these are handled as a blocking call against a Parallel.ForEach(). Never would a CPU-bound task be explicitly wrapped with async/await, but one may be included as part of a larger asynchronous operation.
This stuff used to confuse the hell out of me, and then I finally wrapped my head around the 2 essential code abstractions: async/await for I/O, Parallel.For() (et. al.) for CPU-bound tasks which have parallelism opportunities. Never try to Task.Run or async/await your way out of something that is CPU-bound and is blocking the flow of execution. Try to leverage asynchrony responsibly when delays >1ms are possible in large concurrent volumes.
How? Afaik PHP is faster than Python in most aspects.