80 karma · joined January 20, 2024
Honestly, this setup will keep us going for a long time. There are some design considerations that you need to keep in mind while structuring your Python code. For example, on a gp3 EBS volume, our mean single-write latency IN SQLite is about 100 microseconds, so in theory, if we can keep transactions short, the scale we can achieve on just SQLite is massive. We'll also get serialized isolation on writes and snapshot isolation on reads for "free." However, Django/Python are not super quick, so if we decorate all our Python functions that make writes with `@transaction.atomic` we end up with 10+ MILLIsecond locks on the DB, massively reducing throughput.
This is a price to pay for using SQLite, but honestly, even if we end up having to move to a proper RDBMS later, our database will probably be happier we wrote our code with transaction length in mind.
SQLite actually performs very poorly on EFS in my experience (likely due to the fact that acquiring file locks is over the network).
We're using ECS to manage our containers. What worked best for us is two task definitions, one daemon with only Litestream, another replica task with our web app. The web app task has a placement rule ensuring that Litestream has started before it can be placed, ensuring the database is "in place" before the app starts. Both containers have the same directory on the host mounted as volumes, so they can both access the same database. This series of tests convinced me that this is a safe setup: https://tangentsoft.com/sqlite/doc/trunk/walbanger/README.md
This has been working well for us - the db is fast, we can deploy whenever we want, and I don't feel anxious about the host going down.
One remaining source of anxiety is updating the machine image of the host - that will cause some downtime, of course. I think the safest strategy will be to increase the size of the ASG the host is in to warm up a second, updated EC2, then drain the first one.