Hacker News Reader

Comment by robbiep | Hacker News Reader

Point taken.

Yes, it is easy to understand (and even more than that - yes it can be in its own module/blueprint anyway). My previous thinking has been that if I have a message queue and separate application then I get the benefits of failure reporting through the queue as opposed to potentially missing something if I get a failure through async. Although I guess they aren't insurmountable.

Then, inevitably, you have growth so that one email function it may begin with, but later you're doing other fancy things and even though you may just want to send a single email at the start, later you're making database calls and doing other bits and pieces to make the template more fancy and personalised etc. So, better to start the architecture from the start in a slightly more complicated fashion..?

View on HN

> start in a slightly more complicated fashion

No. Start with "bad" architecture and refactor as needed.

Another layer of abstraction solves any problem. If you don't have a problem yet, then abstraction is unnecessary complexity.

BTW, writing your own async code may actually be more complicated than just having a message queue and an email program. It depends on how much trouble your message queue is giving you. Can you use a Sqlite table instead of a "real" message queue?

I'd have the web-server append "messages" to a sqlite table and have the emailer read from the same table. You can use cron to run the emailer every X minutes. If the table gets huge, use cron to run an archiver that prunes old messages during low-usage hours. Sure, the pruning will temporarily lock the database. If that gets to be an issue, switch from SQLite to PostgreSQL. If that architecture starts to fail, that's a good problem to have.

You could argue that writing to the queue should be done asynchronously inside the web-server, but, again, wait until your customers complain that the server isn't responding quickly enough.

Great, thank-you. Self-teaching the finer points of software architecture is something I find difficult

In some circles, this idea is known as "incremental architecture". The idea is that you should architect and add abstraction layers to solve problems you have /today/, rather than solving problems in the future.

One big reason for this is because we're often wrong about what problems we'll have in the future, or in what way they will manifest. Moreover, often the "bad" solution will work for far longer than we expect, and the day that we're anticipating where we'll be having problem X might never even come.

What's worse is that in the meantime each layer of abstraction adds a tax to how easy it is to fully understand and work with the code, to change things and add new things.

What you do want to do is to write code that is /open to/ being abstracted. E.g. keep things structured nicely, start with a basic function (named similarly) or a few related functions before putting in a full blown class, name things well so it's easy to find them while refactoring, etc.

Thanks.

I've done this with iteration 2 (current version) or my backend but now I'm not a one man shop and have a team (well one other junior backend and a senior mobile). And we are architecting for some pretty major new features so it's interesting trying to strike a balance with both what I've learned along the way and what it seems 'best practice' is when it comes to python

Careful with where you read "best practice" from. Some folks like to brag about their new high-scalability architecture, but it's way over-engineered for most software.

The pattern I described of using SQLite tables to mimic a message queue will be very easy to switch to PostgreSQL or MySQL. If you hit serious scale you can switch to something like Kafka or Kinesis. That'll be a bit more involved, but hopefully your revenue and team scaled to match the compute needs. When you switch to Kafka, you can use Spark or Storm instead of cron. If you switch to Kinesis, you might choose AWS Lambda. So you're not writing yourself into a corner.

For an instant latency boost, you can also pop the sqlite db directory on to a memory tmpfs.

I've always wondered about this: doesn't the OS read buffer pretty much keep the whole thing in memory anyway, if you have memory to spare? Does it make that much of a difference?

It's not the reads but the writes that'll get you. Reads are easy to scale.

Reply on news.ycombinator.com