lekker

They are a little verbose, though it is likely not to cause problems here. I suppose that it may slow things down a very tiny bit. The author could possibly have been trying to future-proof it with a bit more verbosity in the face of allowing for other keys (execute-mode, execute-permissions) etc.

GPU hardware on Amazon is quite pricey. I would be careful since it can get out of hand rather quickly.

Oh, that looks really interesting. I'll definitely give this a bash

It had 512MB of RAM, which I realise is a bit rediculous to run anything like this. I don't have a whole lot of budget for R&D so I try to keep things cheaper where possible.

All right, thank you. I'll look into the other options you mentioned

Currently we have a Postgres database and run manual queries against it. One-off hackjob turned production.

Okay, thank you. I've been looking into Rabbit quite a bit. I like the stream processing aspect of Kafka, where you can introspectively peek into things as they happen rather than a straight message queue.

Some of our stuff is on the GCP, so have also been investigating Pub/Sub and DataFlow, which seem more viable since we don't have the up-front cost of hardware and maintenance but pay for resources used. If we don't use much, we should not pay much.

All right, that makes sense. It is likely that we'll see a peak in the 10s of thousands at a time, if that. Probably more as we move into the future.

I want to avoid deploying a piece of software now that we'll need to swap out in a year since it does not provide what we need anymore, though I suppose that is a natural part of scaling.

What would you recommend as an alternative to Kafka that does adequate real-time processing? The real-time-ness of it is important for our use-case. Maybe even some kind of time-series database system could work.

I've been eyeing Kafka for a long time now - tried to set it up on a smaller DO box and had memory issues. Is it worth using this for smaller operations (<100k requests per day)? The stream processing seems extremely useful for my use case. Is there something else that may be a better fit?