Google Cloud Functions for the win. Very reasonable pricing model too. Doing 55 requests per second, 24/7, and it is about $100 a month, including the managed Cloud SQL postgres instance.
just to clarify our setup: - 1 pubsub 'job' queue - 1 cloud function triggered by a scheduled event populates the job queue - 1 idempotent cloud function to handle a job, triggered by events on the queue.
If you have a big >1h job you have to add locks, make sure deploys don't interrupt the job, handle retries of the whole job, maintain serverless + not serverless, and then inevitably rewrite the whole thing when it takes too long to be viable. All in all a lot of work and complexity as well that is wasted on making a bad design work.
It is better to design things from the start to cut things up into smaller units and parallelize as much as possible. By doing that, you solve the problem I mention... as well as the problem you mention. Two birds.
AWS < GCP
AWS supports more languages, faster deployments, faster execution, cheaper, let’s you roll your own image.