Hacker News Reader

Comment by ak217 | Hacker News Reader

What do folks use for event-driven scatter-gather job control using Lambdas only? For example, imagine I launch 10K Lambda jobs from an orchestrator Lambda. I can't guarantee they'll finish in 300s, so I can't just poll for completion. I could wake up a gather Lambda as a scheduled event, but the schedule granularity is too coarse. Waking up a gather lambda every time a job completes is wasteful and racy.

I would really like AWS to have an event that fires when all the lambdas in a set have completed, but that doesn't seem to exist. Any suggestions?

View on HN

What we do is a separate Python process, not in Lambda, that polls status. Status is persisted in Redis. It supports lambdas calling other lambdas so dynamic workflows just work and can live for longer than 300s. This has been working fine so far but would probably have issues at 10k+.

There's an SQS/Lamba integration. Push 10k messages in a queue as jobs. The lambdas pull them off and delete them once done.

https://cloudonaut.io/integrate-sqs-and-lambda-serverless-ar...

From what I can tell, this doesn't really help you know when your queued up lambdas have all completed, which is what GP wants.

You're right. I assumed the queue depth api call would be useable as a counter. It appears though, that it's only an approximate count unless you use FIFO queues.

I guess the individual Lambdas could pop an "I'm done" message onto a FIFO queue. With a single Lambda consumer of that queue that updates an S3 bucket with a count. Done when count == 10,000. That's atomic, and no polling, but feels a bit smelly.

Have you tried Step Functions? I've used them to orchestrate Lambdas with good success when I needed something that will run longer than Lambdas can individually.

Yeah, they’re good, but more expensive than you’d expect given the relative pricing compared to other AWS services. $0.025/1000 transitions seems and order of magnitude or two high given it’s just ultimately a central job store with some very, very simple logic for changing state.

Step Functions are convenient but for the reason you stated we stopped using them. Pricing seems off for what they do, especially when it's in volume.

I wrote a simple lambda function that takes in a payload and a step function arn. It then fires up a bunch of step function jobs and maps the execution identifier to the input and stores that in s3 for later access. This ends up being really nice for creating distributed cron like jobs by just connecting the lambda to cloudwatch events.

Reply on news.ycombinator.com