Hacker News Reader

Comment by paddy_m | Hacker News Reader

I would be really interested in a distributed job running system with a scheduler. I want a UI where I can set jobs via a schedule that run on one of many workers. In that same UI I want to be able to look at failed jobs and see all of the output from that job. I don't want to have to ssh over to the box where the job ran to retrieve a log file. I once had a job where we used Tidal Enterprise Scheduler which poorly implemented this. What solutions exist for this workflow in the open source world?

View on HN

Look into airflow: https://airflow.apache.org/

Not quite distributed, but https://www.rundeck.com/open-source is pretty nifty.

Rundeck, as others have indicated, will do just that.

I've used Tidal too, it's utterly horrible and universally hated by anyone I know who has used it. What took months to set up and debug in Tidal, took us hours in Rundeck.

Also ndscheduler: https://github.com/Nextdoor/ndscheduler/

Rundeck does exactly this.

If you already have something like Jenkins you could probably bend that to your will, too.

Would you pay for this? How much and what pricing scheme would you prefer?

As a service that runs on Heroku I'd pay around $25/month. that'd be $25/month * 2 (stage+prod). At the bootstrapping stage i'm at now, that's what I'd pay. If I were profitable, I'd pay up to $500 - $1000 a month and happily so, figure out market segmentation so that I'd be forced to pay that much. I would probably run around 1000 jobs a month, which seems like a number that would put me in a higher pricing range. This service would be super useful for around 60 jobs a month even. Output from the jobs would be tiny, maybe 10k at most. I would be happy if output was saved on private S3 buckets.

I guess the big point is, I'm looking to use this for regularly scheduled jobs, not just as a queuing service to spread load.

My email is in my profile if you want to get in touch.

Reply on news.ycombinator.com