My first job I was in ~10 people robotics/hardware startup, and we had CI cluster of about 4 workers, later extended to ~10 machines (managed by buildbot, those were the times...). And the parallel jobs happen all the time in commercial developments, no one likes to wait for a long time when they could wait for a short time. This is not done for various OS'es, instead one machine might be testing service A, while other tests service B, while the third one tests integration.
That said, there is nothing wrong with "Dead Simple" niche - I am sure that there are some users who could use it. Although those would probably want something with less lock-in, your current approach is pretty weird: the jobs which _look_ like they are written in regular bash/python/etc.. but actually they have extra functions ("run_task", "task_var" etc..) magically injected, so you cannot run the file directly nor can you run linters on those. I'd much prefer pretty much anything else which can run unmodified code.