I'd love to hear more about using Buildkite. It appears to to offer greatly flexible agents with out a heavy cluster to manage or poorly written plugins to deal with...
Another similar option we tried is AWS CodeBuild which has per-minute billing and provisions the build machines for you. However, it's very bare-bones and because you always get a from-scratch instance for each build you have to distribute your docker cache which is not ideal.