- treat steps of the process as functions that return data, instead of jobs that return at most success/fail, then provide me with ways to harness, manipulate, track, and display that data. (And a documented structure by which I can define what data I choose to return from a given step; I'm thinking mostly about two scenarios, the first being metrics about a build, and the second being structured information about errors/stacktraces)
- provide me a way to use your tool locally during development, in place of (or wrapped around) the standard build tool for my language, and a guarantee that (within reason) it will give the same results on my machine as it does when running an automated job managed by the server. Give me clean, detailed ways to inspect the difference in situations where the behavior differs between my machine and the server. This would in theory help me to reduce the incidence of "works on my machine" sorts of errors.
- Give me a a well-documented http/s api, and a way to add arbitrary endpoints to that api, which trigger arbitrary scripts/jobs (or just query arbitrary subsets of data from my first point above). I should be able to decide whether my new endpoint is specific to some arbitrary grouping defined for my org (and thus only visible to members of that grouping) or available to anybody who can reach the endpoint. (I'm thinking of teams practicing radical transparency, who might want to, say, provide a public endpoint that anybody can hit that provides various build metrics; or teams who want to provide their product owners or business stakeholders a controlled window into parts of the process, without having to build their own services to provide the api for that window)
- give me an easy way to wrap a debugger around a given job or step, locally or remotely, and a clean interface for stepping through it and (if appropriate) walking the stack.
- save a run as an artifact, so that it can be marked, saved, exported, reviewed, downloaded, and stepped through safely at a later date and on a different machine. I should be able to see responses that we got from remote resources without having to touch those resources when reviewing a job saved in this manner.