I just rewrote some part of my system to kubernetes job. Gist is here: https://gist.github.com/kozikow/86ee26998c5de1cf3780b6bc0cd3... .

- As mentioned previously, retry count limit to job/pod would be useful.

- Minor nit: Deleting old job with the pod is cumbersome with current python API. Either delete_namespaced_job with orphan_dependencies: False, but I would need to check if job exists beforehand (deleting if job does not exist fails). On the other hand delete_collection_namespaced_job lets me delete job by label selector, but I also need to manually clean up the pod.

- Another nit: When exporting stackdriver logs collected by GKE from a job to big query, each log ends up as a separate logsource.

- When I used them last time, there was no python api. New https://github.com/kubernetes-incubator/client-python is a big improvement. Http API required sidecar proxy container.

- Retry control is course grained. IIUC the only option are "no retry at all" or "terminate if job have been active for X seconds". If my jobs tend to be long and failure happens early, second option can lead to hundreds of retries. Each job retry is expensive, as it creates a new pod. I remember ending up with 100s pods.

Not the OP, but I'll chime in with what we're missing when we took a look at it. AFAICT there is no way to tell k8s to retry a job X number of times; you have to futz around with deadlines.