For inference, we extended KServe (previously KFServing from Kubeflow) to fit our on-prem cluster needs. Highly recommended!