I'd love to know more about this. Do you have your own hardware, or do you run it in the cloud. Did you set everything up yourself. I'm somewhat intimidated by k8s, let alone the other stuff on top, but very curious.
> has anyone noticed they're starting to have an "AI-dependence"?
Yup +1
Running two AI-based startups. We use GPU compute from "everywhere possible" including cloud and on-prem. Cloud is a combo of auto-scaling clusters with spot instances (GKE, EKS, etc.).
Have done some proofs of concept on serverless (runpod, modal), but they are very unpredictable in terms of latency and scalability.
How do we make it all work? K8s is the magic. The auto-scaling cloud clusters and the local cloud clusters all just look like k8s endpoints.
The configs are deployed with basic k8s yaml + jinja2, but helm is an option (we're trying to keep it simple.)
We wrote a bit about this here, if anyone is interested: https://www.beam.cloud/blog/serverless-autoscaling
Orthogonal solution but could help.
Model heterogeneity seems like a real challenge there — you could optimize usage if you know all the sizes ahead of times and actually have gpu capacity to do efficient allocations, but it’s way harder than just doling 1 gpu per pod.
e: also, latency because of reduced resources? Or what do you mean?