Hacker News Reader

Comment by thelastparadise | Hacker News Reader

Have been having intermittent issues the past couple days. It's times like these I'm happy to have llama running in k8s on GPUs (https://microk8s.io/docs/addon-gpu https://hub.docker.com/r/pytorch/pytorch).

By the way, has anyone noticed they're starting to have an "AI-dependence"?

These AI tools are infiltrating my workflow, and it's becoming more noticeable when they go down...

View on HN

I have an AI-dependence in the same way that I had a Stack Overflow dependence, and before that a Google dependence, and before that a reference manual dependence, and with that a (book) library dependence.

> It's times like these I'm happy to have llama running in k8s on GPUs

I'd love to know more about this. Do you have your own hardware, or do you run it in the cloud. Did you set everything up yourself. I'm somewhat intimidated by k8s, let alone the other stuff on top, but very curious.

> has anyone noticed they're starting to have an "AI-dependence"?

Yup +1

> Do you have your own hardware, or do you run it in the cloud

Running two AI-based startups. We use GPU compute from "everywhere possible" including cloud and on-prem. Cloud is a combo of auto-scaling clusters with spot instances (GKE, EKS, etc.).

Have done some proofs of concept on serverless (runpod, modal), but they are very unpredictable in terms of latency and scalability.

How do we make it all work? K8s is the magic. The auto-scaling cloud clusters and the local cloud clusters all just look like k8s endpoints.

The configs are deployed with basic k8s yaml + jinja2, but helm is an option (we're trying to keep it simple.)

What are you using for K8s autoscaling? We initially tried a few standard K8s scaling mechanisms and found that they didn't work well for GPU workloads. For example, if we were serving a low-RAM Huggingface model on GPU, it wouldn't trigger autoscaling. But since the GPU can only process one request at a time, the system would get bottlenecked while it waited to process requests one-by-one.

We wrote a bit about this here, if anyone is interested: https://www.beam.cloud/blog/serverless-autoscaling

If you aren’t fully utilizing gpu memory, can you use time slicing? Or MIG for A100?

Orthogonal solution but could help.

Sharing GPUs only really makes sense for GPUs that are large enough to share. MIGs can work for 80Gi A100s but won't work with smaller cards like T4s. It also adds latency to the GPU operations. Unfortunately there's not yet a silver bullet for this stuff.

That’s why I was curious about utilization since you mentioned low memory usage. I believe time slicing can work on those smaller cards these days. Did you explore any other optimizations like batching or concurrency for same model?

Model heterogeneity seems like a real challenge there — you could optimize usage if you know all the sizes ahead of times and actually have gpu capacity to do efficient allocations, but it’s way harder than just doling 1 gpu per pod.

e: also, latency because of reduced resources? Or what do you mean?

yeah I have been using it so much that I definitely know when it's down.

Reply on news.ycombinator.com