Hacker News Reader

For my hobby project, I:

1/ bought a gaming rig off craigslist 2/ set it up to run as a server (auto-start services, auto-start if power is cut, etc.) 3/ setup a cloud redis server 4/ gaming rig fetches tasks from redis queue. processes them. and updates them.

Queueing in front of ML models is important. GPUs are easily overwhelmed and will basically fail 90% of traffic and have max latency if you send it too much. Let the GPU server run at its own pace.

View on HN

> Queueing in front of ML models is important

This sounds so clean! A super fast NoSQL to hold incoming requests.

Could even throttle your free users and prioritize the paying customers.

Btw, is the project online now?

Here is a queueing api server for self hosted inference backends: https://github.com/aime-team/aime-api-server from a friend of mine. Very light weight and easy to use. You can even serve models from Jupyter Notebooks with it without needing to worry about overwhelming the server. It just gets slower the more load you send to it.

Really cool! I like that they have live demos to prove it out. Thanks for sharing

I use ML for content moderation at Grab and my side project. Both are online. Both use queues, but Grab's traffic is more organic and real time than the side project's.

Could you please elaborate on the OS/tools you’re using to run the model locally?

Reply on news.ycombinator.com