Hacker News Reader

> Queueing in front of ML models is important

This sounds so clean! A super fast NoSQL to hold incoming requests.

Could even throttle your free users and prioritize the paying customers.

Btw, is the project online now?

Here is a queueing api server for self hosted inference backends: https://github.com/aime-team/aime-api-server from a friend of mine. Very light weight and easy to use. You can even serve models from Jupyter Notebooks with it without needing to worry about overwhelming the server. It just gets slower the more load you send to it.

Really cool! I like that they have live demos to prove it out. Thanks for sharing

I use ML for content moderation at Grab and my side project. Both are online. Both use queues, but Grab's traffic is more organic and real time than the side project's.

Reply on news.ycombinator.com