Hacker News Reader

I agree. The ML Algorithms and models can run into huge sizes. I have one with half a gig of serialised model, when running takes up close to 3.5 Gigs. This approach of directly using it in the views would pose scaling issues.

It really depends. For sure there are models which should be processed in the background. It is a very basic tutorial that shows how simple ML models can be used as REST API. In the tutorial I was using Random Forest trained on UCI Adult Income data set. The final model was small, and computing predictions was fast.

I would love to add more advanced part of the tutorial with background processing for large models.

Inference should be fast, so an async view (which we're excited finally happened!) should be fine. It's similar to waiting on a db. Read consistency means using similar memory. Likewise, you want it to be warm, not reloaded between calls.

In our case, we do on-the-fly GPU calls, with weird spiking behavior, so been looking for a good serverless GPU solution for this kind of stuff.