Any idea what these are ? especially the pre-computation/caching and batching. I'm not able to see what advantage does batching bring...or how you can really cache a prediction request
Pre-compute the recommended hotels for my top users every night. Now when that user comes back, they see a slightly stale recommendation, but it's lightning fast.
You can also pre-compute and cache some of the inputs cheer model, like maybe a vector representation of the description of a hotel.
For the same hardware load, you can process several samples instead of just one.
Pre-computation means running your model on samples in advance, before the model result is needed, so it's ready to use instantly when needed.
Caching works probably because there are model results that are reused again and again, so it makes sense to cache them. For example, there are deep models that process room pictures, room and customer characteristics. Only customer characteristics change between customers, so it makes sense to cache the features output by the deep CNN that processes the room pictures.
Once you start doing prediction at scale, there are lots of these optimizations to pick up.
All our models are balanced using multi-armed bandits, so for our recommendations engine, we run lots of arms that depends on the incoming channel, were in the app is being shown the recommendation, etc and just combine the outputs of the models.