Thanks for the writeup
Our special case is a single model, high throughput and latency unsensitive.
Can we use multiple lower-memory GPUs, split the the model "horizontally", and pipeline each batch/inference across the GPUs?
Is this a common scenarios handled by the engines you mentioned ?