You can leverage Nvidia Triton Inference server to do this.

In case you want a managed Infra service look at this cookbook on how you can use Transformers, Inferless to serve multiple LLMs - https://docs.inferless.com/cookbook/serverless-voice-chatbot