I was just looking into this today!
The options I've found, but yet to evaluate:
- TorchScript + tch = Use `torch.jit.trace` to create a traced model, load with tch/rust-tokenizers
- rust-bert + tch = Seems to provide slightly higher-level usage, also use traced model
- ONNX Runtime - Convert (via transformers.onnx) .pt model to .onnx encoder and decoder, then use onnxruntime+ndarray for inference
- Candle crate - Seems to have the smallest API for basic inference, and AFAIK can load up models saved with model.save() without conversion or other things
These are the different approaches I've found so far, but probably missed a bunch. All of them seem OK, but on different abstraction-levels obviously, so depends on what you want ultimately. If anyone know any other approach, would be more than happy to hear about it!