Are you doing CPU inferencing? For small models you could check qxotic.ai which is a whole inferencing stack for many popular open models in Java, which can beat llama.cpp in speed in some cases. Its tokenizer toknroll is faster than tiktoken, which is what made me think of it for your use case.

The advantage is that you don't have issues like the GIL or whatever, so there's no need for all the complexity of load balancing between multiple pythons.