Multithreading support in llama.cpp is probably still pretty busted, assuming it uses the same underlying NN inference code as whisper.cpp: https://github.com/ggerganov/whisper.cpp/issues/200#issuecom...
I would fix it if I could be bothered. Instead I will just use the Cuda whisper backend which is pretty nice and fast.