I see from github that you use online hosted API to transcribe voice, you should instead use offline whisper, I recommend to look into whisper.cpp
I haven't had the time to set-up whisper - but I don't even have a gpu. By the way Deepgram hosts Whisper too, but according to https://news.ycombinator.com/item?id=35367655 their models are much better. Transcribing the video was almost instant.