You could do something two pass and collect forced alignment confidence scores similar to what is fine with whisperx. The OP would need to incorporate that into the app but it’s a pretty standard approach.