Regardless of "intelligence" it's got real utility.
There are a couple of GitHub discussions on the Whisper repository with various fixes/hacks to deal with it: https://github.com/openai/whisper/discussions/679 https://github.com/openai/whisper/discussions/813
If you get a chance, I encourage you to try out the other newer models I mentioned, I think you'd be very impressed.
As for the silence, I wonder why the the model even receives it. I would think a lot of that would be compressed out of existence to save bandwidth.
Last week I set up a Whisper instance on my server and have been feeding it with these files. The result is pretty good. I usually can remember what I was saying when I read the transcription, which usually contains a couple of errors. Then there are those added hallucinations which are entire sentences, like:
----
00:00.000 --> 00:05.000 Also temperaturmäßig ist es recht gut. [So temperature wise, it's pretty good.]
00:05.000 --> 00:09.000 Der eine hat 12 Grad, der andere 10. [One has 12 degrees, the other 10. (I have two temperature sensors mounted on the bike, ESP32 streaming the data to the phone via BLE)]
00:09.000 --> 00:12.000 Also sagen wir mal, 10 Grad. [So let's say 10 degrees.]
00:14.000 --> 00:19.000 Es ist bewölkt und windig. [It's cloudy and windy.]
00:20.000 --> 00:24.000 Aber irgendwie vom Wetter her gut. [But somehow from the weather it's good.]
00:24.000 --> 00:31.000 Ich habe heute überhaupt nichts gegessen und sehr wenig getrunken. [I ate nothing at all today and drank very little.]
00:54.000 --> 00:59.000 Vielen Dank für's Zuschauen! [Thanks for watching!]
Transcribed in 77.2 seconds
----
The last sentence, "Thanks for watching!" is a complete hallucination. There were 30 seconds remaining which were me breathing and the wind blowing into the microphone and it came up with that comment.
I usually comment on the weather because I take note of what I am wearing, and it allows me to better prepare for future rides.
77 seconds for the 60 second file because my server has no GPU, so I'm running the large model on the CPU (in a VM which has 8 cores assigned to it from a Ryzen 9 5950X). I've been considering buying a small PC with a 3060 RTX only for inferencing, but it may be too expensive. I tried Google Speech-To-Text and it is nowhere as good as Whisper under these conditions (having the wind noise and the heavy breathing).
This is Google's result:
----
"Also temperaturmäßig es ist recht gut, der eine hat 12° andere 10. Es ist angemalte 10 Grad. Es ist bewölkt und windig, aber er hat sie vom Wetter her gut, ich wollte überhaupt nichts gegessen und sehr wenig getrunken."
["So temperature-wise it's pretty good, one has 12° other 10. It's painted 10 degrees. It's cloudy and windy, but he has it good from the weather, I did not want to eat anything at all and drank very little."]
----
Also, whisper.cpp doesn't seem to generate the same results, and they appear to be not so good (in this case it was almost just as good). I just tested the same file on whisper.cpp with the large model and it's even funnier:
----
[00:00:00.000 --> 00:00:05.000] also temperaturmäßig ist es recht gut [...]
[00:00:05.000 --> 00:00:09.000] der eine hat 12 Grad, der andere 10 [...]
[00:00:09.000 --> 00:00:12.000] also sagen wir mal so 10 Grad [...]
[00:00:12.000 --> 00:00:19.000] es ist bewölkt und windig [...]
[00:00:19.000 --> 00:00:24.000] aber irgendwie vom Wetter her gut [...]
[00:00:27.000 --> 00:00:31.000] ich habe heute überhaupt nichts gegessen und sehr wenig getrunken [...]
[00:00:31.000 --> 00:00:35.000] das ist der Grund, warum ich so viel auf dem Knie gehe [this is the reason why I go so much on the knee]
[00:00:35.000 --> 00:00:39.000] das war's, bis zum nächsten Mal! [that's it, until next time!]
[00:00:39.000 --> 00:00:59.000] Danke fürs Zuschauen! [Thanks for watching!]
`time` yields 567.63s user 1.99s system 755% cpu 1:15.36 total
----
The first 30 seconds, where the text is clearly understood, is inferenced within ~10-15 seconds. It's the "silence" which makes the AI go crazy on the workload.
The idea behind this is to set up a system which then sends me an email with a map and trail of the ride as well as the transcriptions of the notes.
Pay per second GPU processing, with an example of running whisper over 10 GPUs in parallel.
So this is really 0 USD when not in use? I'm also intending to use this for transcribing my phone answering machine recordings, so the transcription requests come in at random times which means that the transcription service should be constantly available.
It's a newer business so I guess you should factor that risk in though.