Hacker News Reader

Comment by tehlike | Hacker News Reader

You might want to look into diarization also http://gladia.io/ seem to be doing it well.

It makes a great difference to have transcripts with speaker annotation.

Thanks, I haven’t seen an easy and reliable way to do this using open source stuff yet. Theoretically just separating out speakers seems like it wouldn’t be that hard; just compute a bunch of FFTs to arrive at a sort of frequency-based “voice fingerprint” for each speaker and then use something like XGboost to match up the audio for each second to one of the speakers. The problem is then what do you with that information? Turning those abstract speaker identifications into actual names would seem to require a fair bit of intelligence and picking up from contextual clues (like if the speaker identifies themselves or introduces another person by name). Anyway, I’ll look into it more. If it could be done reliably without overly complicating the setup, I agree that it would be useful.

That part can be user input, if it needs to be. Sort of like post processing.

Founder of Gladia shares some information on twitter, and i think there's some research papers you can find through that (my memory is fuzzy on when). IT's not a simple problem, especially when words include space fillers like "umm" etc.

For me main use for such cases would be podcast. sometimes i just want to read them without listening.

I built it for our podcast, I'm sure you can re-use it for YT as a source rather than raw audio file: https://github.com/FanaHOVA/smol-podcaster

The whisperX project has most of it covered they are integrating the new v3 model waiting for a ccp GGUF version also there is a workaround for the latest speaker diarization but it has an active user base working on it

https://github.com/m-bain/whisperX

I was dismayed to learn that this requires OpenAI API keys for speaker diarization.

Does it? I only see references to HuggingFace api keys so it can do a one time download of some additional models.

Oh, maybe that's what I saw. I'll have to look again. Thanks for keeping me honest.

Thanks, took a look at it. Seems quite heavy though, lots of huge dependencies like pytorch and torchaudio, and seems like the speaker diarization requires a GPU if I'm not mistaken. And as another poster pointed out, it does require a Huggingface API key as well.

I wanted to keep my script lighter weight and also GPU optional (i.e., a GPU will work and make it faster, but it also works acceptably with just the CPU). I really feel in my gut that the speaker diarization doesn't need to be so complicated or hard once you already have the accurate timestamps of each transcribed segment and the underlying audio file-- no reason why it shouldn't be able to run fine on a CPU and get good enough accuracy.

to zoom out a bit, this is known as the 'cocktail party problem' and is very much considered unsolved (particularly when there is an unknown number of speakers or overlapping speech).

> Turning those abstract speaker identifications into actual names would seem to require a fair bit of intelligence

As someone looking for this functionality, this is the easiest part for me to do manually. Just give me "Speaker A, Speaker B, Speaker C", and I can change their names. It's the breaking apart of audio into separate speakers that's difficult - I'm trying out a few different tools, and none of them do a great job so far.

Wow, why are they so expensive? Like even the regular whisperAPI by OpenAI is less expensive.

This is also why I decided to create https://www.betterwhisperapi.com/ . I believe most of the companies are charging pretty insane amounts for transcriptions...

Deepgram is really good and around your price point too. They also have $200 free credit which should be more than enough for most hobby protects.

Reply on news.ycombinator.com