Hacker News Reader

Comment by wkirby | Hacker News Reader

I'm very intrigued by AI note takers, but I'm absolutely unwilling to expose me or my clients to this exact problem.

The solution (theoretically) is a purely local note taker, but I haven't found one that's any good. Tried meetily and others in the same vein, including briefly rolling my own. The breakdown in the pipeline seems to be reliable local diarization and speaker identification; even if the transcription is good, when speakers aren't accurately identified and speech isn't well grouped, there's no rescuing it in the summary step.

View on HN

> I'm very intrigued by AI note takers, but I'm absolutely unwilling to expose me or my clients to this exact problem

Unfortunately it’s mostly not up to you. It’s a weakest-link problem. It doesn’t matter if you don’t use a note taker AI, if even one person on the call uses one. Their tool doesn’t notify you and usually the person doesn’t either.

It also has the reverse impact to the person using the note taker, where people say less around them. Same as if I'm talking to someone with Meta glasses.

I wonder if the people who use these tools know the people they meet with speak less during their meetings, and then all of the participants have a post-meeting call without them to say what they really thought.

> It doesn’t matter if you don’t use a note taker AI, if even one person on the call uses one

Yeah, but I'm unwilling to be that person.

I'm definitely biased as the developer, but maybe try https://whistle-enterprise.com and see how it works for you.

It's a hard problem I've been working away on for a while now. It's far from perfect but every step brings it a bit closer.

Literally starting my monday weekly standup now, I'll run it and see what's up. Thanks!

Thank you! Feel free to drop me and email (info in bio or on the website) if you run into any issues or have any suggestions.

Seems interesting. What would you say are the biggest missing points currently or things you want to get working/improved but couldn't yet?

Thank you, great question! Hard one to answer, thought about it a lot and it's going to be the diarisation of more than 5+ speakers per audio stream (your microphone + system audio for a max of 10). I actually spent a lot of time that went completely nowhere trying to fine tune my own diarisation model, it was fun to a degree but painful to see my output end up worse than what I currently had after days of work. Having a bot join the call would be such an easy way of diarising, the call software has already done it for you, but feels like a bit of a cop out.

Two more improvements, audio quality improvements which is currently in the works and close to release and a new document generation model. I'm currently using a custom fine tuned Phi-4 (released December 2024!) model, that's _so old_ in the grand scheme of LLMs, I just haven't had time to benchmark and properly test some new models whilst this currently does a good job as it is. There has to be some gains here, but who knows!

Interested also, which text-to-speech model do you use? For diarisation Granola uses a chrome extension instead of a bot if that can give you ideas.

I think this is just to capture the audio to ship off to their servers and do the crunching. You mentioned the word "extension" and thank you so much, I've been thinking about how to do integrations as it's something a few users have mentioned, but keep the whole "completely offline" angle. I could build standalone extensions that integrate with it if it's something a user wants. So damn obvious in hindsight!

Ahh yea as for the models:

Speech to text - Nvidia Parakeet TDT 0.6b V3

Diarisation - Nvidia Marblenet for the speech detection, TitaNet-Large for the embeddings and then using NeMo multi-scale to do clustering around them

Since the participants are known and limited, have you tried building around samples tagged with user/person names?

I tried something similar by putting together a "global ledger" of speaker identities. This was more happening during the call than having a predefined one but I just couldn't get it to work properly. The issue being that as soon as some speech gets assigned to a new label or an incorrect one, everything ends up getting misaligned and gets messy quickly.

I might take another look into doing it in a different way that gradually builds up from successful calls, I just need to think of how to do this in a simple(ish) way for non-technical users and a way that still works well enough on low to mid tier laptops.

I know this is hn and not a product development meeting, but:

If you’re targeting data for people to use on the same call then requires more intense work, while if you’re targeting data for people to use after or on an ongoing basis (eg an established business meeting with staff/vendors/etc) then having a predefined one seems good. Same interface as a CRM: picture, some audio clips for users to choose from, and confidence scores on each section of the recording for cases where people sound similar or are talking over each other. Over time the diarization gets better as users accept/reject/tag samples and that work helps them feel more aligned with the tool.

100% looking at it from an after use basis, whilst there is live transcription, trying to do anything above that on 6 year old laptop CPU just grinds stuff to a halt.

Some great suggestions, appreciate you taking the time to write it out, it's given me some things to think about.

That’s a very fair concern. I completely understand why you’d want to avoid exposing yourself or your clients to that risk.

I also agree that diarization and speaker identification are probably one of the hardest parts to get right if the speakers aren’t separated correctly, even a great summary won’t fix it.

We’re looking into more privacy preserving approaches for MindNote (a multimodal AI Notetaker), including local/on-device processing, so this is really useful feedback. Thanks for sharing your experience!

> The breakdown in the pipeline seems to be reliable local diarization

Yep, diarization just hasn't been well solved yet. As soon as it has, the quality in note-takers, meeting transcripts, etc, will sky-rocket across the board.

Drafts.app is hideous but it has great routing capability and a dictation feature. I use it to capture what my thoughts and route based on content. I have a button that routes to an internal voice agent named KiKo. Ideas get routed to Things or todoist. Issues get routed to github, etc. It’s one universal surface for note capture.

But man is it ugly.

My ideal use case is to pipe audio from both my microphone and capture system audio so things like our weekly team standup or my 1:1s with my devs can all have reliable, decent notes without taking me out of the flow of the conversation.

I think clearly the _leader_ in the space is granola, but I'm just not going to use a cloud provider for this.

Drafts have anything like that?

Reply on news.ycombinator.com