Hacker News Reader

Comment by theturtletalks | Hacker News Reader

Someone built an open-source version of this, but it’s far from polished:

There was this really interesting one where it reads your lips:

I’m using these to build my own so I can work in co-working spaces, planes, and libraries. Though moving my mouth with no words might draw some eyes.

View on HN

Yup we worked with IMUs as well when we started this. They measure aggregated face motion, so cannot distinguish between fine phonetic classes, only broad words. The open source demo got to a 5 word classifier, I believe. We tried a lot to make IMUs work too, in various form factors, but the signal doesn't contain enough information for open ended decoding beyond classifying between a few words. We would have loved to be able to just push software!

> There was this really interesting one where it reads your lips

This is what I thought the original post was about. The glasses you have to wear is where that lost me.

If we could get lip-reading via your computer/phone camera down perfectly, that's the real win. Making people wear your goofy frames is where you lose a massive portion of people. Nobody wants to look like Zuckerberg lol

Lip reading is one of the other modalities we tried as well beyond IMU, EMG and RF! We worked a lot on fine tuning Auto-AVSR, one of the most commonly used open source lipreading models. It's very enticing to work on, since it gets to about 17% word error rate, which looks good enough that you would think working on it further would get it down to product quality, but we found there's an information-theoretic limit on the modality. Speech has 40 phonemes, but only about 12 to 15 visemes (externally visually separable phonetic states). For example P and B look very similar from a camera. This causes the scaling laws on lipreading to flatten out at an error that's too high to be product ready but just low enough to make for a good demo. Our original plan was actually to ship a lipreading software to see if people would want this behavior, but we realized you couldn't make lipreading work!

Any deaf/HoH person who has tried to do lipreading could have told you all of this too.

Good luck on the project!

Now that is insightful! The demo definitely fooled me and even the first repo requires you to train the model.

Lip reading AI would be pretty big since privacy would probably go out the window.

Why not a pencil in the ear or something you can put in a shirt pocket? Or even earbuds?

but can't a decision model or some other model deduce "P" or "B" based on the context (i.e., previous words said)?

reminds me of tesla fsd v. waymo. may be a market for both? i.e., camera-only or camera + specific hardware.

Not well enough! You can get to 15-20%ish WER but going beyond that doesn't seem feasible with the info available from a camera, even with a great language model

Cool thanks for the reply. And congrats on the launch here.

Reply on news.ycombinator.com