Google Research and Daydream Labs: Seeing eye to eye in mixed reality | Hacker News Reader

Google Research and Daydream Labs: Seeing eye to eye in mixed reality

blog.google

137 pointspetethomas10 comments

That is really cool! They are using a realistic 3D model of the persons face and eye-tracking in the VR headset to make a good guess at their facial expression, effectively reconstructing the face that's hidden by the headset.

Not something you can easily do as a consumer yet, but for high quality mixed reality production this seems like a great boon.

For an explanation of mixed reality videos are made I recommend this: https://www.youtube.com/watch?v=8T7ux3DXP_w

This is really cool work. I'm not sure about anyone else, but this doesn't get past the Uncanny Valley for me.

I really liked the approach that Oculus are taking and which Zuck demo'd last year[1], with cartoon avatars that are still expressive enough to convey facial expressions.

It definitely falls on the other side of the uncanny valley, and anyway, who is going to want to render a realistic version of themselves into the VR world, when you can be <insert favourite person/creature/character>?

That said the tech in the OP is very cool, and I can see plenty of use cases (e.g. business meetings or celebrity broadcasts).

[1]: https://www.facebook.com/zuck/videos/10103154531425531/

Well that's designed for in-VR avatars, this method is designed for mixed reality videos (VR footage with the real-world person superimposed on it)

In the FB video, I think the facial expressions were like emoticons: they had to be manually invoked. This approach could do it automatically in realtime, by mapping the facial movements onto a cartoon avatar (probably easier than onto the real face).

Interesting, do you have a source for that info, or is that a guess?

Watching the video the eyebrow movements look a bit twitchy, I'd assumed they were face-tracking them. I have no supporting evidence beyond watching the video, so I'd be interested to know more details either way.

That's very cool, but a LOT of production work.

Probably fails the cost-benefit analysis for most people doing Mixed Reality videos for their VR products. However, if it was productised, that'd be a different story.

Reply on news.ycombinator.com