Realistic one-shot mesh-based head avatars | Hacker News Reader

Realistic one-shot mesh-based head avatars

samsunglabs.github.io

147 pointslnyan50 comments

Very interesting to see the cross-reenactment renders.

In that case, I think FOMM's interpretation is more recognizable but Bi-Layer David Spade, is particularly entertaining.

Such a shame that Putin took a country with such amazing technological potential down a dead end path of hydrocarbons, political repression, violence, and war.

But hey, more world class engineers for the rest of us.

Good catch. Samsung just saved tens of millions of dollars hiring such talent.

Yeah, Russia has the brains and education to have made their own Samsung…but too bad, an aging psychopath wants to silence anybody with half a brain and dig in the mud for “wealth”.

This is the resource curse that is well known.

The west should be paying to help get this talent fleeing their country.

Many countries are! Talent bonanza.

What is the real use case for this?

Making avatars without the 3D camera thing that phones do. Could also make avatars of famous people, deceased loved ones...

I think this could be used as a great basis for next generation video conferencing. Rather than compressing raster images of people speaking, we could send a model as an initial payload and subsequent instructions on how to manipulate it as the conversation progresses.

Most current networks can handle video conferencing (relatively) well. However, if 8k video or something like Light Field displays ever catch on, then we might end up needing orders of magnitude more bandwidth to drive video conferencing, so this might be an enabling technology.

Might not be that simple. In a recent demo from Unreal Engine they said that the facial animation data is the main bottleneck to shipping titles with more realistic avatars. Their minute-long demo took many TBs to store the motion data alone.

My layperson understanding makes this baffling to me. Isn't motion inherently analog, and haven't we've developed high performing analog compression already? Motion seems significantly more amenable to lossy compression as well, and I could imagine a use-case for ML-based decompression to fill in between the keyframes.

You also have to consider that faces are one of the things that we have the strongest perception of, with lots of our neurons dedicated to the task, so when you get things wrong it's far more noticeable than many other bodily animations would be.

One of my favorite “weird” conjectures is that the existence of the uncanny valley implies that at some point it was evolutionarily advantageous for us to recognize something that looked human, but wasn’t… and to be afraid of it.

neanderthals?

Think also about face-like things we perceive because of pareidolia but might in fact be dangerous. One part of our brain is opportunistic about finding patterns while another is a sanity check on those patterns telling us not to just trust them outright.

Is there any further reading on this topic in particular that is not leading to wild conspiracy theories?

I had the same reaction (note: I actually got things mixed up, the one in question is the Unity "Enemies" demo [1], not Unreal), but if you consider the amount of detail - the saliva 'sticking' to the corners of the mouth, the 40+ muscles of the face, lips, eyelids, the iris, etc - it makes sense that it would be a ton of data. IIRC they are working on some kind of ML-powered compression so that this becomes actually possible to ship (the demo is not available to download, presumably it's too big and would barely run on a 3090TI).

[1] https://www.youtube.com/watch?v=eXYUNrgqWUU

Facial animation is a real issue. This is facial mapping, which is an easier to solve problem, in that throwing artists at the problem is a solution.

> Rather than compressing raster images of people speaking, we could send a model as an initial payload and subsequent instructions on how to manipulate it as the conversation progresses.

What about latency though? Would that improve?

Or, at least, the faces that aren't actively speaking could be replaced by avatars. It will also help keep a clean image in case stuff is going on the the background, etc.

I guess, would watching an animated head avatar in 8k or a Light Field display be a better experience than watching lower resolution video?

I wonder whether the experience would be too undermined by artifacts from e.g. your coworker's cat, for which the system doesn't have a model, wanders into frame partway through a call, and suddenly the system must dynamically either (a) 'downgrade' to just video (b) be able to create a new model for an arbitrary new thing on the fly (c) create some hybrid of partial video data with part viewer-rendered animation or (d) your coworker looks crazy interacting with something the system hides from you entirely.

there's another thread about just this, but very different approach. reduces bandwidth a lot specifically for video conferencing

This would be useful for anyone that has specific company requirements for their avatar/image as long as it can be combined with something that detects the face and eliminates the background

It's a stepping stone towards the integration between NeRF derived and DALL-E 2 CLIP diffusion type systems to create user defined on demand real-time volumetric 3D digital spaces for spatial computing.

Is "user defined on demand real-time volumetric 3D digital spaces for spatial computing" in this instance a euphemism for fanfic porn?

I would make a model of myself create a nice background, then detect face expressions, apply them to the model and stream the model while I work shirtless from a dirty basement.

Easier rigged characters in video games that can be personalized to a player

I think it could be used to create a videoconferencing experience akin to sitting around a table with multiple people, while still selectively facing or making eye contact with individuals.

User avatars for games or video conferences.

Post fx tooling for movies. Cheaply touch up or add virtual extras to a scene.

Video conferences. You can create a 3d world projecting 3d avatars made from images from peoples web cams. Creating a 3d sense of being in the same room.

Acting in 3d recreation of your favorite movies. If you can scan faces you could play in rendered 3d recreations of movies.

Video games you could from a web cam image play a 3d avatar of yourself in a game.

Impressive work! The main cons of this seems to be unbaking natural light from the texture

Why is Samsung employing people in Russia?

Maybe a group of people did a thing and Samsung wanted it

Strong math and programming school? OpenCV was largely developed by Intel Russia, for example.

And Intel fired all their Russian employees, like all ethical western companies should be doing.

It's not the employees' fault that the people running their country have decided to wage war.

But they are paying taxes to the regime. If they want to feel unimplicated, they should leave Russia.

From my experience, generally programmers in Russia are less conservative and more pro-West than the general population (especially those working for Western companies), I don't see anything ethical about firing them. In their statement, Intel said "We are working to support all of our employees through this difficult situation, including our 1,200 employees in Russia" because they understand those people have nothing to do with Putin. From what I gathered, most companies stop operations because, first, paying salary and selling has become difficult due to sanctions, and, second, it's bad PR for them if they keep working in Russia when everyone else left. So it doesn't look to me that it's about ethics, just a business decision.

They should leave Russia. They’re currently paying taxes to the regime.

Emigration isn't an easy process, it can't happen overnight. You need a visa (now harder to obtain for Russians), you need enough savings (not everyone has enough money), you should be able to speak a foreign language (not everyone is skilled enough to get a job from day 0), you will have to leave all your friends and most of your relatives behind, everything/everyone you loved and invested into, both financially and emotionally - due to a military conflict which may end soon (no one knows). I have elderly parents I take care of - I can't emigrate without them. It's easy to judge people when it doesn't affect you personally. If I remember correctly, 75% federal budget of Russia comes from oil/gas revenues, so in a sense, Europeans are more complicit in funding Putin's war crimes than Russian taxpayers. I wonder why Europeans aren't emigrating en masse. And it won't really help if everyone who opposes Putin's war emigrates, it will only worsen the situation because there will be no one left to change the country from within, the country will only get even more radicalized/conservative. It's not so simple.

https://www.russia-briefing.com/news/president-putin-spief-2...

From Putin’s most recent speech - “Many Russian AI and Big Data solutions are the best in the world.”

Putin is bragging about his AI and Big Data solutions. This is not a bunch of isolated engineers working alongside the west. Putin will be leveraging these engineers to drive massive disinformation campaigns worldwide.

We should not be celebrating this work.

Why shouldn't they?

Sir, this is not Twitter

Is this... a Wendy's?

I'll show myself out.

Sir, this is not the time to overlook Samsung enabling the economy of Russia.

It was supposed to be a lighthearted joke but now I'd seriously suggest you to review your sources of information on what enables the economy of Russia

Half of which is their petroleum exports. There is another half. Sorry to be such a stickler, but Samsung is enabling Putin. They wouldn’t do the same for Kim Jong-un. It’s insulting they’re helping Putin.

They wouldn't do the same for NK for the sole reason they are not allowed to do it. They and many other companies that ostentatiously left the russian market, have absolutely no qualms about who they are helping. Just look at all the McDonalds in Saudi Arabia, I guess they are fine with Yemen genocide as long as the world's moral police is looking another way

Reply on news.ycombinator.com