Hacker News Reader

Comment by dougabug | Hacker News Reader

Not so much. Check out E3GAN from Gordon Wetzstein’s lab at Stanford: https://matthew-a-chan.github.io/EG3D/. In particular, note how they are able to apply a process called GAN Inversion to find a point in the 3D GAN’s representation space to generate a 3D model of a person’s face (one of the collaborators) from a single query image.

View on HN

Seems like this will also get to a point where we would just take a photo and animate it completely as a 3D mesh. Like that picture, just on a few photos, at some point, would generate a photorealistic, ray-traced 3D asset and also a fully interactable 2D motion paired with GPT-4 or 5 where you could have indistinguishable level of human like interaction (ex. Fable studio)

After a couple dozen more papers and prototypes, we could have ultrasound haptic feedback and some 3D holographic projector where we achieve a true augmented reality.

Going a bit further with this idea, you could essentially create augmented 3D objects that would be indistinguishable from the real thing until you apply enough force enough to break through some upper limit of ultrasound generated haptic pressure (as to not hurt the end user through some legislative regulation). For example, opposing magnets that can be pushed together past a certain point where the illusion of tactility is gone.

What would a reality like that be like? Where you could not only have limitless content media generator but you could conjure up those assets in your living room and have them interact kinetically with the real world. We are talking 3D holographic assets with ultrasound haptic simulator that can apply pressure to any object from all directions.

https://www.youtube.com/watch?v=fwTcfwbrNO4

https://www.youtube.com/watch?v=jDRTghGZ7XU

We might not even need expensive robots in the future. Just conjure up a 3D human via some ultrasound/holograph projector which can apply enough force to objects.

But it's much trickier to find the point in a GAN's representation space for "the exact same fluffy pink earrings that the actress was wearing in the last scene".

Mostly because one can't write code to judge equal-ness of pink earrings. Or the fact the trash can in the background had a lid in the last scene and is now missing the lid. Or the fact the brick wall has turned into a painted wall. etc.

Having said that, I'm sure some filmmakers will go for it anyway - after all, human-made films have some consistency issues, and many viewers simply don't care.

I think the main thing is it becomes a programming problem rather than an AI limitation. From what I've seen it's now possible for the AI to generate EarringsAsset57 and KitchenWall51 and then to ask the AI to draw the actor wearing EaringsAsset57 in front of Kitchenwall51 and for stills at least, it can produce acceptable results

You can decompose an entire scene into a collection of 3D GANs, each of which is fit by GAN Inversion to the object instance which best explains what is seen in the images.

Also, we absolutely can tell the difference between trash cans with and without lids, as well as brick walls vs painted walls. This falls under many areas of ML-based Computer Vision, such as metric, contrastive, and zero shot machine learning, also open world object detection / recognition. These frame level semantic discrepancies can be detected by discriminative DNNs, which we can use to provide a temporal consistency loss to guide the generator to produce temporally consistent videos.

> But it's much trickier to find the point in a GAN's representation space for "the exact same fluffy pink earrings that the actress was wearing in the last scene".

Not so much, since you can supply an image of the desired item and get back a bunch of vectors to reuse.

Reply on news.ycombinator.com