HNHacker News
TopNewBestAskShowJobs

andrew-w

32 karma · joined December 15, 2023

submissionscomments
andrew-w··on Show HN: Infinity – Realistic AI characters that can speak
I agree, I think power users are happy to go to specific platforms, but APIs open up more use cases that can reach a broader audience. What kind of application would you use it for? We don't have specific plans at the moment, but are gauging interest.
andrew-w··on Show HN: Infinity – Realistic AI characters that can speak
No plans at the moment, but there seems to be a decent amount of interest here. Our main focus has been making the model as good as it can be, since there are still many failure modes. What kind of application would you use it for?
andrew-w··on Show HN: Infinity – Realistic AI characters that can speak
To be honest, one of our main goals as a startup is to move quickly, and using readily accessible cloud providers for training makes that much more easy.
andrew-w··on Show HN: Infinity – Realistic AI characters that can speak
The web app does allow you to upload any audio, but in order to use your voice, you would need to either record a sample for each video or clone your voice with a 3rd party TTS provider. We would like to make it easier to do all that within our site - hopefully soon!
andrew-w··on Show HN: Infinity – Realistic AI characters that can speak
We use Modal for cloud compute and autoscaling. The model is our own.
andrew-w··on Show HN: Infinity – Realistic AI characters that can speak
It's something we're thinking about. Our main focus right now is to make the model as good as it can be. There are still many edge cases and failure modes.
andrew-w··on Show HN: Infinity – Realistic AI characters that can speak
I see a lot of potential in animating memes and making them more fun to share with friends. Hopefully, we can do better on orcs soon!
andrew-w··on Show HN: Infinity – Realistic AI characters that can speak
I know what will be in my nightmares tonight...
andrew-w··on Show HN: Infinity – Realistic AI characters that can speak
No, it's just a hallucination of the model. The audio in your clip is synthetic and doesn't reflect any video in the real world.

Hopefully we can animate your bear cartoon one day!

andrew-w··on Show HN: Infinity – Realistic AI characters that can speak
Cartoons are definitely a limitation of the current model.
andrew-w··on Show HN: Infinity – Realistic AI characters that can speak
This is why we do what we do lol
andrew-w··on Show HN: Infinity – Realistic AI characters that can speak
Totally agree. We tweaked some settings after other commenters asked about speed, and got it up to 23fps generation (at the cost of lower resolution). Here is the example: https://6ammc3n5zzf5ljnz.public.blob.vercel-storage.com/fast...
andrew-w··on Show HN: Infinity – Realistic AI characters that can speak
All for suggestions! We've tried white noise as well, but it only works on plain talking samples (not music, for example). My guess is that the most robust solution will come from updating how it's trained.
andrew-w··on Show HN: Infinity – Realistic AI characters that can speak
I don't think we've seen laughing quite that expressive before. Good find!
andrew-w··on Show HN: Infinity – Realistic AI characters that can speak
Thank you! It has learned a surprising amount of world knowledge.
andrew-w··on Show HN: Infinity – Realistic AI characters that can speak
Very creative use cases!

We are trying to better understand the model behavior at the very end of the video. We currently extend the audio a bit to mitigate other end-of-video artifacts (https://news.ycombinator.com/item?id=41468520), but this can sometimes cause uncanny behavior similar to what you are seeing.

andrew-w··on Show HN: Infinity – Realistic AI characters that can speak
This is exactly what we do today! It seems to work better the more you extend it, but extending it too much introduces other side effects (e.g. the avatar will start to open its mouth, as if it were preparing to talk).
andrew-w··on Show HN: Infinity – Realistic AI characters that can speak
This is a good idea. We have discussed incorporating an additional "identity" signal to the conditioning, but simply enforcing consistency with the original character as a post-processing step would be a lot easier to try. Are there any tools you know of that do that?
andrew-w··on Show HN: Infinity – Realistic AI characters that can speak
Haha, wouldn't we all love that? In the long run, we will definitely need to move beyond talking heads, and have tools that can generate full actors that are just as expressive. We are optimistic that the approach used in our V2 model will be able to get there with enough compute.
andrew-w··on Show HN: Infinity – Realistic AI characters that can speak
The model configuration that is publicly available is about 5x slower than real-time (~6fps). At lower resolution and with a less conservative number of diffusion steps, we are able to generate the video at 20-23 fps, which is just about real-time. Here is an example: https://6ammc3n5zzf5ljnz.public.blob.vercel-storage.com/fast...

We use rectified flow for denoising, which is a (relatively) recent advancement in diffusion models that allow them to run a lot faster. We also use a 3D VAE that compresses the video along both spatial and temporal dimensions. Temporal compression also improves speed.

andrew-w··on Show HN: Infinity – Realistic AI characters that can speak
Thanks for the feedback! We used to have more voices, but didn't love the experience, since users had no way of knowing what each voice sounded like without creating a clip themselves. Probably having pre-generated samples for each one would solve that. Let us know if you have any other ideas.

We're also very excited about the template idea! Would love to add that soon.

andrew-w··on Show HN: Infinity – Realistic AI characters that can speak
Thanks for the feedback. The current model was trained at ~320x320 resolution. We believe going higher will result in better videos with finer detail, which we plan to do soon.
andrew-w··on Show HN: Infinity – Realistic AI characters that can speak
Yeah, it's about 5x slower than realtime with the current configuration. The good news is that diffusion models and transformers are constantly benefitting from new acceleration techniques. This was a big reason we wanted to take a bet on those architectures.

Edit: If we generate videos at a lower resolution and with a fewer number of diffusion steps compared to what's used in the public configuration, we are able to generate videos at 20-23 fps, which is just about real-time. Here is an example: https://6ammc3n5zzf5ljnz.public.blob.vercel-storage.com/fast...

andrew-w··on Show HN: Infinity – Realistic AI characters that can speak
It is possible to fine-tune the model with videos of a specific actor, but not images. You need videos to train the model.

We actually did this in early overfitting experiments (to confirm our code worked!), and it worked surprisingly well. This is exciting to us, because it means we can have actor-specific models that learn the idiosyncratic gestures of particular person.

andrew-w··on Show HN: Infinity – Realistic AI characters that can speak
This can happen with non-humanoid images. The model doesn't know how to animate them.
← PreviousPage 2 of 2