I tried making this short clip [0] of Baron Vladimir Harkonnen announcing the beginning of the clone war, and it's almost fine, but the last frame somehow completely breaks.
[0]: https://6ammc3n5zzf5ljnz.public.blob.vercel-storage.com/inf2...
[0]: https://6ammc3n5zzf5ljnz.public.blob.vercel-storage.com/inf2...
Our hypothesis is that the "breakdown" happens when there's a sudden change in audio levels (from audio to silence at the end). We extend the end of the audio clip and then cut it out the video to try to handle this, but it's not working well enough.
Hmmmmmmmm
Ohmmmmmmm
Our V2 model is trained on specific durations of audio (2s, 5s, 10s, etc) as input. So, if give the model a 7s audio clip during inference, it will generate lower quality videos than at 5s or 10s. So, instead, we buffer the audio to the nearest training bucket (10s in this case). We have tried buffering it with a zero array, white noise and just concatenating the input audio (inverted) to the end. The drawback is that the last frame (the one at 7s) has a higher likelihood to fail. We need to solve this.
And, no shade on HeyGen. It's literally what we did before. And their videos look hyper realistic, which is great for B2B content. The drawback is you are always constrained to the hand motions and environment of the on-boarding video, which is more limiting for entertainment content.